[go: up one dir, main page]

WO2024049998A1 - Neuromorphic analog signal processor for predictive maintenance of machines - Google Patents

Neuromorphic analog signal processor for predictive maintenance of machines Download PDF

Info

Publication number
WO2024049998A1
WO2024049998A1 PCT/US2023/031692 US2023031692W WO2024049998A1 WO 2024049998 A1 WO2024049998 A1 WO 2024049998A1 US 2023031692 W US2023031692 W US 2023031692W WO 2024049998 A1 WO2024049998 A1 WO 2024049998A1
Authority
WO
WIPO (PCT)
Prior art keywords
analog
implementations
neural network
network
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2023/031692
Other languages
French (fr)
Inventor
Boris Maslov
Aleksandrs TIMOFEJEVS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Polyn Technology Ltd
Original Assignee
Polyn Technology Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/902,757 external-priority patent/US20230081715A1/en
Application filed by Polyn Technology Ltd filed Critical Polyn Technology Ltd
Priority to JP2025507806A priority Critical patent/JP2025531660A/en
Priority to CN202380053827.6A priority patent/CN119547080A/en
Priority to EP23772700.3A priority patent/EP4581521A1/en
Publication of WO2024049998A1 publication Critical patent/WO2024049998A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01H—MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES
    • G01H1/00—Measuring characteristics of vibrations in solids by using direct conduction to the detector
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01M—TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
    • G01M13/00—Testing of machine parts
    • G01M13/04—Bearings
    • G01M13/045—Acoustic or vibration analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06G—ANALOGUE COMPUTERS
    • G06G7/00—Devices in which the computing operation is performed by varying electric or magnetic quantities
    • G06G7/12—Arrangements for performing computing operations, e.g. operational amplifiers
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G06N3/065—Analogue means

Definitions

  • Neuromorphic processors based on spike neural networks are limited in their applications.
  • power and speed of such architectures are limited by data transmission speed. Data transmission can consume up to 80% of chip power, and can significantly impact the speed of calculations.
  • Edge applications demand low power consumption, but there are currently no known performant hardware implementations that consume less than 50 milliwatts of power.
  • Memristor-based architectures that use cross-bar technology remain impractical for manufacturing recurrent and feed-forward neural networks.
  • memristor-based cross-bars have a number of disadvantages, including high latency and leakage of currents during operation, which make them impractical.
  • memristor-based cross-bars there are reliability issues in manufacturing memristor-based cross-bars, especially when neural networks have both negative and positive weights.
  • memristor-based cross-bars cannot be used for simultaneous propagation of different signals, which in turn complicates summation of signals, when neurons are represented by operational amplifiers.
  • memristor-based analog integrated circuits have a number of limitations, such as a small number of resistive states, first cycle problems when forming memristors, the complexity with channel formation when training the memristors, unpredictable dependency on dimensions of the memristors, slow operations of memristors, and drift of state of resistance. [0005] Additionally, the training process required for neural networks presents unique challenges for hardware realization of neural networks.
  • a trained neural network is used for specific inferencing tasks, such as classification. Once a neural network is trained, a hardware equivalent is manufactured. When the neural network is retrained, the hardware manufacturing process is repeated, driving up costs. Although some reconfigurable hardware solutions exist, such hardware cannot be easily mass produced, and costs a lot more than hardware that is not reconfigurable (e.g., five times the cost). Further, edge environments, such as smart-home applications, do not require re-programmability as such. For example, 85% of all applications of neural networks do not require any retraining during operation, so on-chip learning is not that useful. Furthermore, edge applications include noisy environments, which can cause reprogrammable hardware to become unreliable. [0006] One such application is predictive maintenance of machines.
  • Vibrational sensors are commonly used to measure vibrations in machinery, tracks, railway cars, wind turbines, and oil and gas pumps. Sensors that monitor rotation or reciprocating parts of machines typically generate signals in the range from 0 to 20 kHz. This makes the sensors consume a lot of power, making it almost impossible to use batteries. Signals due to the vibrations may be transferred wirelessly to an equipment for analysis. A wideband connection to a control unit is typically necessary because the signals can include a large amount of data flow. This data flow may shorten the battery life of operating sensor nodes. SUMMARY [0007] Accordingly, there is a need for methods, circuits and/or interfaces that address at least some of the deficiencies identified above.
  • Analog circuits that model trained neural networks and manufactured according to the techniques described herein can provide improved performance per watt advantages, can be useful in implementing hardware solutions in edge environments, and can tackle a variety of applications, such as drone navigation and autonomous cars.
  • the cost advantages provided by the proposed manufacturing methods and/or analog network architectures are even more pronounced with larger neural networks.
  • analog hardware implementations of neural networks provide improved parallelism and neuromorphism.
  • neuromorphic analog components are not sensitive to noise and temperature changes, when compared to digital counterparts.
  • Chips manufactured according to the techniques described herein provide orders of magnitude in improvement over conventional systems in size, power, and performance, and are ideal for edge environments, including for retraining purposes.
  • Such analog neuromorphic chips can be used to implement edge computing applications or in Internet-of-Things (IoT) environments. Due to the analog hardware, initial processing (e.g., formation of descriptors for image recognition), that can consume over 80-90% of power, can be moved onto a chip, thereby decreasing energy consumption and network load, which can open up new markets for applications. [0009] Various edge applications can benefit from the use of such analog hardware. For example, for video processing, the techniques described herein can be used to include direct connection to a CMOS sensor without a digital interface. Various other video processing applications include road sign recognition for automobiles, camera-based true depth and/or simultaneous localization and mapping for robots, room access control without a server connection, and always-on solutions for security and healthcare.
  • Such chips can be used for data processing from radars and lidars, and for low-level data fusion.
  • Such techniques can be used to implement battery management features for large battery packs, sound/voice processing without connection to data centers, voice recognition on mobile devices, wake up speech instructions for IoT sensors, translators that translate one language to another, large sensors arrays of IoT with low signal intensity, and/or configurable process control with hundreds of sensors.
  • such techniques can be used for implementing human activity recognition.
  • Substituting conventional software algorithms with analog neural network implementations can help offload computation from device micro-controllers so that the total power consumption of the device is decreased substantially (e.g., power consumption decreases by over 40%).
  • neural network determination of human activity, or generating descriptors allows recognition of more than fifty different human activities.
  • Neural networks allow personalization of activities, and descriptors can be generated as unique digital fingerprint for specific users.
  • Neuromorphic analog chips can be mass produced after standard software- based neural network simulations/training, according to some implementations.
  • a client’s neural network can be easily ported, regardless of the structure of the neural network, with customized chip design and production.
  • a library of ready to make on-chip solutions are provided, according to some implementations.
  • the techniques described herein can be used to design and/or manufacture an analog neuromorphic integrated circuit that is mathematically equivalent to a trained neural network (either feed-forward or recurrent neural networks).
  • the process begins with a trained neural network that is first converted into a transformed network comprised of standard elements. Operations of the transformed network are simulated using software with known models representing the standard elements. The software simulation is used to determine the individual resistance values for each of the resistors in the transformed network. Lithography masks are laid out based on the arrangement of the standard elements in the transformed network.
  • each of the standard elements is laid out in the masks using an existing library of circuits corresponding to the standard elements to simplify and speed up the process.
  • the resistors are laid out in one or more masks separate from the masks including the other elements (e.g., operational amplifiers) in the transformed network. In this manner, if the neural network is retrained, only the masks containing the resistors, or other types of fixed-resistance elements, representing the new weights in the retrained neural network need to be regenerated, which simplifies and speeds up the process.
  • the lithography masks are then sent to a fab for manufacturing the analog neuromorphic integrated circuit.
  • a method is provided for hardware realization of neural networks, according to some implementations.
  • the method incudes obtaining a neural network topology and weights of a trained neural network.
  • the method also includes transforming the neural network topology to an equivalent analog network of analog components.
  • the method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network.
  • the method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. [0013]
  • generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the method further includes obtaining new weights for the trained neural network, computing a new weight matrix for the equivalent analog network based on the new weights, and generating a new resistance matrix for the new weight matrix.
  • the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: for each layer of the one or more layers of neurons: (i) identifying one or more function blocks, based on the respective mathematical function, for the respective layer.
  • Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function; and (ii) generating a respective multi-layer network of analog neurons based on arranging the one or more function blocks.
  • Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multi-layer network is connected to one or more analog neurons of a second layer of the multi-layer network.
  • ⁇ ⁇ represents an i-th input and ⁇ ⁇ represents a j-th input, and ⁇ ⁇ ⁇ ⁇ ⁇ is a predetermined coefficient;
  • a hyperbolic tangent activation block with a block output ⁇ ⁇ ⁇ ⁇ tanh ( ⁇ ⁇ ⁇ ⁇ ).
  • ⁇ ⁇ represents an input, and A and B are predetermined coefficient values; and
  • a signal delay block with a block output ⁇ ( ⁇ ) ⁇ ( ⁇ ⁇ ⁇ ⁇ ).
  • identifying the one or more function blocks includes selecting the one or more function blocks based on a type of the respective layer.
  • the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions.
  • Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks.
  • Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network.
  • the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting the one or more weights such that combining the one or more intermediate functions results in the mathematical function.
  • the method further includes: (i) generating equivalent digital network of digital components for one or more output layers of the neural network topology; and (ii) connecting output of one or more layers of the equivalent analog network to the equivalent digital network of digital components.
  • the analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • selecting component values of the analog components includes performing a gradient descent method to identify possible resistance values for the plurality of resistors.
  • the neural network topology includes one or more GRU or LSTM neurons, and transforming the neural network topology includes generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.
  • the one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology.
  • the neural network topology includes one or more layers of neurons that perform unlimited activation functions
  • transforming the neural network topology includes applying one or more transformations selected from the group consisting of: (i) replacing the unlimited activation functions with limited activation; and (ii) adjusting connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized.
  • the method further includes generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix.
  • the method further includes: (i) obtaining new weights for the trained neural network; (ii) computing a new weight matrix for the equivalent analog network based on the new weights; (iii) generating a new resistance matrix for the new weight matrix; and (iv) generating a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix.
  • the trained neural network is trained using software simulations to generate the weights.
  • a method for hardware realization of neural networks is provided, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network.
  • the method also includes calculating one or more connection constraints based on analog integrated circuit (IC) design constraints.
  • the method also includes transforming the neural network topology to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints.
  • the method also includes computing a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes deriving a possible input connection degree and output connection degree ⁇ ⁇ , according to the one or more connection constraints.
  • the neural network topology includes at least one densely connected layer with ⁇ inputs and ⁇ outputs and a weight matrix ⁇ .
  • transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with ⁇ inputs, ⁇ outputs, and ⁇ ⁇ 1 layers, such that input connection degree does not exceed ⁇ ⁇ , and output connection degree does not exceed ⁇ ⁇ .
  • the neural network topology includes at least one densely connected layer with ⁇ inputs and ⁇ outputs and a weight matrix ⁇ .
  • transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with ⁇ inputs, ⁇ outputs, and ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ layers.
  • Each layer m is represented by a corresponding weight matrix ⁇ ⁇ , where absent connections are represented with zeros, such that input connection degree does not exceed ⁇ ⁇ , and output connection degree does not exceed ⁇ ⁇ .
  • the equation ⁇ ⁇ ⁇ is satisfied with a predetermined precision.
  • the neural network topology includes a single sparsely connected layer with ⁇ inputs and ⁇ outputs, a maximum input connection degree of ⁇ ⁇ , a maximum output connection degree of ⁇ ⁇ , and a weight matrix of ⁇ , where absent connections are represented with zeros.
  • transforming the single sparsely connected layer includes constructing the equivalent sparsely connected network with ⁇ inputs, ⁇ outputs, ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ layers, each layer ⁇ represented by a corresponding weight matrix ⁇ ⁇ , where absent connections are represented with zeros, such that input connection degree does not exceed ⁇ ⁇ , and output connection degree does not exceed ⁇ ⁇ .
  • the neural network topology includes a convolutional layer with ⁇ inputs and ⁇ outputs.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the convolutional layer into a single sparsely connected layer with ⁇ inputs, ⁇ outputs, a maximum input connection degree of ⁇ ⁇ , and a maximum output connection degree of ⁇ ⁇ .
  • the neural network topology includes a recurrent neural layer.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes transforming the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections.
  • the neural network topology includes a recurrent neural layer.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with ⁇ inputs and ⁇ output and a weight matrix ⁇ , where absent connections are represented with zeros.
  • the neural network topology includes K inputs, a weight vector ⁇ ⁇ ⁇ ⁇ , and a single layer perceptron with a calculation neuron with an activation function F.
  • the equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes calculating a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U.
  • the neural network topology includes K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network of the L single layer perceptron networks: (a) constructing a respective equivalent pyramid-like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (b) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs.
  • the neural network topology includes K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons Li and corresponding weight matrices V i that includes a row of weights for each calculation neuron of the L i calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the ⁇ calculation neurons.
  • K i, j is number of inputs for the respective calculation neuron in the multi-layer perceptron; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K i, j inputs, the m layers and the connection degree N.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*Ki, j inputs.
  • the neural network topology includes a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons L i and corresponding weight matrices V i that includes a row of weights for each calculation neuron of the Li calculation neurons.
  • CNN Convolutional Neural Network
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*Ki, j inputs.
  • the sparse weight matrix ⁇ ⁇ ⁇ ⁇ ⁇ represents connections between the layers LAp and LAh
  • the sparse weight matrix ⁇ ⁇ ⁇ ⁇ ⁇ represents connections between the layers LA h and LA o ,.
  • performing the trapezium transformation further includes: in accordance with a determination that ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ + ⁇ ⁇ ⁇ ⁇ : (i) splitting the layer Lp to obtain a sub-layer Lp1 with K’ neurons and a sub-layer Lp2 with (K - K’) neurons such that ⁇ ′ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ + ⁇ ′ ⁇ ⁇ ⁇ ; (ii) for the sub-layer Lp1 with K’ neurons, performing the constructing, and generating steps; and (iii) for the sub-layer Lp2 with K - K’ neurons, recursively performing the splitting, constructing, and generating steps.
  • the neural network topology includes a multilayer perceptron network. In such cases, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the neural network topology includes a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non-linear function calculation.
  • RNN recurrent neural network
  • the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation.
  • the neural network topology includes a long short- term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations.
  • LSTM long short- term memory
  • GRU gated recurrent unit
  • the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations.
  • the neural network topology includes a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers and (ii) one or more fully-connected layers.
  • CNN convolutional neural network
  • the method further includes: (i) transforming the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and (ii) for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the neural network topology includes K inputs, L output neurons, and a weight matrix ⁇ ⁇ ⁇ ⁇ , where R is the set of real numbers, each output neuron performs an activation function F.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer ⁇ of the ⁇ layers of the multilayer perceptron: (i) constructing a respective pyramid-trapezium network PTNNX j by performing the approximation transformation to a respective single layer perceptron consisting of ⁇ ⁇ inputs, ⁇ ⁇ output neurons, and a weight matrix ⁇ ⁇ ; and (ii) constructing the equivalent sparsely connected network by stacking each pyramid trapezium network.
  • a method is provided for hardware realization of neural networks, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network.
  • the method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • the method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.
  • the method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • w is the respective element of the weight matrix
  • ⁇ ⁇ is a predetermined relative tolerance value for resistances.
  • the predetermined range of possible resistance values includes resistances according to nominal series E24 in the range 100 K ⁇ to 1 M ⁇ .
  • ⁇ ⁇ and ⁇ ⁇ are chosen independently for each layer of the equivalent analog network.
  • ⁇ ⁇ and ⁇ ⁇ are chosen independently for each analog neuron of the equivalent analog network.
  • a first one or more weights of the weight matrix and a first one or more inputs represent one or more connections to a first operational amplifier of the equivalent analog network.
  • the method further includes, prior to generating the resistance matrix: (i) modifying the first one or more weights by a first value; and (ii) configuring the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function.
  • the method further includes: (i) obtaining a predetermined range of weights; and (ii) updating the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input.
  • the trained neural network is trained so that each layer of the neural network topology has quantized weights.
  • the method further includes retraining the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network. [0059] In some implementations, the method further includes retraining the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. [0060] In another aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors.
  • Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • the method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.
  • the method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix.
  • the method also includes pruning the equivalent analog network to reduce number of the plurality of operational amplifiers or the plurality of resistors, based on the resistance matrix, to obtain an optimized analog network of analog components.
  • pruning the equivalent analog network includes substituting, with conductors, resistors corresponding to one or more elements of the resistance matrix that have resistance values below a predetermined minimum threshold resistance value. [0062] In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value. [0063] In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the weight matrix that are approximately zero. [0064] In some implementations, pruning the equivalent analog network further includes removing one or more analog neurons of the equivalent analog network without any input connections.
  • pruning the equivalent analog network includes: (i) ranking analog neurons of the equivalent analog network based on detecting use of the analog neurons when making calculations for one or more data sets; (ii) selecting one or more analog neurons of the equivalent analog network based on the ranking; and (iii) removing the one or more analog neurons from the equivalent analog network.
  • detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring propagation of analog signals by using the model to generate calculations for the one or more data sets.
  • detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring output signals of the model by using the model to generate calculations for the one or more data sets. [0068] In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring power consumed by the analog neurons by using the model to generate calculations for the one or more data sets.
  • the method further includes subsequent to pruning the equivalent analog network, and prior to generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network, recomputing the weight matrix for the equivalent analog network and updating the resistance matrix based on the recomputed weight matrix.
  • the method further includes, for each analog neuron of the equivalent analog network: (i) computing a respective bias value for the respective analog neuron based on the weights of the trained neural network, while computing the weight matrix; (ii) in accordance with a determination that the respective bias value is above a predetermined maximum bias threshold, removing the respective analog neuron from the equivalent analog network; and (iii) in accordance with a determination that the respective bias value is below a predetermined minimum bias threshold, replacing the respective analog neuron with a linear junction in the equivalent analog network.
  • the method further includes reducing number of neurons of the equivalent analog network, prior to generating the weight matrix, by increasing number of connections from one or more analog neurons of the equivalent analog network.
  • the method further includes pruning the trained neural network to update the neural network topology and the weights of the trained neural network, prior to transforming the neural network topology, using pruning techniques for neural networks, so that the equivalent analog network includes less than a predetermined number of analog components.
  • the pruning is performed iteratively taking into account accuracy or a level of match in output between the trained neural network and the equivalent analog network.
  • the method further includes, prior to transforming the neural network topology to the equivalent analog network, performing network knowledge extraction.
  • an integrated circuit is provided, according to some implementations.
  • the integrated circuit includes an analog network of analog components fabricated by a method that includes: (i) obtaining a neural network topology and weights of a trained neural network; (ii) transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron; (iii) computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection; (iv) generating a resistance matrix for the weight matrix.
  • the integrated circuit further includes one or more digital to analog converters configured to generate analog input for the equivalent analog network of analog components based on one or more digital.
  • the integrated circuit further includes an analog signal sampling module configured to process 1-dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit.
  • the integrated circuit further includes a voltage converter module to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers.
  • the integrated circuit further includes a tact signal processing module configured to process one or more frames obtained from a CCD camera.
  • the trained neural network is a long short-term memory (LSTM) network.
  • the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing.
  • the integrated circuit further includes one or more analog to digital converters configured to generate digital signal based on output of the equivalent analog network of analog components.
  • the integrated circuit further includes one or more signal processing modules configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications.
  • the trained neural network is trained, using training datasets containing signals of arrays of gas sensors on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected.
  • the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections.
  • the trained neural network is trained, using training datasets containing thermal aging time series data for different MOSFETs, for predicting remaining useful life (RUL) of a MOSFET device.
  • the neural network topology includes 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons, and (v) between 123,000 and 124,000 connections.
  • the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons, and (v) between 51,000 and 52,000 connections.
  • the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons, and (v) between 2,200 and 2,400 connections.
  • the trained neural network is trained, using training datasets containing speech commands, for identifying voice commands.
  • the neural network topology is a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron.
  • DS-CNN Depthwise Separable Convolutional Neural Network
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections.
  • the trained neural network is trained, using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor, for determining pulse rate during physical exercises based on PPG sensor data and 3- axis accelerometer data.
  • PPG photoplethysmography
  • the neural network topology includes two Conv1D layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively.
  • the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons, and (vi) between 12,000 and 12,500 connections.
  • the trained neural network is trained to classify different objects based on pulsed Doppler radar signal.
  • the neural network topology includes multi-scale LSTM neural network.
  • the trained neural network is trained to perform human activity type recognition, based on inertial sensor data.
  • the neural network topology includes three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes.
  • the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons, and (vi) between 20,000 and 21,000 connections.
  • the trained neural network is further trained to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation.
  • a method is provided for generating libraries for hardware realization of neural networks. The method includes obtaining a plurality of neural network topologies, each neural network topology corresponding to a respective neural network.
  • the method also includes transforming each neural network topology to a respective equivalent analog network of analog components.
  • the method also includes generating a plurality of lithographic masks for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components.
  • the method further includes obtaining a new neural network topology and weights of a trained neural network.
  • the method also includes selecting one or more lithographic masks from the plurality of lithographic masks based on comparing the new neural network topology to the plurality of neural network topologies.
  • the method also includes computing a weight matrix for a new equivalent analog network based on the weights.
  • the method also includes generating a resistance matrix for the weight matrix.
  • the method also includes generating a new lithographic mask for fabricating a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithographic masks.
  • the new neural network topology includes a plurality of subnetwork topologies, and selecting the one or more lithographic masks is further based on comparing each subnetwork topology with each network topology of the plurality of network topologies.
  • one or more subnetwork topologies of the plurality of subnetwork topologies fails to compare with any network topology of the plurality of network topologies.
  • the method further includes: (i) transforming each subnetwork topology of the one or more subnetwork topologies to a respective equivalent analog subnetwork of analog components; and (ii) generating one or more lithographic masks for fabricating one or more circuits, each circuit of the one or more circuits implementing a respective equivalent analog subnetwork of analog components.
  • transforming a respective network topology to a respective equivalent analog network includes: (i) decomposing the respective network topology to a plurality of subnetwork topologies; (ii) transforming each subnetwork topology to a respective equivalent analog subnetwork of analog components; and (iii) composing each equivalent analog subnetwork to obtain the respective equivalent analog network.
  • decomposing the respective network topology includes identifying one or more layers of the respective network topology as the plurality of subnetwork topologies.
  • each circuit is obtained by: (i) generating schematics for a respective equivalent analog network of analog components; and (ii) generating a respective circuit layout design based on the schematics.
  • the method further includes combining one or more circuit layout designs prior to generating the plurality of lithographic masks for fabricating the plurality of circuits.
  • a method is provided for optimizing energy efficiency of analog neuromorphic circuits, according to some implementations.
  • the method includes obtaining an integrated circuit implementing an analog network of analog components including a plurality of operational amplifiers and a plurality of resistors.
  • the analog network represents a trained neural network
  • each operational amplifier represents a respective analog neuron
  • each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron.
  • the method also include generating inferences using the integrated circuit for a plurality of test inputs, including simultaneously transferring signals from one layer to a subsequent layer of the analog network.
  • the method also includes, while generating inferences using the integrated circuit: (i) determining if a level of signal output of the plurality of operational amplifiers is equilibrated; and (ii) in accordance with a determination that the level of signal output is equilibrated: (a) determining an active set of analog neurons of the analog network influencing signal formation for propagation of signals; and (turning off power for one or more analog neurons of the analog network, distinct from the active set of analog neurons, for a predetermined period of time. [00101] In some implementations, determining the active set of analog neurons is based on calculating delays of signal propagation through the analog network.
  • determining the active set of analog neurons is based on detecting the propagation of signals through the analog network.
  • the trained neural network is a feed-forward neural network, and the active set of analog neurons belong to an active layer of the analog network, and turning off power includes turning off power for one or more layers prior to the active layer of the analog network.
  • the predetermined period of time is calculated based on simulating propagation of signals through the analog network, accounting for signal delays.
  • the trained neural network is a recurrent neural network (RNN), and the analog network further includes one or more analog components other than the plurality of operational amplifiers, and the plurality of resistors.
  • the method further includes, in accordance with a determination that the level of signal output is equilibrated, turning off power, for the one or more analog components, for the predetermined period of time.
  • the method further includes turning on power for the one or more analog neurons of the analog network after the predetermined period of time.
  • determining if the level of signal output of the plurality of operational amplifiers is equilibrated is based on detecting if one or more operational amplifiers of the analog network is outputting more than a predetermined threshold signal level.
  • the method further includes repeating the turning off for the predetermined period of time and turning on the active set of analog neurons for the predetermined period of time, while generating the inferences.
  • the method further includes: (i) in accordance with a determination that the level of signal output is equilibrated, for each inference cycle: (a) during a first time interval, determining a first layer of analog neurons of the analog network influencing signal formation for propagation of signals; and (b) turning off power for a first one or more analog neurons of the analog network, prior to the first layer, for the predetermined period of time; and (ii) during a second time interval subsequent to the first time interval, turning off power for a second one or more analog neurons including the first layer of analog neurons and the first one or more analog neurons of the analog network, for the predetermined period.
  • the one or more analog neurons consist of analog neurons of a first one or more layers of the analog network, and the active set of analog neurons consist of analog neurons of a second layer of the analog network, and the second layer of the analog network is distinct from layers of the first one or more layers.
  • a method for recognizing human activities. The method includes using one or more sensors to track activity of a user, including obtaining a plurality of electrical signals from the one or more sensors. The method also includes forming a feature vector by extracting a plurality of features from the plurality of electrical signals. The features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities.
  • the method also includes applying an analog neurocomputing hardware device to the feature vector to generate an embedding vector that specifies a descriptor.
  • the analog neurocomputing hardware device implements the trained neural network model.
  • the method also includes applying a trained machine learning classifier to the embedding vector to classify the activity of the user as one of the predefined human activities.
  • the trained neural network model is an autoencoder that includes an encoder and a decoder.
  • the trained machine learning classifier is a KNN (K-Nearest Neighbors) classifier. In some implementations, a number of neighbors for the KNN classifier equals five.
  • the trained machine learning classifier is trained separately for each of the predefined human activities using binary classification.
  • the one or more sensors include one or more of IMUs, cameras, microphones, and biofeedback devices.
  • the method further includes smoothing an output of the trained machine learning classifier to obtain a basic class of activity.
  • the analog neurocomputing hardware device is fabricated by steps including: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model, wherein each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating an integrated circuit, according to the schematic model, using a lithographic process.
  • generating the schematic model includes generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the trained machine learning classifier is implemented using one or more digital components and the trained machine learning classifier can be retrained for new users.
  • a method is provided for recognizing human activities. The method includes obtaining a sequence of electrical signals from one or more sensors that track activity of a user. The method also includes forming a plurality of feature vectors by extracting features from the sequence of electrical signals.
  • the features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities.
  • the method also includes applying the analog neurocomputing hardware device to the plurality of feature vectors to generate a plurality of embedding vectors that each specify a corresponding descriptor.
  • the method also includes using the plurality of embedding vectors for classifying the activity of the user as one of the predefined human activities.
  • the method further includes: receiving, from the user, a set of descriptors that describes specific physical activities; and using the set of descriptors and the plurality of embedding vectors to classify the activity of the user as one of the specific physical activities.
  • the method further includes generating statistics of personal daily routines of the user based on classifying the activity of the user as one of the specific physical activities.
  • the method further includes: storing, for the user, the plurality of embedding vectors as describing a specific activity; and using the plurality of embedding vectors for classifying subsequent activities of the user as the specific activity.
  • the method further includes: receiving, from a trainer distinct from the user, a set of descriptors that describes a specific activity; and providing feedback to the user if the activity matches the specific activity based on the plurality of embedding vectors and the set of descriptors.
  • a human activity recognition device is provided.
  • the device includes an integrated circuit for human activity recognition.
  • the integrated circuit includes an analog network of analog components configured to implement a trained neural network model that is trained to generate a plurality of descriptors for a plurality of predefined human activities based on a plurality of features extracted from a plurality of electrical signals from one or more sensors.
  • the device also includes one or more digital components configured to classify human activity as one of the plurality of predefined human activities according to the plurality of descriptors generated by the integrated circuit.
  • the human activity recognition device further includes the one or more sensors configured to collect the plurality of electrical signals during the human activity.
  • the trained neural network model is an autoencoder that includes an encoder and a decoder.
  • the one or more digital components implement a trained machine learning classifier that is a KNN (K-Nearest Neighbors) classifier which can be retrained.
  • KNN K-Nearest Neighbors
  • a number of neighbors for the KNN classifier equals five.
  • the trained machine learning classifier is trained separately for each of the plurality of predefined human activities using binary classification.
  • the one or more digital components are further configured to smooth the output of the trained machine learning classifier to obtain a basic class of activity.
  • the one or more sensors include one or more of IMUs, cameras, microphones, and biofeedback devices.
  • the integrated circuit is fabricated by steps comprising: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model.
  • Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating the integrated circuit, according to the schematic model, using a lithographic process.
  • generating the schematic model includes generating a resistance matrix for the weight matrix.
  • the system may include a hardware apparatus.
  • the hardware apparatus may include a vibration sensor configured to sense vibrations of a vibration source and an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors.
  • the analog circuit may be coupled to the vibration sensor, and configured to (i) receive an analog signal from the vibration sensor and (ii) compute an output based on the analog signal, by performing a portion of a trained neural network.
  • the hardware apparatus further includes a transceiver coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN).
  • LPWAN low power wide area network
  • the vibration sensor is disposed adjacent to a movable part (e.g., a rotating or reciprocating part) of a machine (sometimes called a vibration source), and the vibration sensor is configured to collect vibration signals from the rotating or reciprocating part.
  • the rotating part includes a ball bearing of the machine.
  • the output of the analog circuit represents embeddings used for at least one of: defining a source of vibration, predicting failures of a machine coupled to the vibration source, and generating suggestions for maintenance of the machine.
  • the vibration sensor is disposed in or on a tire and is configured to collect vibration data for the tire, which is an example of a vibration source.
  • the output represents embeddings used to predict at least one of: a road surface, a physical condition, a tire condition, a suspension condition, or a time-to-failure of the vibration source.
  • the vibration sensor is configured to sample signals in a range of 0 to 20 kilohertz (kHz). [00139] In some implementations, the vibration sensor is configured to sample signals up to 41 kHz. [00140] In some implementations, the vibration sensor is configured to sample signals below a Nyquist sampling rate for fault detection and classification in a compressed domain. [00141] In some implementations, the vibration sensor is configured to sample signals for compressed sensing (CS) for condition classification of rolling element bearings in rotating machines. [00142] In some implementations, the trained neural network comprises a plurality of layers of neurons including a first set of layers and a second set of layers and the analog circuit is configured to implement the first set of layers.
  • the second set of layers consists of a last layer of the trained neural network.
  • the trained neural network includes a deep neural network for unsupervised learning based on a sparse autoencoder.
  • the trained neural network comprises a ResNet Convolutional Neural Network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings.
  • the vibration sensor is configured to sample and output a one-dimensional time domain signal of a rolling bearing fault signal.
  • the trained neural network comprises a stacked noise reduction autoencoder.
  • the analog circuit is configured to be powered by vibrations of the vibration source.
  • the hardware apparatus further includes a power harvesting circuit configured to harvest power from vibrations of the vibration source and supply power to the analog circuit.
  • the plurality of operational amplifiers is configured to implement neurons of the portion of the trained neural network
  • the plurality of resistors is configured to implement axons or connections between neurons of the portion of the trained neural network.
  • the analog circuit is configured to implement an optimized neural network corresponding to the trained neural network.
  • values of the plurality of resistors are based on weights of connections of the trained neural network.
  • the plurality of resistors is configured to connect the plurality of operational amplifiers.
  • the analog circuit comprises resistors in a backend- of-the-line (BEOL).
  • the trained neural network is an autoencoder comprising an encoder portion and a decoder portion.
  • the encoder portion reconstructs an input vector at an output layer after nonlinear transformations performed by hidden layers.
  • the analog circuit corresponds to the encoder portion of the autoencoder.
  • the encoder portion comprises the hidden layers.
  • the analog circuit is configured to compute a representation of the input vector in fewer dimensions than an input space of the input vector.
  • the analog circuit is configured to generate compressed data that encodes vibration sensor data based on vibration features from the vibration sensor.
  • a system in another aspect, includes a hardware apparatus, a transceiver and a digital circuit.
  • the hardware apparatus includes a vibration sensor configured to sense vibrations of the vibration source of a machine.
  • the hardware apparatus also includes an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors.
  • the analog circuit is coupled to the vibration sensor and is configured to: receive an analog signal from the vibration sensor; and compute an output based on the analog signal, by performing a portion of a trained neural network.
  • the transceiver is coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN). The analog signal is converted before transmitting.
  • LPWAN low power wide area network
  • An analog-to-digital converter may be included in the hardware apparatus, or may be included as part of the transceiver.
  • the digital circuit is communicatively coupled to the transceiver of the hardware apparatus via the LPWAN.
  • the digital circuit is configured to receive the output from the analog circuit (via the transceiver) and predict the state of the machine for maintenance, based on the output.
  • the digital circuit includes one or more digital computing units selected from the group consisting of: CPUs, GPUs, RISCs, FPGAs, and ASICs.
  • the digital circuit comprises a processor configured to perform data classification.
  • the data classification is performed by a neural network that is distinct from the trained neural network.
  • the data classification is performed using k-nearest neighbors (k-NN).
  • the output of the analog circuit represents embeddings and the digital circuit is configured to use the embeddings to classify the analog signal.
  • a method is provided for vibration sensing. The method includes sensing vibrations of a vibration source using a vibration sensor to obtain an analog signal. The method also includes computing an output based on the analog signal, by performing a portion of a trained neural network, using an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors. The method also includes transmitting the output over a low power wide area network (LPWAN) using a transceiver.
  • LPWAN low power wide area network
  • a computer system has one or more processors, memory, and a display.
  • the one or more programs include instructions for performing any of the methods described herein.
  • a non-transitory computer readable storage medium stores one or more programs configured for execution by a computer system having one or more processors, memory, and a display.
  • the one or more programs include instructions for performing any of the methods described herein.
  • Figure 1A is a block diagram of a system for hardware realization of trained neural networks using analog components, according to some implementations.
  • Figure 1B is a block diagram of an alternative representation of the system of Figure 1A for hardware realization of trained neural networks using analog components, according to some implementations.
  • Figure 1C is a block diagram of another representation of the system of Figure 1A for hardware realization of trained neural networks using analog components, according to some implementations.
  • Figure 2A is a system diagram of a computing device in accordance with some implementations.
  • Figure 2B shows optional modules of the computing device, according to some implementations.
  • Figure 3A shows an example process for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations.
  • Figure 3B shows an example manual prototyping process used for generating a target chip model, according to some implementations.
  • Figures 4A, 4B, and 4C show examples of neural networks that are transformed to mathematically equivalent analog networks, according to some implementations.
  • Figure 5 shows an example of a math model for a neuron, according to some implementations.
  • Figures 6A-6C illustrate an example process for analog hardware realization of a neural network for computing an XOR of input values, according to some implementations.
  • Figure 7 shows an example perceptron, according to some implementations.
  • Figure 8 shows an example Pyramid-Neural Network, according to some implementations.
  • Figure 9 shows an example Pyramid Single Neural Network, according to some implementations.
  • Figure 10 shows an example of a transformed neural network, according to some implementations.
  • Figures 11A-11C show an application of a T-transformation algorithm for a single layer neural network, according to some implementations.
  • Figure 12 shows an example Recurrent Neural Network (RNN), according to some implementations.
  • RNN Recurrent Neural Network
  • Figure 13A is a block diagram of a LSTM neuron, according to some implementations.
  • Figure 13B shows delay blocks, according to some implementations.
  • Figure 13C is a neuron schema for a LSTM neuron, according to some implementations.
  • Figure 14A is a block diagram of a GRU neuron, according to some implementations.
  • Figure 14B is a neuron schema for a GRU neuron, according to some implementations.
  • Figures 15A and 15B are neuron schema of variants of a single Conv1D filter, according to some implementations.
  • Figure 16 shows an example architecture of a transformed neural network, according to some implementations.
  • Figures 17A – 17C provide example charts illustrating dependency between output error and classification error or weight error, according to some implementations.
  • Figure 18 provides an example scheme of a neuron model used for resistors quantization, according to some implementations.
  • Figure 19A shows a schematic diagram of an operational amplifier made on CMOS, according to some implementations.
  • Figure 19B shows a table of description for the example circuit shown in Figure 19A, according to some implementations.
  • Figures 20A-20E show a schematic diagram of a LSTM block, according to some implementations.
  • Figure 20F shows a table of description for the example circuit shown in Figure 20A-20D, according to some implementations.
  • Figures 21A-21I show a schematic diagram of a multiplier block, according to some implementations.
  • Figure 21J shows a table of description for the schematic shown in Figures 21A-21I, according to some implementations.
  • Figure 22A shows a schematic diagram of a sigmoid neuron, according to some implementations.
  • Figure 22B shows a table of description for the schematic diagram shown in Figure 22A, according to some implementations.
  • Figure 23A shows a schematic diagram of a hyperbolic tangent function block, according to some implementations.
  • Figure 23B shows a table of description for the schematic diagram shown in Figure 23A, according to some implementations.
  • Figures 24A-24C show a schematic diagram of a single neuron CMOS operational amplifier, according to some implementations.
  • Figure 24D shows a table of description for the schematic diagram shown in Figure 24A-24C, according to some implementations.
  • Figures 25A-25D show a schematic diagram of a variant of a single neuron CMOS operational amplifiers according to some implementations.
  • Figure 25E shows a table of description for the schematic diagram shown in Figure 25A-25D, according to some implementations.
  • Figures 26A-26K show example weight distribution histograms, according to some implementations.
  • Figures 27A-27J show a flowchart of a method for hardware realization of neural networks, according to some implementations.
  • Figures 28A-28S show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations.
  • Figures 29A-29F show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations.
  • Figures 30A-30M show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations.
  • Figures 31A-31Q show a flowchart of a method for fabricating an integrated circuit that includes an analog network of analog components, according to some implementations.
  • Figures 32A-32E show a flowchart of a method for generating libraries for hardware realization of neural networks, according to some implementations.
  • Figures 33A-33K show a flowchart of a method for optimizing energy efficiency of analog neuromorphic circuits (that model trained neural networks), according to some implementations.
  • Figure 34 shows a table describing the MobileNet v1 architecture, according to some implementations.
  • Figures 35 shows an example 1-D convolutional neural network used for voice clarification, according to some implementations.
  • Figure 36 shows an example T-transformation of fully connected layers of the neural network shown in Figure 35, according to some implementations.
  • Figure 37 shows an example T-transformation of 1-D convolution of the neural network shown in Figure 35, according to some implementations.
  • Figure 38 shows an example T-transformation of a MaxPooling operator of the neural network shown in Figure 35, according to some implementations.
  • Figure 39 shows an example of splitting a dataset to a training dataset and a validation dataset, according to some implementations.
  • Figure 40 shows two graph plots of activity labels, according to some implementations.
  • Figures 41A and 41B show an example graph plot for splitting experiments to training or testing test data, according to some implementations.
  • Figures 42A, 42B, and 42C show schematic diagrams of an example encoder, an example decoder, and an example classifier, respectively, according to some implementations.
  • Figures 43A-43F show graphs of results, according to some implementations.
  • Figures 44A-44D show graphs of results, according to some implementations.
  • Figures 45A and 45B show a table that presents predictions for a base class activity classifier model, according to some implementations.
  • Figures 46A-46F show graphs of results, according to some implementations.
  • Figures 47A-47F show graphs of results, according to some implementations.
  • Figures 49A-49D show graphs of results, according to some implementations.
  • Figure 48 shows a visualization for t-distributed stochastic neighbor embedding (t-SNE) by exercise sets for embedding different approaches of the crossfit_situp exercise in experiment shown in Figures 47E and 47F, according to some implementations.
  • Figure 50 shows an example human activity recognition device, according to some implementations.
  • Figure 51 shows a flowchart of an example method for recognizing human activities, according to some implementations.
  • Figure 52 shows a flowchart of another example method for recognizing human activities, according to some implementations.
  • Figure 53 shows an example sparse autoencoder for bearing fault classification, according to some implementations.
  • Figure 54 shows an example deep ResNet framework with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings, according to some implementations.
  • GAP global average pooling
  • Figure 55 shows a flowchart for an example method for deep autoencoder training for bearing fault classification, according to some implementations.
  • Figure 56 shows an example system for predictive maintenance of a machine, according to some implementations.
  • FIG. 1A is a block diagram of a system 100 for hardware realization of trained neural networks using analog components, according to some implementations.
  • the system includes transforming (126) trained neural networks 102 to analog neural networks 104.
  • analog integrated circuit constraints 184 constrain (146) the transformation (126) to generate the analog neural networks 104.
  • the system derives (calculates or generates) weights 106 for the analog neural networks 104 by a process that is sometimes called weight quantization (128).
  • the analog neural network includes a plurality of analog neuron, each analog neuron represented by an analog component, such as an operational amplifier, and each analog neuron connected to another analog neuron via a connection.
  • the connections are represented using resistors that reduce the current flow between two analog neurons.
  • the system transforms (148) the weights 106 to resistance values 112 for the connections.
  • the system subsequently generates (130) one or more schematic models 108 for implementing the analog neural networks 104 based on the weights 106.
  • the system optimizes resistance values 112 (or the weights 106) to form optimized analog neural networks 114 which is further used to generate (150) the schematic models 108.
  • the system generates (132) lithographic masks 110 for the connections and/or generates (136) lithographic masks 120 for the analog neurons.
  • the system fabricates (134 and/or 138) analog integrated circuits 118 that implement the analog neural networks 104.
  • the system generates (152) libraries of lithographic masks 116 based on the lithographic masks for connections 110 and/or lithographic masks 120 for the analog neurons.
  • the system uses (154) the libraries of lithographic masks 116 to fabricate the analog integrated circuits 118.
  • the system regenerates (or recalculates) (144) the resistance values 112 (and/or the weights 106), the schematic model 108, and/or the lithographic masks for connections 110.
  • the system reuses the lithographic masks 120 for the analog neurons 120. In other words, in some implementations, only the weights 106 (or the resistance values 112 corresponding to the changed weights), and/or the lithographic masks for the connections 110 are regenerated.
  • FIG. 1B is a block diagram of an alternative representation of the system 100 for hardware realization of trained neural networks using analog components, according to some implementations.
  • FIG. 1C is a block diagram of another representation of the system 100 for hardware realization of trained neural networks using analog components, according to some implementations.
  • the system is distributed as a software development kit (SDK) 180, according to some implementations.
  • SDK software development kit
  • a user develops and trains (164) a neural network and inputs the trained neural net 166 to the SDK 180.
  • the SDK estimates (168) complexity of the trained neural net 166.
  • the SDK 180 prunes (178) the trained neural net and retrains (182) the neural net to obtain an updated trained neural net 166.
  • the SDK 180 transforms (170) the trained neural net 166 into a sparse network of analog components (e.g., a pyramid- or a trapezia-shaped network).
  • the SDK 180 also generates a circuit model 172 of the analog network.
  • the SDK estimates (176) a deviation in an output generated by the circuit model 172 relative to the trained neural network for a same input, using software simulations.
  • the SDK 180 prompts the user to reconfigure, redevelop, and/or retrain the neural network.
  • a threshold error e.g., a value set by the user
  • the SDK automatically reconfigures the trained neural net 166 so as to reduce the estimated error. This process is iterated multiple times until the error is reduced below the threshold error.
  • the dashed line from the block 176 (“Estimation of error raised in circuitry”) to the block 164 (“Development and training of neural network”) indicates a feedback loop.
  • FIG. 1 is a system diagram of a computing device 200 in accordance with some implementations.
  • the term “computing device” includes both personal devices 102 and servers.
  • a computing device 200 typically includes one or more processing units/cores (CPUs) 202 for executing modules, programs, and/or instructions stored in the memory 214 and thereby performing processing operations; one or more network or other communications interfaces 204; memory 214; and one or more communication buses 212 for interconnecting these components.
  • the communication buses 212 may include circuitry that interconnects and controls communications between system components.
  • a computing device 200 may include a user interface 206 comprising a display device 208 and one or more input devices or mechanisms 210.
  • the input device/mechanism 210 includes a keyboard; in some implementations, the input device/mechanism includes a “soft” keyboard, which is displayed as needed on the display device 208, enabling a user to “press keys” that appear on the display 208.
  • the display 208 and input device / mechanism 210 comprise a touch screen display (also called a touch sensitive display).
  • the memory 214 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices.
  • the memory 214 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.
  • the memory 214 includes one or more storage devices remotely located from the CPU(s) 202.
  • the memory 214 or alternatively the non-volatile memory device(s) within the memory 214, comprises a computer readable storage medium.
  • the memory 214, or the computer readable storage medium of the memory 214 stores the following programs, modules, and data structures, or a subset thereof: • an operating system 216, which includes procedures for handling various basic system services and for performing hardware dependent tasks; • a communications module 218, which is used for connecting the computing device 200 to other computers and devices via the one or more communication network interfaces 204 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on; • trained neural networks 220 that includes weights 222 and neural network topologies 224.
  • a neural network transformation module 226 that includes transformed analog neural networks 228, mathematical formulations 230, the basic function blocks 232, analog models 234 (sometimes called neuron models), and/or analog integrated circuit (IC) design constraints 236.
  • Example operations of the neural network transformation module 226 are described below in reference to at least Figures 5, 6A-6C, 7, 8, 9, 10, and 11A-11C, and the flowcharts shown in Figures 27A-27J, and Figures 28A-28S; and/or • a weight matrix computation (sometimes called a weight quantization) module 238 that includes weights 272 of transformed networks, and optionally includes resistance calculation module 240, resistance values 242.
  • a weight matrix computation sometimes called a weight quantization
  • Example operations of the weight matrix computation module 238 and/or weight quantization are described in reference to at least Figures 17A-17C, Figure 18, and Figures 29A- 29F, according to some implementations.
  • Some implementations include one or more optional modules 244 as shown in Figure 2B.
  • Some implementations include an analog neural network optimization module 246.
  • Some implementations include a lithographic mask generation module 248 that further includes lithographic masks 250 for resistances (corresponding to connections), and/or lithographic masks for analog components (e.g., operational amplifiers, multipliers, delay blocks, etc.) other than the resistances (or connections).
  • lithographic masks are generated based on chip design layout following chip design using Cadence, Synopsys, or Mentor Graphics software packages. Some implementations use a design kit from a silicon wafer manufacturing plant (sometimes called a fab).
  • Lithographic masks are intended to be used in that particular fab that provides the design kit (e.g., TSMC 65 nm design kit).
  • the lithographic mask files that are generated are used to fabricate the chip at the fab.
  • the Cadence, Mentor Graphics, or Synopsys software packages-based chip design is generated semi-automatically from the SPICE or Fast SPICE (Mentor Graphics) software packages.
  • a user with chip design skill drives the conversion from the SPICE or Fast SPICE circuit into Cadence, Mentor Graphics or Synopsis chip design.
  • Cadence design blocks for single neuron unit, establishing proper interconnects between the blocks.
  • Some implementations include a library generation module 254 that further includes libraries of lithographic masks 256. Examples of library generation are described below in reference to Figures 32A-32E, according to some implementations.
  • Some implementations include Integrated Circuit (IC) fabrication module 258 that further includes Analog-to-Digital Conversion (ADC), Digital-to-Analog Conversion (DAC), or similar other interfaces 260, and/or fabricated ICs or models 262. Example integrated circuits and/or related modules are described below in reference to Figures 31A- 31Q, according to some implementations.
  • Some implementations include an energy efficiency optimization module 264 that further includes an inferencing module 266, a signal monitoring module 268, and/or a power optimization module 270.
  • Each of the above identified executable modules, applications, or sets of procedures may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • the memory 214 stores a subset of the modules and data structures identified above.
  • the memory 214 stores additional modules or data structures not described above.
  • Figure 2A shows a computing device 200
  • Figure 2A is intended more as a functional description of the various features that may be present rather than as a structural schematic of the implementations described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated.
  • Example Process for Generating Schematic Models of Analog Networks [00236]
  • Figure 3A shows an example process 300 for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations.
  • a trained neural network 302 e.g., MobileNet
  • T-transformation a process that is sometimes called T-transformation
  • the target neural network (sometimes called a T-network) 304 is exported (324) to SPICE (as a SPICE model 306) using a single neuron model (SNM), which is exported (326) from SPICE to CADENCE and full on-chip designs using a CADENCE model 308.
  • the CADENCE model 308 is cross-validated (328) against the initial neural network for one or more validation inputs.
  • a math neuron is a mathematical function which receives one or more weighted inputs and produces a scalar output.
  • a math neuron can have memory (e.g., long short-term memory (LSTM), recurrent neuron).
  • a SNM is a schematic model with analog components (e.g., operational amplifiers, resistors R 1 , ..., R n , and other components) representing a specific type of math neuron (for example, trivial neuron) in schematic form.
  • SNM formula is equivalent to math neuron formula, with a desired weights set.
  • the weights set is fully determined by resistors used in a SNM.
  • a target (analog) neural network 304 (sometimes called a T-network) is a set of math neurons which have defined SNM representation, and weighted connections between them, forming a neural network.
  • a T-network follows several restrictions, such as an inbound limit (a maximum limit of inbound connections for any neuron within the T-network), an outbound limit (a maximum limit of outbound connections for any neuron within the T-network), and a signal range (e.g., all signals should be inside pre-defined signal range).
  • T-transformation (322) is a process of converting some desired neural network, such as MobileNet, to a corresponding T-network.
  • a SPICE model 306 is a SPICE Neural Network model of a T- network 304, where each math neuron is substituted with corresponding one or more SNMs.
  • a Cadence NN model 310 is a Cadence model of the T-network 304, where each math neuron is substituted with a corresponding one or more SNMs.
  • two networks L and M have mathematical equivalence, if for all neuron outputs of these networks ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , where ⁇ ⁇ ⁇ is relatively small (e.g., between 0.1-1% of operating voltage range).
  • Figure 3B shows an example manual prototyping process used for generating a target chip model 320 based on a SNM model on Cadence 314, according to some implementations.
  • Cadence alternate tools from Mentor Graphic design or Synopsys (e.g., Synopsys design kit) may be used in place of Cadence tools, according to some implementations.
  • the process includes selecting SNM limitations, including inbound and outbound limits and signal limitation, selecting analog components (e.g., resistors, including specific resistor array technology) for connections between neurons, and developing a Cadence SNM model 314.
  • a prototype SNM model 316 (e.g., a PCB prototype) is developed (330) based on the SNM model on Cadence 314.
  • the prototype SNM model 316 is compared with a SPICE model for equivalence.
  • a neural network is selected for an on-chip prototype, when the neural network satisfies equivalence requirements. Because the neural network is small in size, the T-transformation can be hand-verified for equivalence.
  • an on-chip SNM model 318 is generated (332) based on the SNM model prototype 316.
  • the on-chip SNM model is optimized as possible, according to some implementations.
  • an on- chip density for the SNM model is calculated prior to generating (334) a target chip model 320 based on the on-chip SNM model 318, after finalizing the SNM.
  • FIGS. 4A, 4B, and 4C show examples of trained neural networks (e.g., the neural networks 220) that are input to the system 100 and transformed to mathematically equivalent analog networks, according to some implementations.
  • Figure 4A shows an example neural network (sometimes called an artificial neural network) that are composed of artificial neurons that receive input, combine the input using an activation function, and produce one or more outputs.
  • the input includes data, such as images, sensor data, and documents.
  • each neural network performs a specific task, such as object recognition.
  • the networks include connections between the neurons, each connection providing the output of a neuron as an input to another neuron. After training, each connection is assigned a corresponding weight.
  • the neurons are typically organized into multiple layers, with each layer of neurons connected only to the immediately preceding and following layer of neurons.
  • An input layer of neurons 402 receives external input (e.g., the input X1, X2, ..., Xn).
  • the input layer 402 is followed by one or more hidden layers of neurons (e.g., the layers 404 and 406), that is followed by an output layer 408 that produces outputs 410.
  • connection patterns connect neurons of consecutive layers, such as a fully-connected pattern that connects every neuron in one layer to all the neurons of the next layer, or a pooling pattern that connect output of a group of neurons in one layer to a single neuron in the next layer.
  • the neural network shown in Figure 4B includes one or more connections from neurons in one layer to either other neurons in the same layer or neurons in a preceding layer.
  • the example shown in Figure 4B is an example of a recurrent neural network, and includes two input neurons 412 (that accepts an input X1) and 414 (that accepts an input X2) in an input layer followed by two hidden layers.
  • the first hidden layer includes neurons 416 and 418 that is fully connected with neurons in the input layer, and the neurons 420, 422, and 424 in the second hidden layer.
  • the output of the neuron 420 in the second hidden layer is connected to the neuron 416 in the first hidden layer, providing a feedback loop.
  • the hidden layer including the neurons 420, 422, and 424 are input to a neuron 426 in the output layer that produces an output y.
  • Figure 4C shows an example of a convolutional neural network (CNN), according to some implementations.
  • the example shown in Figure 4C includes different types of neural network layers, that includes a first stage of layers for feature learning, and a second stage of layers for classification tasks, such as object recognition.
  • the feature learning stage includes a convolution and Rectified Linear Unit (ReLU) layer 430, followed by a pooling layer 432, that is followed by another convolution and ReLU layer 434, which is in turn followed by another pooling layer 436.
  • the first layer 430 extracts features from an input 428 (e.g., an input image or portions thereof), and performs a convolution operation on its input, and one or more non-linear operations (e.g., ReLU, tanh, or sigmoid).
  • a pooling layer such as the layer 432, reduces the number of parameters when the inputs are large.
  • the output of the pooling layer 436 is flattened by the layer 438 and input to a fully connected neural network with one or more layers (e.g., the layers 440 and 442).
  • the output of the fully-connected neural network is input to a softmax layer 444 to classify the output of the layer 442 of the fully-connected network to produce one of many different output 446 (e.g., object class or type of the input image 428).
  • Some implementations store the layout or the organization of the input neural networks including number of neurons in each layer, total number of neurons, operations or activation functions of each neuron, and/or connections between the neurons, in the memory 214, as the neural network topology 224.
  • Figure 5 shows an example of a math model 500 for a neuron, according to some implementations.
  • the math model includes incoming signals 502 input multiplied by synaptic weights 504 and summed by a unit summation 506. The result of the unit summation 506 is input to a nonlinear conversion unit 508 to produce an output signal 510, according to some implementations.
  • Figures 6A-6C illustrate an example process for analog hardware realization of a neural network for computing an XOR (classification of XOR results) of input values, according to some implementations.
  • Figure 6A shows a table 600 of possible input values X 1 and X2 along x- and y-axis, respectively.
  • the expected result values are indicated by hollow circle (represents a value of 1) and a filled or dark circle (represents a value of 0) – this is a typical XOR problem with 2 input signals and 2 classes. Only if either, not both, of the values X1 and X2 are 1, the expected result is 1, and 0, otherwise.
  • Training set consists of 4 possible input signal combinations (binary values for the X1 and X2 inputs).
  • Figure 6B shows a ReLU- based neural network 602 to solve the XOR classification of Figure 6A, according to some implementations. The neurons do not use any bias values, and use ReLU activation.
  • Inputs 604 and 606 (that correspond to X1 and X2, respectively) are input to a first ReLU neuron 608- 2.
  • the inputs 604 and 606 are also input to a second ReLU neuron 608-4.
  • the results of the two ReLU neurons 608-2 and 608-4 are input to a third neuron 608-6 that performs linear summation of the input values, to produce an output value 510 (the Out value).
  • the neural network 602 has the weights -1 and 1 (for the input values X 1 and X 2 , respectively) for the ReLU neuron 608-2, the weights 1 and -1 (for the input values X 1 and X 2 , respectively) for the ReLU neuron 608-4, and the weights 1 and 1 (for the output of the ReLU neurons 608-2 and 608-4, respectively).
  • the weights of trained neural networks are stored in memory 214, as the weights 222.
  • Figure 6C shows an example equivalent analog network for the network 602, according to some implementations.
  • the analog equivalent inputs 614 and 616 of the X1 and X 2 inputs 604 and 606 are input to analog neurons N1618 and N2620 of a first layer.
  • the neurons N1 and N2 are densely connected with neurons N3 and N4 of a second layer.
  • the neurons of a second layer i.e. neuron N3622 and neuron N4624
  • Some implementations use Keras learning that converges in approximately 1000 iterations, and results in weights for the connections.
  • the weights are stored in memory 214, as part of the weights 222.
  • data format is ‘Neuron [1 st link weight, 2 nd link weight, bias]’.
  • N1 [-0.9824321, 0.976517, -0.00204677];
  • N2 [1.0066702, -1.0101418, -0.00045485];
  • N3 [1.0357606, 1.0072469, -0.00483723]; • N4 [-0.07376373, -0.7682612, 0.0]; and • N5 [1.0029935, -1.1994369, -0.00147767].
  • resistor range For connections between the neurons, some implementations compute resistor range. Some implementations set resistor nominal values (R+, R-) of 1 M ⁇ , possible resistor range of 100 K ⁇ to 1 M ⁇ and nominal series E24. Some implementations compute w1, w2, wbias resistor values for each connection as follows. For each weight value wi (e.g., the weights 222), some implementations evaluate all possible (Ri-, Ri+) resistor pairs options within the chosen nominal series and choose a resistor pair which ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ + ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ . The following table provides example values for the weights w1, w2, and bias, for each connection, according to some implementations.
  • pyramids or trapezium over cross bars Some of the advantages of pyramid or trapezium over cross bars include lower latency, simultaneous analog signal propagation, possibility for manufacture using standard integrated circuit (IC) design elements, including resistors and operational amplifiers, high parallelism of computation, high accuracy (e.g., accuracy increases with the number of layers, relative to conventional methods), tolerance towards error(s) in each weight and/or at each connection (e.g., pyramids balance the errors), low RC (low Resistance Capacitance delay related to propagation of signal through network), and/or ability to manipulate biases and functions of each neuron in each layer of the transformed network.
  • IC integrated circuit
  • pyramids are excellent computation block by itself, since it is a multi-level perceptron, which can model any neural network with one output.
  • a pyramid can be thought of as a multi-layer perceptron with one output and several layers (e.g., N layers), where each neuron has n inputs and 1 output.
  • a trapezium is a multilayer perceptron, where each neuron has n inputs and m outputs.
  • Each trapezium is a pyramid-like network, where each neuron has n inputs and m outputs, where n and m are limited by IC analog chip design limitations, according to some implementations.
  • Some implementations perform lossless transformation of any trained neural network into subsystems of pyramids or trapezia.
  • pyramids and trapezia can be used as universal building blocks for transforming any neural networks.
  • An advantage of pyramid- or trapezia-based neural networks is the possibility to realize any neural network using standard IC analog elements (e.g., operational amplifiers, resistors, signal delay lines in case of recurrent neurons) using standard lithography techniques. It is also possible to restrict the weights of transformed networks to some interval. In other words, lossless transformation is performed with weights limited to some predefined range, according to some implementations.
  • Another advantage of using pyramids or trapezia is the high degree of parallelism in signal processing or the simultaneous propagation of analog signals that increases the speed of calculations, providing lower latency.
  • analog neuromorphic trapezia-like chips possess a number of properties, not typical for analog devices. For example, signal to noise ratio is not increasing with the number of cascades in analog chip, the external noise is suppressed, and influence of temperature is greatly reduced. Such properties make trapezia-like analog neuromorphic chips analogous to digital circuits.
  • individual neurons based on operational amplifier, level the signal and are operated with the frequencies of 20,000-100,000 Hz, and are not influenced by noise or signals with frequency higher than the operational range, according to some implementations.
  • Trapezia-like analog neuromorphic chip also perform filtration of output signal due to peculiarities in how operational amplifiers function. Such trapezia-like analog neuromorphic chip suppresses the synphase noise. Due to low-ohmic outputs of operational amplifiers, the noise is also significantly reduced. Due to the leveling of signal at each operational amplifier output and synchronous work of amplifiers, the drift of parameters, caused by temperature does not influence the signals at final outputs.
  • Trapezia-like analogous neuromorphic circuit is tolerant towards the errors and noise in input signals and is tolerant towards deviation of resistor values, corresponding to weight values in neural network. Trapezia-like analog neuromorphic networks are also tolerant towards any kind of systemic error, like error in resistor value settings, if such error is same for all resistors, due to the very nature of analog neuromorphic trapezia-like circuits, based on operational amplifiers.
  • Example Lossless Transformation (T-Transformation) of Trained Neural Networks are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • Figure 7 shows an example perceptron 700, according to some implementations.
  • There is an output layer with 4 neurons 704-2, ..., 704- 8, in an output layer, that correspond to L 4 outputs.
  • FIG. 8 shows an example Pyramid-Neural Network (P-NN) 800, a type of Target-Neural Network (T-NN, or TNN), that is equivalent to the perceptron shown in Figure 7, according to some implementations.
  • P-NN Pyramid-Neural Network
  • T-NN Target-Neural Network
  • the set of neurons 804, including neurons 802-20, ..., 802-34, is a copy of the neurons 802-2, ..., 802- 18, and the input is replicated.
  • the network shown in Figure 8 includes 40 connections.
  • FIG. 9 shows a Pyramid Single Neural Network (PSNN) 900 corresponding to an output neuron of Figure 8, according to some implementations.
  • the PSNN includes a layer (LPSI) of input neurons 902-02, ..., 902-16 (corresponding to the 8 input neurons in the network 700 of Figure 7).
  • An output layer LPSO consists of 1 neuron 906 with an activation function F, that is connected to both the neurons 904-02 and 904-04 of the hidden layer.
  • For calculating weight matrix for the PSNN 900 some implementations compute a vector WPSH1 that is equal to the first row of WP, for the LPSH1 layer.
  • For the LPSOlayer some implementations compute a weight vector WPSO with 2 elements, each element equal to 1. The process is repeated for the first, second, third, and fourth output neurons.
  • a P-NN such as the network shown in Figure 8, is a union of the PSNNs (for the 4 output neurons). Input layer for every PSNN is a separate copy of P’s input layer.
  • Example Transformations with Target Neurons with N Inputs and 1 Output are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or analog design constraints 236, to obtain the transformed neural networks 228.
  • Single Layer Perceptron with One Output [00255] Suppose a single layer perceptron SLP(K,1) includes K inputs and one output neuron with activation function F.
  • ⁇ ⁇ ⁇ ⁇ is a vector of weights for SLP(K,1).
  • the following algorithm Neuron2TNN1 constructs a T-neural network from T-neurons with N inputs and 1 output (referred to as TN(N,1)).
  • the T-NN’s output is equal to: [00263] This is the same value as the one calculated in SLP(K,1) for the same input vector x. So output values of SLP(K,1) and constructed T-NN are equal.
  • Single Layer Perceptron with Several Outputs [00264] Suppose there is a single layer perceptron SLP(K, L) with K inputs and L output neurons, each neuron performing an activation function F. Suppose further ⁇ ⁇ ⁇ ⁇ is a weight matrix for SLP(K, L). The following algorithm Layer2TNN1 constructs a T-neural network from neurons TN(N, 1). Algorithm Layer2TNN1 1.
  • Output of the PTNN is equal to the SLP(K, L)’s output for the same input vector because output of every pair SLP i (K, 1) and TNN i are equal.
  • Multilayer Perceptron [00266] Suppose a multilayer perceptron (MLP) includes K inputs, S layers and L i calculation neurons in i-th layer, represented as MLP(K, S, L 1 ,... L S ). Suppose ⁇ ⁇ ⁇ ⁇ ⁇ is a weight matrix for the i-th layer. [00267] The following is an example algorithm to construct a T-neural network from neurons TN(N, 1)., according to some implementations. Algorithm MLP2TNN1 1.
  • Example T-transformations with Target neurons with N I Inputs and N O Outputs are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • Example Transformation of Single Layer Perceptron with Several Outputs [00270] Suppose a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each neuron performing an activation function F. Suppose further ⁇ ⁇ ⁇ ⁇ is a weight matrix for SLP(K,L).
  • the following algorithm constructs a T-neural network from neurons TN(NI, NO), according to some implementations.
  • Algorithm Layer2TNNX 1. Construct a PTNN from SLP(K,L) by using the algorithm Layer2TNN1 (see description above). PTNN has an input layer consisting of L groups of K inputs. 2. Compose ⁇ ⁇ ⁇ ⁇ ⁇ subsets from L groups. Each subset contains no more than N O groups of input vector copies. 3. Replace groups in every subset with one copy of input vector. 4. Construct PTNNX by rebuild connections in every subset by making NO output connections from every input neuron.
  • FIGS 11A-11C show an application 1100 of the above algorithm for a single layer neural network (NN) with 2 output neurons and TN(N I , 2), according to some implementations.
  • Figure 11A shows an example source or input NN, according to some implementations.
  • K inputs are input to two neurons 1 and 2 belonging to a layer 1104.
  • Figure 11B shows a PTNN constructed after the first step of the algorithm, according to some implementations.
  • the PTNN consists of two parts implementing subnets corresponding to the output neuron 1 and neuron 2 of the NN shown in Figure 11A.
  • the input 1102 is replicated and input to two sets of input neurons 1106-2 and 1106-4.
  • Each set of input neurons is connected to a subsequent layer of neurons with two sets of neurons 1108-2 and 1108-4, each set of neurons including m1 neurons.
  • the input layer is followed by identity transform blocks 1110-2 and 1110-4, each block containing one or more layers with identity weight matrix.
  • the output of the identity transform block 1110-2 is connected to the output neuron 1112 (corresponding to the output neuron 1 in Figure 11A), and the output of the identity transform block 1110-4 is connected to the output neuron 1114 (corresponding to the output neuron 1 in Figure 11A).
  • Figure 11C shows application of the final steps of the algorithm, including replacing two copies of the input vector (1106-2 and 1106-4) with one vector 1116 (step 3), and rebuilding connections in the first layer 1118 by making two output links from every input neuron: one link connects to subnet related to output 1 and another link connects to subnet for the output 2.
  • MLP multilayer perceptron
  • the following example algorithm constructs a T-neural network from neurons TN(NI, NO), according to some implementations.
  • PTNNX i is constructed as a result. 2. Construct MTNNX by stacking all PTNNX i into one neural net: a. Output of a TNNXi-1 is set as input for TNNXi. [00274] According to some implementations.
  • a Recurrent Neural Network contains backward connection allowing saving information.
  • Figure 12 shows an example RNN 1200, according to some implementations.
  • the example shows a block 1204 performing an activation function A, that accepts an input X t 1206 and performs an activation function A, and outputs a value h t 1202.
  • the backward arrow from the block 1204 to itself indicates a backward connection, according to some implementations.
  • An equivalent network is shown on the right up to the point in time when the activation block receives the input X t 1206.
  • the network accepts input X t 1208 and performs the activation function A 1204, and outputs a value ho 1210;
  • the network accepts input X1 1212 and the output of the network at time 0, and performs the activation function A 1204, and outputs a value h 1 1214;
  • the network accepts input X21216 and the output of the network at time 1, and performs the activation function A 1204, and outputs a value h11218.
  • the first and third operations can be implemented by trapezium-based network (one fully connected layer is implemented by pyramid-based network, a special case of trapezium networks).
  • the second operation is a common operation that can be implemented in networks of any structure.
  • the RNN’s layer without recurrent connections is transformed by means of Layer2TNNX algorithm described above. After transformation is completed, recurrent links are added between related neurons. Some implementations use delay blocks described below in reference to Figure 13B.
  • Example Transformation of LSTM Network [00279] A Long Short-Term Memory (LSTM) neural network is a special case of a RNN.
  • FIG. 13A is a block diagram of a LSTM neuron 1300, according to some implementations.
  • a sigmoid ( ⁇ ) block 1318 processes the inputs h ⁇ 1330 and ⁇ ⁇ 1332, and produces the output ⁇ ⁇ 1336.
  • a second sigmoid ( ⁇ ) block 1320 processes the inputs h ⁇ 1330 and ⁇ ⁇ 1332, and produces the output ⁇ ⁇ 1338.
  • a hyperbolic tangent (tanh) block 1322 processes the inputs h ⁇ 1330 and ⁇ ⁇ 1332, and produces the output ⁇ ⁇ 1340.
  • a third sigmoid ( ⁇ ) block 1328 processes the inputs h ⁇ 1330 and ⁇ ⁇ 1332, and produces the output ⁇ ⁇ 1342.
  • a multiplier block 1304 processes ⁇ ⁇ 1336 and the output of a summing block 1306 (from a prior time instance) ⁇ ⁇ 1302 to produce an output that is in turn summed by the summing block 1306 along with the output of a second multiplier block 1314 that multiplies the outputs ⁇ ⁇ 1338 and ⁇ ⁇ 1340 to produce the output ⁇ ⁇ 1310.
  • the output ⁇ ⁇ 1310 is input to another tanh block 1312 that produces an output that is multiplied a third multiplier block 1316 with the output ⁇ ⁇ 1342 to produce the output h ⁇ 1334.
  • FIG. 13B shows delay blocks, according to some implementations. As described above, some of the expressions in the equations for the LSTM operations depend on saving, restoring, and/or recalling an output from a previous time instance. For example, the multiplier block 1304 processes the output of the summing block 1306 (from a prior time instance) ⁇ ⁇ 1302. Figure 13B shows two examples of delay blocks, according to some implementations.
  • the example 1350 includes a delay block 1354 on the left accepts input x t 1352 at time t, and outputs the input after a delay of dt indicated by the output xt-dt 1356.
  • the example 1360 on the right shows cascaded (or multiple) delay blocks 1364 and 1366 outputs the input x t 1362 after 2 units of time delays, indicated by the output x t-2dt 1368, according to some implementations.
  • Figure 13C is a neuron schema for a LSTM neuron, according to some implementations.
  • the schema includes weighted summator nodes (sometimes called adder blocks) 1372, 1374, 1376, 1378, and 1396, multiplier blocks 1384, 1392, and 1394, and delay blocks 1380 and 1382.
  • the input ⁇ ⁇ 1332 is connected to the adder blocks 1372, 1374, 1376, and 1378.
  • the output 1330 for a prior input is also input to the adder blocks 1372, 1374, 1376, and 1378.
  • the adder block 1372 produces an output that is input to a sigmoid block 1394-2 that produces the output ⁇ ⁇ 1336.
  • the adder block 1374 produces an output that is input to the sigmoid block 1386 that produces the output ⁇ ⁇ 1338.
  • the adder block 1376 produces an output that is input to a hyperbolic tangent block 1388 that produces the output ⁇ ⁇ 1340.
  • the adder block 1378 produces an output that is input to the sigmoid block 1390 that produces the output ⁇ ⁇ 1342.
  • the multiplier block 1392 uses the outputs ⁇ ⁇ 1338, ⁇ ⁇ 1336, and output of the adder block 1396 from a prior time instance 1302 to produce a first output.
  • the multiplier block 1394 uses the outputs ⁇ ⁇ 1338 and ⁇ ⁇ 1340 to produce a second output.
  • the adder block 1396 sums the first output and second output to produce the output ⁇ ⁇ 1310.
  • the output ⁇ ⁇ 1310 is input to a hyperbolic tangent block 1398 that produces an output that is input, along with the output of the sigmoid block 1390, ⁇ ⁇ 1342, to the multiplier block 1384 to produce the output h ⁇ 1334.
  • the delay block 1382 is used to recall (e.g., save and restore) the output of the adder block 1396 from a prior time instance.
  • the delay block 1380 is used to recall or save and restore the output of the multiplier block 1384 for a prior input (e.g., from a prior time instance). Examples of delay blocks are described above in reference to Figure 13B, according to some implementations.
  • a Gated Recurrent Unit (GRU) neural network is a special case of RNN.
  • FIG. 14A is a block diagram of a GRU neuron, according to some implementations.
  • a sigmoid ( ⁇ ) block 1418 processes the inputs h ⁇ 1402 and ⁇ ⁇ 1422, and produces the output ⁇ ⁇ 1426.
  • a second sigmoid ( ⁇ ) block 1420 processes the inputs h ⁇ 1402 and ⁇ ⁇ 1422, and produces the output ⁇ ⁇ 1428.
  • a multiplier block 1412 multiplies the output ⁇ ⁇ 1426 and the input 1402 to produce and output that is input (along with the input ⁇ ⁇ 1422) to a hyperbolic tangent (tanh) block 1424 to produce the output ⁇ ⁇ 1430.
  • a second multiplier block 1414 multiplies the output ⁇ ⁇ 1430 and the output ⁇ ⁇ 1428 to produce a first output.
  • the block 1410 computes 1 – the output ⁇ ⁇ 1428 to produce an output that is input to a third multiplier block 1404 that multiplies the output and the input h ⁇ 1402 to produce a product that is input to an adder block 1406 along with the first output (from the multiplier block 1414) to produce the output h ⁇ 1408.
  • Figure 14B is a neuron schema for a GRU neuron 1440, according to some implementations.
  • the schema includes weighted summator nodes (sometimes called adder blocks) 1404, 1406, 1410, 1406, and 1434, multiplier blocks 1404, 1412, and 1414, and delay block 1432.
  • the input ⁇ ⁇ 1422 is connected to the adder blocks 1404, 1410, and 1406.
  • the output 1402 for a prior input is also input to the adder blocks 1404 and 1406, and the multiplier blocks 1404 and 1412.
  • the adder block 1404 produces an output that is input to a sigmoid block 1418 that produces the output ⁇ ⁇ 1428.
  • the adder block 1406 produces an output that is input to the sigmoid block 1420 that produces the output ⁇ ⁇ 1426 that is input to the multiplier block 1412.
  • the output of the multiplier block 1412 is input to the adder block 1410 whose output is input to a hyperbolic tangent block 1424 that produces an output 1430.
  • the output 1430 as well as the output of the sigmoid block 1418 are input to the multiplier block 1414.
  • the output of the sigmoid block 1418 is input to the multiplier block 1404 that multiplies that output with the input from the delay block 1432 to produce a first output.
  • the multiplier block produces a second output.
  • the adder block 1434 sums the first output and the second output to produce the output h ⁇ 1408.
  • the delay block 1432 is used to recall (e.g., save and restore) the output of the adder block 1434 from a prior time instance. Examples of delay blocks are described above in reference to Figure 13B, according to some implementations.
  • Operation types used in GRU are the same as the operation types for LSTM networks (described above), so GRU is transformed to trapezium-based networks following the principles described above for LSTM (e.g., using the Layer2TNNX algorithm), according to some implementations.
  • Convolutional Neural Networks include several basic operations, such as convolution (a set of linear combinations of image’s (or internal map’s) fragments with a kernel), activation function, and pooling (e.g., max, mean, etc.). Every calculation neuron in a CNN follows the general processing scheme of a neuron in an MLP: linear combination of some inputs with subsequent calculation of activation function. So a CNN is transformed using the MLP2TNNX algorithm described above for multilayer perceptrons, according to some implementations.
  • Conv1D is a convolution performed over time coordinate.
  • Figures 15A and 15B are neuron schema of variants of a single Conv1D filter, according to some implementations.
  • a weighted summator node 1502 (sometimes called adder block, marked ‘+’) has 5 inputs, so it corresponds to 1Dconvolution with a kernel of 5.
  • the inputs are ⁇ ⁇ 1504 from time ⁇ , ⁇ ⁇ 1514 from time ⁇ ⁇ 1 (obtained by inputting the input to a delay block 1506), ⁇ ⁇ 1516 from time ⁇ ⁇ 2 (obtained by inputting the output of the delay block 1506 to another delay block 1508), ⁇ ⁇ 1518 from time ⁇ ⁇ 3 (obtained by inputting the output of the delay block 1508 to another delay block 1510), and ⁇ ⁇ 1520 from time ⁇ ⁇ 4 (obtained by inputting the output of the delay block 1510 to another delay block 1512.
  • Some implementations substitute several small delay blocks for one large delay block, as shown in Figure 15B.
  • the example uses a delay_3 block 1524 that produces ⁇ ⁇ 1518 from time ⁇ ⁇ 3, and another delay block 1526 that produces the ⁇ ⁇ 1522 from time ⁇ ⁇ 5.
  • the delay_31524 block is an example of multiple delay blocks, according to some implementations. This operation does not decrease total number of blocks, but it may decrease total number of consequent operations performed over the input signal and reduce accumulation of errors, according to some implementations.
  • convolutional layers are represented by trapezia- like neurons and fully connected layer is represented by cross-bar of resistors.
  • Some implementations use cross-bars, and calculate resistance matrix for the cross-bars.
  • Example Approximation Algorithm for Single Layer Perceptron with Multiple Outputs [00294]
  • the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, and/or the analog neural network optimization module 246, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each output neuron performing an activation function F.
  • ⁇ ⁇ ⁇ ⁇ is a weight matrix for SLP(K, L).
  • Layer2TNNX_Approx an approximation algorithm
  • the algorithm applies Layer2TNN1 algorithm (described above) at the first stage in order to decrease a number of neurons and connections, and subsequently applies Layer2TNNX to process the input of the decreased size.
  • the outputs of the resulted neural net are calculated using shared weights of the layers constructed by the Layer2TNN1 algorithm.
  • the number of these layers is determined by the value p, a parameter of the algorithm. If p is equal to 0 then Layer2TNNX algorithm is applied only and the transformation is equivalent.
  • C is any constant not equal to zero
  • ⁇ ( ⁇ ⁇ 1) ⁇ ⁇ + 1, for all weights j of this neuron except k
  • All other weights of the PNN net are set to 1.
  • ⁇ ( ⁇ ) ⁇ ⁇ represents a weight for the first layer (as denoted by the superscript (1)) for the connection between the neuron i and the neuron ⁇ ⁇ in the first layer. 5.
  • FIG. 16 shows an example architecture 1600 of the resulting neural net, according to some implementations.
  • the example includes a PNN 1602 connected to a TNN 1606.
  • the PNN 1602 includes a layer for ⁇ inputs and produce ⁇ ⁇ outputs, that is connected as input 1612 to the TNN 1606.
  • the TNN 1606 generates L outputs 1610, according to some implementations.
  • MLP multilayer perceptron
  • the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, and/or the analog neural network optimization module 246, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • This section describes example methods of compression of transformed neural networks, according to some implementations. Some implementations compress analog pyramid-like neural networks in order to minimize the number of operational amplifiers and resistors, necessary to realize the analog network on chip. In some implementations, the method of compression of analog neural networks is pruning, similar to pruning in software neural networks.
  • the transformation of dense neural networks into sparsely connected pyramid or trapezia or cross-bar like neural networks presents opportunities to prune the sparsely connected pyramid or trapezia-like analog networks, which are then represented by operational amplifiers and resistors in analog IC chips.
  • such techniques are applied in addition to conventional neural network compression techniques.
  • the compression techniques are applied based on the specific architecture of the input neural network and/or the transformed neural networks (e.g., pyramids versus trapezia versus cross-bars).
  • some implementations determine the current which flows through the operational amplifier when the standard training dataset is presented, and thereby determine if a knot (an operational amplifier) is needed for the whole chip or not. Some implementations analyze the SPICE model of the chip and determine the knots and connections, where no current is flowing and no power is consumed. Some implementations determine the current flow through the analog IC network and thus determine the knots and connections, which are then pruned. Besides, some implementations also remove the connections if the weight of connection is too high, and/or substitute resistor to direct connector if the weight of connection is too low.
  • Some implementations prune the knot if all connections leading to this knot have weights that are lower than a predetermined threshold (e.g., close to 0), deleting the connections where an operational amplifier always provides zero at output, and/or changing an operational amplifier to a linear junction if the amplifier gives linear function without amplification.
  • a predetermined threshold e.g., close to 0
  • Some implementations apply compression techniques specific to pyramid, trapezia, or cross-bar types of neural networks. Some implementations generate pyramids or trapezia with larger amount of inputs (than without the compression), thus minimizing the number of layers in pyramid or trapezia. Some implementations generate a more compact trapezia network by maximizing the number of outputs of each neuron.
  • Example Generation of Optimal Resistor Set the example computations described herein are performed by the weight matrix computation or weight quantization module 238 (e.g., using the resistance calculation module 240) that compute the weights 272 for connections of the transformed neural networks, and/or corresponding resistance values 242 for the weights 272.
  • This section describes an example of generating an optimal resistor set for a trained neural network, according to some implementations.
  • An example method is provided for converting connection weights to resistor nominals for implementing the neural network (sometimes called a NN model) on a microchip with possibly less resistor nominals and possibly higher allowed resistor variance.
  • a test set ‘Test’ includes around 10,000 values of input vector (x and y coordinates) with both coordinates varying in the range [0;1], with a step of 0.01.
  • the following compares a mathematical network model M with a schematic network model S.
  • Output error is defined by the following equation: [00306]
  • Classification error is defined by the following equation: ( X ⁇ ) ⁇ Class_m(X ⁇ ) ⁇ ⁇ [00307]
  • Some implementations set the desired classification error as no more than 1%.
  • Example Error Analysis [00308]
  • Figure 17A shows an example chart 1700 illustrating dependency between output error and classification error on the M network, according to some implementations.
  • the x-axis corresponds to classification margin 1704
  • the y-axis corresponds to total error 1702 (see description above).
  • the graph shows total error (difference between output of model M and real data) for different classification margins of output signal.
  • the optimal classification margin 1706 is 0.610.
  • Maximum weight modulus (maximum of absolute value of weights among all weights) for the neural network is 1.94.
  • Example Process for Choosing Resistor Set [00311] A resistor set together with a ⁇ R+, R- ⁇ pair chosen from this set has a value function over the required weight range [-wlim; wlim] with some degree of resistor error r_err.
  • value function of a resistor set is calculated as follows: • Possible weight options array is calculated together with weight average error dependent on resistor error; • The weight options in the array is limited to the required weight range [-wlim; wlim]; • Values that are worse than neighboring values in terms of weight error are removed; • An array of distances between neighboring values is calculated; and • The value function is a composition of square mean or maximum of the distances array. [00312] Some implementations iteratively search for an optimal resistor set by consecutively adjusting each resistor value in the resistor set on a learning rate value. In some implementations, the learning rate changes over time.
  • an initial resistor set is chosen as uniform (e.g., [1;1;...;1]), with minimum and maximum resistor values chosen to be within two orders of magnitude range (e.g., [1;100] or [0.1;10]).
  • Some implementation choose R+ R-.
  • the iterative process converges to a local minimum. In one case, the process resulted in the following set: [0.17, 1.036, 0.238, 0.21, 0.362, 1.473, 0.858, 0.69, 5.138, 1.215, 2.083, 0.275].
  • Some implementations do not use the whole available range [rmin; rmax] for finding a good local optimum. Only part of the available range (e.g., in this case [0.17; 5.13]) is used.
  • the resistor set values are relative, not absolute. Is this case, relative value range of 30 is enough for the resistor set.
  • the following resistor set of length 20 is obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02].
  • This set is subsequently used to produce weights for NN, producing corresponding model S.
  • the model S’s mean square output error was 11 mV given the relative resistor error is close to zero, so the set of 20 resistors is more than required.
  • Maximum error over a set of input data was calculated to be 33 mV.
  • S, DAC, and ADC converters with 256 levels were analyzed as a separate model, and the result showed 14 mV mean square output error and 49 mV max output error.
  • An output error of 45 mV on NN corresponds to a relative recognition error of 1%.
  • the 45 mV output error value also corresponds to 0.01 relative or 0.01 absolute weight error, which is acceptable.
  • Maximum weight modulus in NN is 1.94. In this way, the optimal (or near optimal) resistor set is determined using the iterative process, based on desired weight range [-wlim; wlim], resistors error (relative), and possible resistors range.
  • a very broad resistor set is not very beneficial (e.g., between 1-1/5 orders of magnitude is enough) unless different precision is required within different layers or weight spectrum parts. For example, suppose weights are in the range of [0, 1], but most of the weights are in the range of [0, 0.001], then better precision is needed within that range. In the example described above, given the relative resistor error is close to zero, the set of 20 resistors is more than sufficient for quantizing the NN network, with given precision.
  • Example Process for Quantization of Resistor Values [00315]
  • the example computations described herein are performed by the weight matrix computation or weight quantization module 238 (e.g., using the resistance calculation module 240) that compute the weights 272 for connections of the transformed neural networks, and/or corresponding resistance values 242 for the weights 272.
  • This section describes an example process for quantizing resistor values corresponding to weights of a trained neural network, according to some implementations.
  • the example process substantially simplifies the process of manufacturing chips using analog hardware components for realizing neural networks.
  • some implementations use resistors to represent neural network weights and/or biases for operational amplifiers that represent analog neurons.
  • the example process described here specifically reduces the complexity in lithographically fabricating sets of resistors for the chip. With the procedure of quantizing the resistor values, only select values of resistances are needed for chip manufacture. In this way, the example process simplifies the overall process of chip manufacture and enables automatic resistor lithographic mask manufacturing on demand.
  • Figure 18 provides an example scheme of a neuron model 1800 used for resistors quantization, according to some implementations.
  • the circuit is based on an operational amplifier 1824 (e.g., AD824 series precision amplifier) that receives input signals from negative weight fixing resistors (R1- 1804, R2- 1806, Rb- bias 1816, Rn- 1818, and R- 1812), and positive weight fixing resistors (R1+ 1808, R2+ 1810, Rb+ bias 1820, Rn+ 1822), and R+ 1814).
  • the positive weight voltages are fed into direct input of the operational amplifier 1824 and negative weights voltages are fed into inverse input of the operational amplifier 1824.
  • the operational amplifier 1824 is used to allow weighted summation operation of weighted outputs from each resistor, where negative weights are subtracted from positive weights.
  • the operational amplifier 1824 also amplifies signal to the extent necessary for the circuit operation. In some implementations, the operational amplifier 1824 also accomplishes RELU transformation of output signal at it’s output cascade.
  • the following example optimization procedure quantizes the values of each resistance and minimize the error of neural network output, according to some implementations: 1. Obtain a set of connection weights and biases ⁇ w1, ..., wn, b ⁇ .. 2. Obtain possible minimum and maximum resistor values ⁇ Rmin, Rmax ⁇ . These parameters are determined based on the technology used for manufacturing. Some implementations use TaN or Tellurium high resistivity materials.
  • the minimum value of resistor is determined by minimum square that can be formed lithographically.
  • the maximum value is determined by length, allowable for resistors (e.g., resistors made from TaN or Tellurium) to fit to the desired area, which is in turn determined by the area of an operational amplifier square on lithographic mask.
  • the area of arrays of resistors is smaller than the area of one operational amplifier, since the arrays of resistors are stacked (e.g., one in BEOL, another in FEOL). 3. Assume that each resistor has r_err relative tolerance value 4.
  • the goal is to select a set of resistor values ⁇ R1, ..., Rn ⁇ of given length N within the defined [Rmin; Rmax], based on ⁇ w1, ..., wn, b ⁇ values.
  • An example search algorithm is provided below to find sub-optimal ⁇ R1, ..., Rn ⁇ set based on particular optimality criteria. 5.
  • Another algorithm chooses ⁇ Rn, Rp, Rni, Rpi ⁇ for a network given that ⁇ R1..Rn ⁇ is determined.
  • Example ⁇ R1, ..., Rn ⁇ Search Algorithm [00320] Some implementations use an iterative approach for resistor set search. Some implementations select an initial (random or uniform) set ⁇ R1, ..., Rn ⁇ within the defined range.
  • Weight options list is limited or restricted to [-wlim; wlim] range • Some values, which have expected error beyond a high threshold (e.g., 10 times r_err), are removed •
  • Value function is calculated as a square mean of distance between two neighboring weight options. So, value function is minimal when weight options are distributed uniformly within [-wlim; wlim] range [00322]
  • rmin and rmax are minimum and maximum values for resistances, respectively.
  • resistor set of length 20 was obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] M ⁇ .
  • Example ⁇ Rn, Rp, Rni, Rpi ⁇ Search Algorithm [00324] Some implementations determine Rn and Rp using an iterative algorithm such as the algorithm described above.
  • mean square output error sometimes called S mean square output error, described above
  • S model was analyzed along with digital-to-analog converters (DAC), analog-to-digital converters (ADC), with 256 levels as a separate model.
  • the model produced 14 mV mean square output error and 49 mV max output error on the same data set, according to some implementations.
  • DAC and ADC have levels because they convert analog value to bit value and vice-versa. 8 bits of digital value is equal to 256 levels. Precision cannot be better than 1/256 for 8-bit ADC.
  • Some implementations calculate the resistance values for analog IC chips, when the weights of connections are known, based on Kirchhoff’s circuit laws and basic principles of operational amplifiers (described below in reference to Figure 19A), using Mathcad or any other similar software.
  • operational amplifiers are used both for amplification of signal and for transformation according to the activation functions (e.g., ReLU, sigmoid, Tangent hyperbolic, or linear mathematical equations), [00327]
  • Some implementations manufacture resistors in a lithography layer where resistors are formed as cylindrical holes in the SiO2 matrix and the resistance value is set by the diameter of hole.
  • Some implementations use amorphous TaN, TiN of CrN or Tellurium as the highly resistive material to make high density resistor arrays.
  • Some ratios of Ta to N Ti to N and Cr to N provide high resistance for making ultra-dense high resistivity elements arrays.
  • FIG. 19A shows a schematic diagram of an operational amplifier made on CMOS (CMOS OpAmp) 1900, according to some implementations.
  • In+ positive input or pos
  • In- negative input or neg
  • Vdd- positive supply voltage relative to GND
  • Contact Vss- negative supply voltage or GND
  • the circuit output is Out 1410 (contact output).
  • Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel) to W (the width of the gate channel), examples of which are shown in the Table shown in Figure 19B (described below).
  • the current mirror is made on NMOS transistors M111944, M121946, and resistor R11921 (with an example resistance value of 12 k ⁇ ), and provides the offset current of the differential pair (M11926 and M31930).
  • the differential amplifier stage (differential pair) is made on the NMOS transistors M11926 and M3 1930.
  • Transistors M1, M3 are amplifying, and PMOS transistors M21928 and M41932 play the role of active current load.
  • the signal is input to the gate of the output PMOS transistor M71936. From the transistor M1, the signal is input to the PMOS transistor M5 (inverter) 1934 and the active load on the NMOS transistor M61934. The current flowing through the transistor M5 1934 is the setting for the NMOS transistor M8 1938.
  • Transistors M71936 is included in the scheme with a common source for a positive half-wave signal.
  • the M8 transistors 1938 are enabled by a common source circuit for a negative half- wave signal.
  • the M71936 and M81938 outputs include an inverter on the M91940 and M101942 transistors. Capacitors C11912 and C21914 are blocking.
  • Figure 19B shows a table 1948 of description for the example circuit shown in Figure 19A, according to some implementations.
  • the values for the parameters are provided as examples, and various other configurations are possible.
  • the transistors M1, M3, M6, M8, M10, M11, and M12 are N-Channel MOSFET transistors with explicit substrate connection.
  • the other transistors M2, M4, M5, M7, and M9 are P-Channel MOSFET transistors with explicit substrate connection.
  • the Table shows example shutter ratio of length (L, column 1) and width (W, column 2) are provided for each of the transistors (column 3).
  • operational amplifiers such as the example described above are used as the basic element of integrated circuits for hardware realization of neural networks.
  • the operational amplifiers are of the size of 40 square microns and fabricated according to 45 nm node standard.
  • activation functions such as ReLU, Hyperbolic Tangent, and Sigmoid functions are represented by operational amplifiers with modified output cascade.
  • RELU, Sigmoid, or Tangent function is realized as an output cascade of an operational amplifier (sometimes called OpAmp) using corresponding well-known analog schematics, according to some implementations.
  • the operational amplifiers are substituted by inverters, current mirrors, two-quadrant or four quadrant multipliers, and/or other analog functional blocks, that allow weighted summation operation.
  • FIGS 20A-20E show a schematic diagram of a LSTM neuron 20000, according to some implementations.
  • the inputs of the neuron are Vin120002 and Vin220004 that are values in the range [-0.1,0.1].
  • the LSTM neuron also input the value of the result of calculating the neuron at time H(t-1) (previous value; see description above for LST neuron) 20006 and the state vector of the neuron at time C(t-1) (previous value) 20008.
  • Outputs of the neuron LSTM (shown in Figure 20B) include the result of calculating the neuron at the present time H(t) 20118 and the state vector of the neuron at the present time C(t) 20120.
  • the scheme includes: • a “neuron O” assembled on the operational amplifiers U120094 and U220100, shown in Figure 20A.
  • Resistors R_Wo1 20018, R_Wo2 20016, R_Wo3 20012, R_Wo4 20010, R_Uop120014, R_Uom120020, Rr 20068 and Rf220066 set the weights of connections of the single “neuron O”.
  • the “neuron O” uses a sigmoid (module X1 20078, Figure 20B) as a nonlinear function; • a "neuron C“ assembled on the operational amplifiers U320098 (shown in Figure 20C) and U420100 (shown in Figure 20A).
  • Resistors R_Wc120030, R_Wc220028, R_Wc3 20024, R_Wc420022, R_Ucp120026, R_Ucm120032, Rr 20122, and Rf220120 set the weights of connections of the “neuron ⁇ ”.
  • the “neuron C” uses a hyperbolic tangent (module X222080, Figure 2B) as a nonlinear function; • a “neuron I” assembled on the operational amplifiers U520102 and U620104, shown in Figure 20C.
  • Resistors R_Wi120042, R_Wi2 20040, R_Wi320036, and R_Wi4 20034, R_Uip120038, R_Uim120044, Rr 20124, and Rf220126 set the weights of connections of the “neuron I”.
  • the “neuron I” uses a sigmoid (module X320082) as a nonlinear function; and • a "neuron f“ assembled on the operational amplifiers U720106 and U820108, as shown in Figure 20D.
  • Resistors R_Wf120054, R_Wf220052, R_Wf320048, R_Wf420046, R_Ufp1 20050, R_Ufm1 20056, Rr 20128 and Rf2 20130 set the weights of connections of the “neuron f”.
  • the “neuron f” uses a sigmoid (module X420084) as a nonlinear function.
  • the outputs of modules X220080 ( Figure 20B) and X320082 ( Figure 20C) are input to the X5 multiplier module 20086 ( Figure 20B).
  • the outputs of modules X420084 ( Figure 20D) and buffer to U920010 are input to the multiplier module X620088.
  • the outputs of the modules X520086 and X620088 are input to the adder (U1020112).
  • a divider 10 is assembled on the resistors R1 20070, R2 20072, and R3 20074.
  • a nonlinear function of hyperbolic tangent (module X720090, Figure 20B) is obtained with the release of the divisor signal.
  • the output C(t) 20120 (a current state vector of the LSTM neuron) is obtained with the buffer-inverter on the U1120114 output signal.
  • the outputs of modules X120078 and X7 20090 is input to a multiplier (module X820092) whose output is input to a buffer divider by 10 on the U1220116.
  • Figure 20E shows example values for the different configurable parameters (e.g., voltages) for the circuit shown in Figures 20A-20D, according to some implementations.
  • Vdd 20058 is set to +1.5V
  • Vss 20064 is set to –1.5V
  • Vdd120060 is set to +1.8V
  • Vss120062 is set to -1.0V
  • GND 20118 is set to GND, according to some implementations.
  • Figure 20F shows a table 20132 of description for the example circuit shown in Figure 20A-20D, according to some implementations.
  • the values for the parameters are provided as examples, and various other configurations are possible.
  • the transistors U1 – U12 are CMOS OpAmps (described above in reference to Figures 19A and 19B).
  • X1, X3, and X4 are modules that perform the Sigmoid function.
  • X2 and X7 are modules that perform the Hyperbolic Tangent function.
  • X5 and X8 are modules that perform the multiplication function.
  • FIGS 21A-21I show a schematic diagram of a multiplier block 21000, according to some implementations.
  • the neuron 21000 is based on the principle of a four- quadrant multiplier, assembled using operational amplifiers U121040 and U221042 (shown in Figure 21B), U321044 (shown in Figure 21H), and U421046 and U521048 (shown in Figure 21I), and CMOS transistors M121052 through M6821182.
  • the inputs of the multiplier include V_one 21020 21006 and V_two 21008 (shown in Figure 21B), and contact Vdd (positive supply voltage, e.g., +1.5 V relative to GND) 21004 and contact Vss (negative supply voltage, e.g., -1.5 V relative to GND) 21002.
  • additional supply voltages are used: contact Input Vdd1 (positive supply voltage, e.g., +1.8 V relative to GND), contact Vss1 (negative supply voltage, e.g., -1.0 V relative to GND).
  • the result of the circuit calculations are output at mult_out (output pin) 21170 (shown in Figure 21I).
  • input signal (V_one) from V_one 21006 is connected to the inverter with a single gain made on U121040, the output of which forms a signal negA 21006, which is equal in amplitude, but the opposite sign with the signal V_one.
  • the signal (V_two) from the input V_two 21008 is connected to the inverter with a single gain made on U221042, the output of which forms a signal negB 21012 which is equal in amplitude, but the opposite sign with the signal V_two. Pairwise combinations of signals from possible combinations (V_one, V_two, negA, negB) are output to the corresponding mixers on CMOS transistors.
  • V_two 21008 and negA 21010 are input to a multiplexer assembled on NMOS transistors M1921086, M2021088, M2121090, M2221092, and PMOS transistors M2321094 and M2421096.
  • the output of this multiplexer is input to the NMOS transistor M621060 ( Figure 21D).
  • Similar transformations that occur with the signals include: • negB 21012 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M1121070, M122072, M132074, M1421076, and PMOS transistors M15 2078 and M1621080.
  • the output of this multiplexer is input to the M521058 NMOS transistor (shown in Figure 21D); • V_one 21020 and negB 21012 are input to a multiplexer assembled on PMOS transistors M18 21084, M48 21144, M49 21146, and M50 21148, and NMOS transistors M1721082, M4721142.
  • the output of this multiplexer is input to the M9 PMOS transistor 21066 (shown in Figure 21D); • negA 21010 and V_two 21008 are input to a multiplexer assembled on PMOS transistors M52 21152, M54 21156, M55 21158, and M56 21160, and NMOS transistors M5121150, and M5321154.
  • the output of this multiplexer is input to the M2 NMOS transistor 21054 (shown in Figure 21C); • negB 21012 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M11 21070, M12 21072, M13 21074, and M14 21076, and PMOS transistors M1521078, and M1621080.
  • the output of this multiplexer is input to the M10 NMOS transistor 21068 (shown in Figure 21D); • negB 21012 and negA 21010 are input to a multiplexer assembled on NMOS transistors M35 21118, M3621120, M3721122, and M38 21124, and PMOS transistors M39 21126, and M4021128.
  • the output of this multiplexer is input to the M27 PMOS transistor 21102 (shown in Figure 21H); • V_two 21008 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M41 21130, M42 21132, M43 21134, and M44 21136, and PMOS transistors M4521138, and M4621140.
  • the output of this multiplexer is input to the M30 NMOS transistor 21108 (shown in Figure 21H); • V_one 21020 and V_two 21008 are input to a multiplexer assembled on PMOS transistors M58 21162, M60 21166, M61 21168, and M62 21170, and NMOS transistors M5721160, and M5921164.
  • the output of this multiplexer is input to the M34 PMOS transistor 21116 (shown in Figure 21H); and • negA 21010 and negB 21012 are input to a multiplexer assembled on PMOS transistors M6421174, M6621178, M6721180, and M6821182, and NMOS transistors M63 21172, and M6521176.
  • the output of this multiplexer is input to the PMOS transistor M3321114 (shown in Figure 21H).
  • the current mirror powers the portion of the four quadrant multiplier circuit shown on the left, made with transistors M521058, M621060, M721062, M821064, M921066, and M1021068.
  • Current mirrors on transistors M2521098, M2621100, M2721102, and M2821104) power supply of the right portion of the four-quadrant multiplier, made with transistors M2921106, M3021108, M31 21110, M3221112, M3321114, and M3421116.
  • the multiplication result is taken from the resistor Ro 21022 enabled in parallel to the transistor M321054 and the resistor Ro 21188 enabled in parallel to the transistor M2821104, supplied to the adder on U321044.
  • the output of U321044 is supplied to an adder with a gain of 7,1, assembled on U521048, the second input of which is compensated by the reference voltage set by resistors R121024 and R221026 and the buffer U421046, as shown in Figure 21I.
  • the multiplication result is output via the Mult_Out output 21170 from the output of U521048.
  • Figure 21J shows a table 21198 of description for the schematic shown in Figures 21A-21I, according to some implementations.
  • U1 – U5 are CMOS OpAmps.
  • Example Scheme of a Sigmoid Block [00343]
  • Figure 22A shows a schematic diagram of a sigmoid block 2200, according to some implementations.
  • the sigmoid function (e.g., modules X120078, X320082, and X4 20084, described above in reference to Figures 20A-20F) is implemented using operational amplifiers U12250, U22252, U32254, U42256, U52258, U62260, U7, 2262, and U82264, and NMOS transistors M12266, M22268, and M32270.
  • Contact sigm_in 2206 is module input
  • contact Input Vdd12222 is positive supply voltage +1.8 V relative to GND 2208
  • contact Vss12204 is negative supply voltage -1.0 V relative to GND.
  • U42256 has a reference voltage source of -0.2332 V, and the voltage is set by the divider R102230 and R112232.
  • the U52258 has a reference voltage source of 0.4 V, and the voltage is set by the divider R122234 and R132236.
  • the U62260 has a reference voltage source of 0.32687 V, the voltage is set by the divider R142238 and R152240.
  • the U72262 has a reference voltage source of -0.5 V, the voltage is set by the divider R162242 and R172244.
  • the U82264 has a reference voltage source of -0.33 V, the voltage is set by the divider R182246 and R192248.
  • the sigmoid function is formed by adding the corresponding reference voltages on a differential module assembled on the transistors M1 2266 and M2 2268.
  • a current mirror for a differential stage is assembled with active regulation operational amplifier U32254, and the NMOS transistor M32270.
  • the signal from the differential stage is removed with the NMOS transistor M2 and resistor R52220 is input to the adder U22252.
  • the output signal sigm_out 2210 is removed from the U2 adder 2252 output.
  • Figure 22B shows a table 2278 of description for the schematic diagram shown in Figure 22A, according to some implementations.
  • U1-U8 are CMOS OpAmps.
  • FIG. 23A shows a schematic diagram of a hyperbolic tangent function block 2300, according to some implementations.
  • the hyperbolic tangent function (e.g., the modules X2 20080, and X720090 described above in reference to Figures 20A-20F) is implemented using operational amplifiers (U12312, U22314, U32316, U42318, U52320, U62322, U7 2328, and U82330) and NMOS transistors (M12332, M22334, and M32336).
  • operational amplifiers U12312, U22314, U32316, U42318, U52320, U62322, U7 2328, and U82330
  • NMOS transistors M12332, M22334, and M32336
  • contact tanh_in 2306 is module input
  • contact Input Vdd12304 is positive supply voltage +1.8 V relative to GND 2308
  • contact Vss12302 is negative supply voltage -1.0 V relative to GND.
  • U42318 has a reference voltage source of -0.1 V, the voltage set by the divider R102356 and R112358.
  • the U52320 has a reference voltage source of 1.2 V, the voltage set by the divider R122360 and R132362.
  • the U62322 has a reference voltage source of 0.32687 V, the voltage set by the divider R142364 and R152366.
  • the U72328 has a reference voltage source of -0.5 V, the voltage set by the divider R162368 and R172370.
  • the U82330 has a reference voltage source of -0.33 V, the voltage set by the divider R182372 and R19 2374.
  • the hyperbolic tangent function is formed by adding the corresponding reference voltages on a differential module made on transistors M12332 and M22334.
  • a current mirror for a differential stage is obtained with active regulation operational amplifier U3 2316, and NMOS transistor M32336.
  • With NMOS transistor M22334 and resistor R5 2346, the signal is removed from the differential stage and input to the adder U22314.
  • the output signal tanh_out 2310 is removed from the U2 adder 2314 output.
  • Figure 23B shows a table 2382 of description for the schematic diagram shown in Figure 23A, according to some implementations.
  • FIGS. 24A-24C show a schematic diagram of a single neuron OP1 CMOS OpAmp 2400, according to some implementations. The example is a variant of a single neuron on an operational amplifier, made on CMOS according to an OP1 scheme described herein.
  • contacts V12410 and V22408 are inputs of a single neuron
  • contact bias 2406 is voltage +0.4 V relative to GND
  • contact Input Vdd 2402 is positive supply voltage +5.0 V relative to GND
  • contact Vss 2404 is GND
  • contact Out 2474 is output of a single neuron.
  • Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel), and W (the width of the gate channel).
  • This Op Amp has two current mirrors. The current mirror on NMOS transistors M32420, M62426, and M132440 provides the offset current of the differential pair on NMOS transistors M22418 and M52424.
  • the current mirror in the PMOS transistors M72428, M82430, and M152444 provides the offset current of the differential pair on the PMOS transistors M92432 and M102434.
  • NMOS transistors M22418 and M52424 are amplifying, and PMOS transistors M12416 and M42422 play the role of active current load.
  • the signal is output to the PMOS gate of the transistor M132440.
  • the signal is output to the right input of the second differential amplifier stage on PMOS transistors M92432 and M102434.
  • NMOS transistors M112436 and M12 2438 play the role of active current load for the M92432 and M102434 transistors.
  • the M17 2448 transistor is switched on according to the scheme with a common source for a positive half-wave of the signal.
  • the M182450 transistor is switched on according to the scheme with a common source for the negative half-wave of the signal.
  • an inverter on the M172448 and M182450 transistors is enabled at the output of the M132440 and M142442 transistors.
  • Figure 24D shows a table 2476 of description for the schematic diagram shown in Figure 24A-24C, according to some implementations.
  • FIGS 25A-25D show a schematic diagram of a variant of a single neuron 25000 on operational amplifiers, made on CMOS according to an OP3 scheme, according to some implementations.
  • the single neuron consists of three simple operational amplifiers (OpAmps), according to some implementations.
  • Transistors M125028 – M16 25058 are used for summation of negative connections of the neuron.
  • Transistors M1725060 – M3225090 are used for adding the positive connections of the neuron.
  • the RELU activation function is performed on the transistors M3325092 – M4625118.
  • contacts V1 25008 and V225010 are inputs of the single neuron
  • contact bias 25002 is voltage +0.4 V relative to GND
  • contact Input Vdd 25004 is positive supply voltage +2.5 V relative to GND
  • contact Vss 25006 is negative supply voltage -2.5 V
  • contact Out 25134 is output of the single neuron.
  • Parameters of CMOS transistors used in a single neuron are determined by the ratio of geometric dimensions: L (the length of the gate channel) and W (the width of the gate channel).
  • the current mirror on NMOS transistors M325032 (M1925064, M35 25096), M625038 (M2225070, M3825102) and M1625058 (M3225090, M4825122) provides the offset current of the differential pair on NMOS transistors M225030 (M1825062, M34 25094) and M525036 (M2125068, M3525096).
  • PMOS transistors M7 25040 M2325072, M3925104
  • M825042 M2425074, M4025106
  • M1525056 M312588
  • NMOS transistors M225030 M1825062, M3425094
  • M525036 M21 25068, M37 25100
  • PMOS transistors M1 25028 M17 25060, M33 25092
  • M425034 M2025066, M3625098
  • the signal is input to the PMOS gate of the transistor M1325052 (M2925084, M4525116).
  • the signal is input to the right input of the second differential amplifier stage on PMOS transistors M925044 (M2525076, M4125108) and M1025046 (M2625078, M42 25110).
  • NMOS transistors M1125048 (M2725080, M4325112) and M1225048 (M2825080, M44 25114) play the role of active current load for transistors M925044 (M2525076, M41 25108) and M1025046 (M2625078, M4225110).
  • Transistor M1325052 (M2925082, M45 25116) is included in the scheme with a common source for a positive half-wave signal.
  • the transistor M14 25054 (M3025084, M4625118) is switched on according to the scheme with a common source for the negative half-wave of the signal.
  • R feedback 100k – used only for calculating w1, w2, wbias.
  • FIG. 25E shows a table 25136 of description for the schematic diagram shown in Figure 25A-25D, according to some implementations.
  • FIGS 27A-27J show a flowchart of a method 2700 for hardware realization (2702) of neural networks, according to some implementations.
  • the method is performed (2704) at the computing device 200 (e.g., using the neural network transformation module 226) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (2706) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • the trained neural network is trained (2708) using software simulations to generate the weights.
  • the method also includes transforming (2710) the neural network topology to an equivalent analog network of analog components.
  • the neural network topology includes (2724) one or more layers of neurons. Each layer of neurons computing respective outputs based on a respective mathematical function.
  • transforming the neural network topology to the equivalent analog network of analog components includes, performing (2726) a sequence of steps for each layer of the one or more layers of neurons.
  • the sequence of steps include identifying (2728) one or more function blocks, based on the respective mathematical function, for the respective layer.
  • Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function.
  • identifying the one or more function blocks includes selecting (2730) the one or more function blocks based on a type of the respective layer.
  • ⁇ ⁇ represents an i-th input and ⁇ ⁇ represents a j-th input, and ⁇ ⁇ ⁇ ⁇ ⁇ is a predetermined coefficient;
  • a hyperbolic tangent activation block (2742) with a block output ⁇ ⁇ ⁇ ⁇ tanh ( ⁇ ⁇ ⁇ ⁇ ).
  • ⁇ represents a current time-period
  • ⁇ ( ⁇ ⁇ 1) represents an output of the signal delay block for a preceding time period ⁇ ⁇ 1
  • ⁇ ⁇ is a delay value.
  • Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multilayer network is connected to one or more analog neurons of a second layer of the multilayer network.
  • transforming (2710) the neural network topology to an equivalent analog network of analog components requires more complex processing, according to some implementations.
  • Figure 27E suppose the neural network topology includes (2746) one or more layers of neurons. Suppose further that each layer of neurons computes respective outputs based on a respective mathematical function.
  • transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing (2748) a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function.
  • the mathematical function corresponding to the first layer includes one or more weights
  • decomposing the mathematical function includes adjusting (2750) the one or more weights such that combining the one or more intermediate functions results in the mathematical function.
  • the sequence of steps includes selecting (2754) one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and generating (2756) a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks.
  • Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network.
  • transforming the neural network topology includes generating (2770) one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.
  • an external cycle timer activates the one or more signal delay blocks with a constant time period (e.g., 1, 5, or 10 time steps). Some implementations use multiple delay blocks over one signal for producing additive time shift.
  • the activation frequency of the one or more signal delay blocks is/are synchronized to network input signal frequency.
  • the one or more signal delay blocks are activated (2772) at a frequency that matches a predetermined input signal frequency for the neural network topology.
  • this predetermined input signal frequency may be dependent on the application, such as Human Activity Recognition (HAR) or PPG.
  • HAR Human Activity Recognition
  • the predetermined input signal frequency is 30-60 Hz for video processing, around 100 Hz for HAR and PPG, 16 KHz for sound processing, and around 1-3 Hz for battery management.
  • Some implementations activate different signal delay blocks activate at different frequencies.
  • transforming the neural network topology includes applying (2776) one or more transformations selected from the group consisting of: replacing (2778) the unlimited activation functions with limited activation (e.g., replacing ReLU with a threshold ReLU); and adjusting (2780) connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized.
  • the method also includes computing (2712) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network.
  • the method also includes generating (2714) a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components.
  • generating the schematic model includes generating (2716) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the method includes regenerating just the resistance matrix for the resistors for a retrained network.
  • the method further includes obtaining (2718) new weights for the trained neural network, computing (2720) a new weight matrix for the equivalent analog network based on the new weights, and generating (2722) a new resistance matrix for the new weight matrix.
  • the method further includes generating (2782) one or more lithographic masks (e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix.
  • the method includes regenerating just the masks for resistors (e.g., the masks 250) for retrained networks.
  • the method further includes: (i) obtaining (2784) new weights for the trained neural network; (ii) computing (2786) a new weight matrix for the equivalent analog network based on the new weights; (iii) generating (2788) a new resistance matrix for the new weight matrix; and (iv) generating (2790) a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix.
  • the analog components include (2762) a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • Some implementations include other analog components, such as four-quadrant multipliers, sigmoid and hyperbolic tangent function circuits, delay lines, summers, and/or dividers.
  • selecting (2764) component values of the analog components includes performing (2766) a gradient descent method and/or other weight quantization methods to identify possible resistance values for the plurality of resistors.
  • the method further includes implementing certain activation functions (e.g., Softmax) in output layer in digital.
  • the method further includes generating (2758) equivalent digital network of digital components for one or more output layers of the neural network topology, and connecting (2760) output of one or more layers of the equivalent analog network to the equivalent digital network of digital components.
  • Example Methods for Constrained Analog Hardware Realization of Neural Networks [00366]
  • Figures 28A-28S show a flowchart of a method 28000 for hardware realization (28002) of neural networks according to hardware design constraints, according to some implementations The method is performed (28004) at the computing device 200 (e.g., using the neural network transformation module 226) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (28006) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • the method also includes calculating (28008) one or more connection constraints based on analog integrated circuit (IC) design constraints (e.g., the constraints 236).
  • IC design constraints can set the current limit (e.g., 1A), and neuron schematics and operational amplifier (OpAmp) design can set the OpAmp output current in the range [0- 10mA], so this limits output neuron connections to 100.
  • the method also includes transforming (28010) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints.
  • transforming the neural network topology includes deriving (28012) a possible input connection degree and output connection degree ⁇ ⁇ , according to the one or more connection constraints.
  • the neural network topology includes (28018) at least one densely connected layer with ⁇ inputs (neurons in previous layer) and ⁇ outputs (neurons in current layer) and a weight matrix ⁇
  • transforming (28020) the at least one densely connected layer includes constructing (28022) the equivalent sparsely connected network with ⁇ inputs, ⁇ outputs, and ⁇ ⁇ ⁇ ⁇ + ⁇ ⁇ ⁇ ⁇ ⁇ 1 layers, such that input connection degree does not exceed ⁇ , and output connection degree does not exceed ⁇ ⁇ .
  • the neural network topology includes (28024) at least one densely connected layer with ⁇ inputs (neurons in previous layer) and ⁇ outputs (neurons in current layer) and a weight matrix ⁇ , and transforming (28026) the at least one densely connected layer includes: constructing (28028) the equivalent sparsely connected network with ⁇ inputs, ⁇ outputs, and ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ layers.
  • Each layer m is represented by a corresponding weight matrix ⁇ ⁇ , where absent connections are represented with zeros, such that input connection degree does not exceed ⁇ ⁇ , and output connection degree does not exceed ⁇ ⁇ .
  • the equation ⁇ ⁇ ⁇ .. ⁇ ⁇ ⁇ is satisfied with a predetermined precision.
  • the predetermined precision is a reasonable precision value that statistically guarantees that altered networks output differs from referent network output by no more than allowed error value, and this error value is task- dependent (typically between 0.1% and 1%).
  • the neural network topology includes (28030) a single sparsely connected layer with ⁇ inputs and ⁇ outputs, a maximum input connection degree of ⁇ ⁇ , a maximum output connection degree of ⁇ ⁇ , and a weight matrix of ⁇ , where absent connections are represented with zeros.
  • transforming (28032) the single sparsely connected layer includes constructing (28034) the equivalent sparsely connected network with ⁇ inputs, ⁇ outputs, ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ layers.
  • the neural network topology includes (28036) a convolutional layer (e.g., a Depthwise convolutional layer, or a Separable convolutional layer) with ⁇ inputs (neurons in previous layer) and ⁇ outputs (neurons in current layer).
  • a convolutional layer e.g., a Depthwise convolutional layer, or a Separable convolutional layer
  • transforming (28038) the neural network topology to the equivalent sparsely connected network of analog components includes decomposing (28040) the convolutional layer into a single sparsely connected layer with ⁇ inputs, ⁇ outputs, a maximum input connection degree of ⁇ ⁇ , and a maximum output connection degree of ⁇ ⁇ , [00374]
  • the method also includes computing (28014) a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network.
  • the neural network topology includes (28042) a recurrent neural layer, and transforming (28044) the neural network topology to the equivalent sparsely connected network of analog components includes transforming (28046) the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections.
  • the neural network topology includes a recurrent neural layer (e.g., a long short-term memory (LSTM) layer or a gated recurrent unit (GRU) layer), and transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with ⁇ inputs (neurons in previous layer) and ⁇ outputs (neurons in current layer) and a weight matrix ⁇ , where absent connections are represented with zeros.
  • LSTM long short-term memory
  • GRU gated recurrent unit
  • the method includes performing a transformation of a single layer perceptron with one calculation neurons.
  • the neural network topology includes (28054) K inputs, a weight vector ⁇ ⁇ ⁇ ⁇ , and a single layer perceptron with a calculation neuron with an activation function F.
  • the equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers.
  • each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron.
  • computing (28064) the weight matrix for the equivalent sparsely connected network includes calculating (28066) a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U.
  • the method includes performing a transformation of a single layer perceptron with L calculation neurons.
  • the neural network topology includes (28068) K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network (28078) of the L single layer perceptron networks, constructing (28080) a respective equivalent pyramid- like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (v) constructing (28082) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs.
  • the method includes performing a transformation algorithm for multi-layer perceptron.
  • the neural network topology includes (28092) K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons Li and corresponding weight matrices V i that includes a row of weights for each calculation neuron of the L i calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the ⁇ calculation neurons.
  • K i, j is number of inputs for the respective calculation neuron in the multi-layer perceptron, and (b) constructing (28104) the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K i, j inputs, the m layers and the connection degree N.
  • the equivalent pyramid- like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing (28106) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K i, j inputs.
  • the neural network topology includes (28116) a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons L i and corresponding weight matrices V i that includes a row of weights for each calculation neuron of the Li calculation neurons.
  • CNN Convolutional Neural Network
  • Each single layer perceptron network includes a respective calculation neuron of the ⁇ calculation neurons.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing (28130) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub- network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K i, j inputs.
  • the method includes transforming two layers to trapezium-based network.
  • performing the trapezium transformation further includes: in accordance with a determination that ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ + ⁇ ⁇ ⁇ ⁇ : (i) splitting (28154) the layer Lp to obtain a sub-layer Lp1 with K’ neurons and a sub-layer L p2 with (K - K’) neurons such that ⁇ ′ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ + ⁇ ′ ⁇ ⁇ ⁇ ; (ii) for the sub-layer L p1 with K’ neurons, performing (28156) the constructing, and generating steps; and (iii) for the sub-layer Lp2 with K - K’ neurons, recursively
  • the method includes transforming multilayer perceptron to trapezium-based network.
  • the neural network topology includes (28160) a multilayer perceptron network, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing (28162) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the method includes transforming recurrent neural network to trapezium-based network.
  • the neural network topology includes (28164) a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non- linear function calculation.
  • the method further includes performing (28166) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation.
  • Element-wise addition is a common operation that can be implemented in networks of any structure, examples of which are provided above.
  • Non-linear function calculation is a neuron-wise operation that is independent of the No and Ni restrictions, and are usually calculated with ‘sigmoid’ or ‘tanh’ block on each neuron separately.
  • the neural network topology includes (28168) a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations (sigmoid and hyperbolic tangent operations).
  • LSTM long short-term memory
  • GRU gated recurrent unit
  • the method further includes performing (28170) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations.
  • Element-wise addition and Hadamard products are common operations that can be implemented in networks of any structure described above.
  • the neural network topology includes (28172) a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers (e.g., sequence of convolutional and pooling layers; each pooling layer is assumed to be a convolutional later with stride larger than 1) and (ii) one or more fully-connected layers (the sequence ends in the fully-connected layers).
  • CNN convolutional neural network
  • the method further includes (i) transforming (28174) the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing (28176) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the neural network topology includes (28178) K inputs, L output neurons, and a weight matrix ⁇ ⁇ ⁇ ⁇ , where R is the set of real numbers, each output neuron performs an activation function F.
  • each neuron in the pyramid neural network performs identity function; and (iv) constructing (28188) a trapezium neural network with ⁇ ⁇ inputs and ⁇ outputs. Each neuron in the last layer of the trapezium neural network performs the activation function F and all other neurons perform identity function.
  • transforming (28198) the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer ⁇ (28200) of the ⁇ layers of the multilayer perceptron, constructing (28202) a respective pyramid- trapezium network PTNNXj by performing the approximation transformation to a respective single layer perceptron consisting of ⁇ ⁇ inputs, ⁇ ⁇ output neurons, and a weight matrix ⁇ ⁇ ; and (ii) constructing (28204) the equivalent sparsely connected network by stacking each pyramid trapezium network (e.g., output of a pyramid trapezium network PTNNXj-1 is set as an input for PTNNXj).
  • each pyramid trapezium network e.g., output of a pyramid trapezium network PTNNXj-1 is set as an input for PTNNXj.
  • the method further includes generating (28016) a schematic model for implementing the equivalent sparsely connected network utilizing the weight matrix.
  • Example Methods of Calculating Resistance Values for Analog Hardware Realization of Trained Neural Networks [00390]
  • Figures 29A-29F show a flowchart of a method 2900 for hardware realization (2902) of neural networks according to hardware design constraints, according to some implementations. The method is performed (2904) at the computing device 200 (e.g., using the weight quantization module 238) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (2906) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • weight quantization is performed during training.
  • the trained neural network is trained (2908) so that each layer of the neural network topology has quantized weights (e.g., a particular value from a list of discrete values; e.g., each layer has only 3 weight values of +1, 0, -1).
  • the method also includes transforming (2910) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. [00393] The method also includes computing (2912) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. [00394] The method also includes generating (2914) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • generating the resistance matrix for the weight matrix includes a simplified gradient-descent based iterative method to find a resistor set.
  • generating the resistance matrix for the weight matrix includes: (i) obtaining (2916) a predetermined range of possible resistance values ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ and selecting an initial base resistance value ⁇ ⁇ within the predetermined range. For example, the range and the base resistance are selected according to values of elements of the weight matrix; the values are determined by the manufacturing process; ranges - resistors that can be actually manufactured; large resistors are not preferred; quantization of what can be actually manufactured.
  • the predetermined range of possible resistance values includes (2918) resistances according to nominal series E24 in the range 100 K ⁇ to 1 M ⁇ ; (ii) selecting (2920) a limited length set of resistance values, within the predetermined range, that provide most uniform distribution of possible weights within the range [ ⁇ ⁇ ⁇ , ⁇ ⁇ ] for all combinations of ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ within the limited length set of resistance values.
  • ⁇ ⁇ and ⁇ ⁇ are chosen (2924) independently for each layer of the equivalent analog network.
  • ⁇ ⁇ and ⁇ ⁇ are chosen (2926) independently for each analog neuron of the equivalent analog network; and (iv) for each element of the weight matrix, selecting (2928) a respective first resistance value ⁇ ⁇ and a respective second resistance value ⁇ ⁇ that minimizes an error according to equation all possible values of ⁇ ⁇ and ⁇ within the ⁇ predetermined range of possible resistance values.
  • w is the respective element of the weight matrix, and ⁇ ⁇ is a predetermined relative tolerance value for the possible resistance values.
  • a first one or more weights of the weight matrix and a first one or more inputs represent (2930) one or more connections to a first operational amplifier of the equivalent analog network.
  • the method further includes: prior to generating (2932) the resistance matrix, (i) modifying (2934) the first one or more weights by a first value (e.g., dividing the first one or more weights by the first value to reduce weight range, or multiplying the first one or more weights by the first value to increase weight range); and (ii) configuring (2936) the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function.
  • a first value e.g., dividing the first one or more weights by the first value to reduce weight range, or multiplying the first one or more weights by the first value to increase weight range
  • configuring (2936) the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the
  • Some implementations perform the weight reduction so as to change multiplication factor of one or more operational amplifiers.
  • the resistor values set produce weights of some range, and in some parts of this range the error will be higher than in others.
  • there are only 2 nominals (e.g., 1 ⁇ and 4 ⁇ ) these resistors can produce weights [-3; -0.75; 0; 0.75; 3].
  • the first layer of a neural network has weights of ⁇ 0, 9 ⁇ and the second layer has weights of ⁇ 0, 1 ⁇
  • some implementations divide the first layer’s weights by 3 and multiply the second layer’s weights by 3 to reduce overall error.
  • the method further includes restricting weights to intervals.
  • the method further includes obtaining (2938) a predetermined range of weights, and updating (2940) the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input.
  • the method further includes reducing weight sensitivity of network.
  • the method further includes retraining (2942) the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network.
  • some implementations include additional training for an already trained neural network in order to give it less sensitivity to small randomly distributed weight errors. Quantization and resistor manufacturing produce small weight errors.
  • Some implementations transform networks so that the resultant network is less sensitive to each particular weight value. In some implementations, this is performed by adding a small relative random value to each signal in at least some of the layers during training (e.g., similar to a dropout layer). [00399] Referring next to Figure 29F, some implementations include reducing weight distribution range.
  • Some implementations include retraining (2944) the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. Some implementations perform this step via retraining.
  • Example penalty function include a sum over all layers (e.g., A * max(abs(w)) / mean(abs(w)), where max and mean are calculated over a layer. Another example include order of magnitude higher and above. In some implementations, this function impacts weight quantization and network weight sensitivity. For e.g., small relative changes of weights due to quantization might cause high output error.
  • Example techniques include introducing some penalty functions during training that penalize network when it has such weight outcasts.
  • Figures 30A-30M show a flowchart of a method 3000 for hardware realization (3002) of neural networks according to hardware design constraints, according to some implementations.
  • the method is performed (3004) at the computing device 200 (e.g., using the analog neural network optimization module 246) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (3006) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • the method also includes transforming (3008) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • the method further includes pruning the trained neural network. In some implementations, the method further includes pruning (3052) the trained neural network to update the neural network topology and the weights of the trained neural network, prior to transforming the neural network topology, using pruning techniques for neural networks, so that the equivalent analog network includes less than a predetermined number of analog components.
  • the pruning is performed (3054) iteratively taking into account accuracy or a level of match in output between the trained neural network and the equivalent analog network.
  • the method further includes, prior to transforming the neural network topology to the equivalent analog network, performing (3056) network knowledge extraction.
  • Knowledge extraction is unlike stochastic/learning like pruning, but more deterministic than pruning.
  • knowledge extraction is performed independent of the pruning step.
  • connection weights are adjusted according to predetermined optimality criteria (such as preferring zero weights, or weights in a particular range, over other weights) through methods of knowledge extraction, by derivation of causal relationships between inputs and outputs of hidden neurons.
  • predetermined optimality criteria such as preferring zero weights, or weights in a particular range, over other weights
  • connection weights are adjusted according to predetermined optimality criteria (such as preferring zero weights, or weights in a particular range, over other weights) through methods of knowledge extraction, by derivation of causal relationships between inputs and outputs of hidden neurons.
  • predetermined optimality criteria such as preferring zero weights, or weights in a particular range, over other weights
  • the method also includes computing (3010) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.
  • the method further includes removing or transforming neurons based on bias values.
  • the method further includes, for each analog neuron of the equivalent analog network: (i) computing (3044) a respective bias value for the respective analog neuron based on the weights of the trained neural network, while computing the weight matrix; (ii) in accordance with a determination that the respective bias value is above a predetermined maximum bias threshold, removing (3046) the respective analog neuron from the equivalent analog network; and (iii) in accordance with a determination that the respective bias value is below a predetermined minimum bias threshold, replacing (3048) the respective analog neuron with a linear junction in the equivalent analog network.
  • the method further includes minimizing number of neurons or compacting the network.
  • the method further includes reducing (3050) number of neurons of the equivalent analog network, prior to generating the weight matrix, by increasing number of connections (inputs and outputs) from one or more analog neurons of the equivalent analog network.
  • the method also includes generating (3012) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix.
  • the method also includes pruning (3014) the equivalent analog network to reduce number of the plurality of operational amplifiers or the plurality of resistors, based on the resistance matrix, to obtain an optimized analog network of analog components.
  • the method includes substituting insignificant resistances with conductors.
  • pruning the equivalent analog network includes substituting (3016), with conductors, resistors corresponding to one or more elements of the resistance matrix that have resistance values below a predetermined minimum threshold resistance value. [00411] Referring next to Figure 30C, in some implementations, the method further includes removing connections with very high resistances. In some implementations, pruning the equivalent analog network includes removing (3018) one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value. [00412] Referring next to Figure 30D, in some implementations, pruning the equivalent analog network includes removing (3020) one or more connections of the equivalent analog network corresponding to one or more elements of the weight matrix that are approximately zero.
  • pruning the equivalent analog network further includes removing (3022) one or more analog neurons of the equivalent analog network without any input connections.
  • the method includes removing unimportant neurons.
  • pruning the equivalent analog network includes (i) ranking (3024) analog neurons of the equivalent analog network based on detecting use of the analog neurons when making calculations for one or more data sets. For example, training data set used to train the trained neural network; typical data sets; data sets developed for pruning procedure. Some implementations perform ranking of neurons for pruning based on frequency of use of given neuron or block of neurons when subjected to training data set.
  • detecting use of the analog neurons includes: (i) building (3030) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring (3032) propagation of analog signals (currents) by using the model (remove the blocks where the signal is not propagating when using special training sets) to generate calculations for the one or more data sets.
  • a modelling software e.g., SPICe or similar software
  • detecting use of the analog neurons includes: (i) building (3034) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring (3036) output signals (currents or voltages) of the model (e.g., signals at outputs of some blocks or amplifiers in SPICe model or in real circuit, and deleting the areas where output signal for training set is always zero volts) by using the model to generate calculations for the one or more data sets.
  • a modelling software e.g., SPICe or similar software
  • measuring (3036) output signals (currents or voltages) of the model e.g., signals at outputs of some blocks or amplifiers in SPICe model or in real circuit, and deleting the areas where output signal for training set is always zero volts
  • detecting use of the analog neurons includes: (i) building (3038) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring (3040) power consumed by the analog neurons (e.g., power consumed by certain neurons or blocks of neurons, represented by operational amplifiers either in a SPICE model or in real circuit and deleting the neurons or blocks of neurons which did not consume any power) by using the model to generate calculations for the one or more data sets.
  • a modelling software e.g., SPICe or similar software
  • the method further includes, subsequent to pruning the equivalent analog network, and prior to generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network, recomputing (3042) the weight matrix for the equivalent analog network and updating the resistance matrix based on the recomputed weight matrix.
  • Example Analog Neuromorphic Integrated Circuits and Fabrication Methods Example Methods for Fabricating Analog Integrated Circuits for Neural Networks
  • Figures 31A-31Q show a flowchart of a method 3100 for fabricating an integrated circuit 3102 that includes an analog network of analog components, according to some implementations.
  • the method is performed at the computing device 200 (e.g., using the IC fabrication module 258) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (3104) a neural network topology and weights of a trained neural network. [00419]
  • the method also includes transforming (3106) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (for recurrent neural networks, also use signal delay lines, multipliers, Tanh analog block, Sigmoid Analog Block).
  • Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron.
  • the method also includes computing (3108) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.
  • the method also includes generating (3110) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix.
  • the method also includes generating (3112) one or more lithographic masks (e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix, and fabricating (3114) the circuit (e.g., the ICs 262) based on the one or more lithographic masks using a lithographic process.
  • generating (3112) one or more lithographic masks e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix
  • fabricating (3114) the circuit e.g., the ICs 262
  • the integrated circuit further includes one or more digital to analog converters (3116) (e.g., the DAC converters 260) configured to generate analog input for the equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD/CMOS image sensors).
  • the integrated circuit further includes an analog signal sampling module (3118) configured to process 1-dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit (number of inferences for the IC is determined by product Spec - we know sampling rate from Neural Network operation and exact task the chip is intended to solve).
  • the integrated circuit further includes a voltage converter module (3120) to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers.
  • the integrated circuit further includes a tact signal processing module (3122) configured to process one or more frames obtained from a CCD camera.
  • the trained neural network is a long short-term memory (LSTM) network, AND the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing.
  • LSTM long short-term memory
  • the integrated circuit further includes one or more analog to digital converters (3126) (e.g., the ADC converters 260) configured to generate digital signal based on output of the equivalent analog network of analog components.
  • the integrated circuit includes one or more signal processing modules (3128) configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications.
  • the trained neural network is trained (3130), using training datasets containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected (in other words, the operation of trained chip is used to determine each of known to neural network gases in the gas mixture individually, despite the presence of other gases in the mixture).
  • the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes (3132) 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers.
  • the equivalent analog network includes (3134): (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections.
  • the trained neural network is trained (3136), using training datasets containing thermal aging time series data for different MOSFETs (e.g., NASA MOSFET dataset that contains thermal aging time series for 42 different MOSFETs; data is sampled every 400 ms and typically several hours of data for each device), for predicting remaining useful life (RUL) of a MOSFET device.
  • the neural network topology includes (3138) 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively.
  • the equivalent analog network includes (3140): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons (e.g., 3137 analog neurons), and (v) between 123,000 and 124,000 connections (e.g., 123,200 connections).
  • the trained neural network is trained (3142), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery ), for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes (3144) an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values.
  • the equivalent analog network includes (3146): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons (e.g., 1271 analog neurons), and (v) between 51,000 and 52,000 connections (e.g., 51,776 connections).
  • the trained neural network is trained (3148), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery ), for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes (3150) an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron.
  • the equivalent analog network includes (3152): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons (e.g., 201 analog neurons), and (v) between 2,200 and 2,400 connections (e.g., 2,300 connections).
  • the trained neural network is trained (3154), using training datasets containing speech commands (e.g., Google Speech Commands Dataset), for identifying voice commands (e.g., 10 short spoken keywords, including “yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”).
  • the neural network topology is (3156) a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron.
  • the equivalent analog network includes (3158): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections.
  • the trained neural network is trained (3160), using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor (e.g., PPG data from the PPG-Dalia dataset (CHECK LICENSE). Data is collected for 15 individuals performing various physical activities during 1-4 hours each.
  • Wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electrodermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from ECG sensor with sampling around 2 Hz.
  • Original data was split into sequences of 1000 timesteps (around 15 seconds), with a shift of 500 timesteps, thus getting 16541 samples total.
  • Dataset was split into 13233 training samples and 3308 test samples), for determining pulse rate during physical exercises (e.g., jogging, fitness exercises, climbing stairs) based on PPG sensor data and 3-axis accelerometer data.
  • the neural network topology includes (3162) two Conv1D layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively.
  • the equivalent analog network includes (3164): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons (e.g., 713 analog neurons), and (vi) between 12,000 and 12,500 connections (e.g., 12,072 connections). [00436] Referring next to Figure 31O, the trained neural network is trained (3166) to classify different objects (e.g., humans, cars, cyclists, scooters) based on pulsed Doppler radar signal (remove clutter and provide noise to Doppler radar signal), and the neural network topology includes (3168) multi-scale LSTM neural network.
  • different objects e.g., humans, cars, cyclists, scooters
  • the neural network topology includes (3168) multi-scale LSTM neural network.
  • the trained neural network is trained (3170) to perform human activity type recognition (e.g., walking, running, sitting, climbing stairs, exercising, activity tracking), based on inertial sensor data (e.g., 3-axes accelerometers, magnetometers, or gyroscope data, from fitness tracking devices, smart watches or mobile phones; 3-axis accelerometer data as input, sampled at up to 96Hz frequency.
  • inertial sensor data e.g., 3-axes accelerometers, magnetometers, or gyroscope data, from fitness tracking devices, smart watches or mobile phones; 3-axis accelerometer data as input, sampled at up to 96Hz frequency.
  • Network was trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs” etc.).
  • the neural network topology includes (3172) three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes.
  • the equivalent analog network includes (3174): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons (e.g., 1296 analog neurons), and (vi) between 20,000 and 21,000 connections (e.g., 20,022 connections).
  • the trained neural network is further trained (3176) to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation (so as to detect pre-stroke or pre- heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy, etc).
  • Some implementations include components that are not integrated into the chip (i.e., these are external elements, connected to the chip) selected from the group consisting of: voice recognition, video signal processing, image sensing, temperature sensing, pressure sensing, radar processing, LIDAR processing, battery management, MOSFET circuits current and voltage, accelerometers, gyroscopes, magnetic sensors, heart rate sensors, gas sensors, volume sensors, liquid level sensors, GPS satellite signal, human body conductance sensor, gas flow sensor, concentration sensor, pH meter, and IR vision sensors.
  • voice recognition video signal processing
  • image sensing temperature sensing
  • pressure sensing radar processing
  • LIDAR processing LIDAR processing
  • battery management MOSFET circuits current and voltage
  • accelerometers gyroscopes
  • magnetic sensors magnetic sensors
  • heart rate sensors gas sensors
  • volume sensors volume sensors
  • liquid level sensors GPS satellite signal
  • human body conductance sensor gas flow sensor
  • concentration sensor concentration sensor
  • pH meter pH meter
  • a neuromorphic IC is manufactured according to the processes described above.
  • the neuromorphic IC is based on a Deep Convolutional Neural Network trained for selective sensing of different gases in the gas mixture containing some amounts of gases to be detected.
  • the Deep Convolutional Neural Network is trained using training datasets, containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) in response to different gas mixtures.
  • the integrated circuit (or the chip manufactured according to the techniques described herein) can be used to determine one or more known gases in the gas mixture, despite the presence of other gases in the mixture.
  • the trained neural network is a Multi-label 1D- DCNN network used for Mixture Gases Classification.
  • the network is designed for detecting 3 binary gas components based on measurements by 16 gas sensors.
  • the 1D-DCNN includes sensor-wise 1D convolutional block (16 such blocks), 3 common 1D convolutional blocks, and 3 Dense layers.
  • the 1D-DCNN network performance for this task is 96.3%.
  • the resulting T-network has the following properties: 15 layers, approximately 100,000 analog neurons, approximately 4,900,000 connections.
  • Example Analog Neuromorphic IC for MOSFET failure prediction [00445] MOSFET on-resistance degradation due to thermal stress is a well-known serious problem in power electronics. In real-world applications, frequently, MOSFET device temperature changes over a short period of time. This temperature sweeps produce thermal degradation of a device, as a result of which the device might exhibit exponential. This effect is typically studied by power cycling that produces temperature gradients, which cause MOSFET degradation. [00446] In some implementations, a neuromorphic IC is manufactured according to the processes described above.
  • the neuromorphic IC is based on a network discussed in the article titled “Real-time Deep Learning at the Edge for Scalable Reliability Modeling of SI- MOSFET Power Electronics Converters” for predicting remaining useful life (RUL) of a MOSFET device.
  • the neural network can be used to determine Remaining Useful Life (RUL) of a device, with an accuracy over 80%.
  • the network is trained on NASA MOSFET Dataset which contains thermal aging timeseries for 42 different MOSFETs. Data is sampled every 400 ms and typically includes several hours of data for each device.
  • the network contains 4 LSTM layers of 64 neurons each, followed by 2 Dense layers of 64 and 1 neurons.
  • the network is T-transformed with following parameters: maximum input and output connections per neuron is 100; signal limit of 5, and the resulting T-network had following properties: 18 layers, approximately 3,000 neurons (e.g., 137 neurons), and approximately 120,000 connections (e.g., 123200 connections).
  • Example Analog Neuromorphic IC for Lithium Ion Battery Health and SoC Monitoring [00449] In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC can be used for predictive analytics of Lithium Ion batteries to use in Battery Management Systems (BMS).
  • BMS Battery Management Systems
  • BMS device typically presents such functions as overcharge and over-discharge protection, monitoring State of Health (SOH) and State of Charge (SOC), and load balancing for several cells.
  • SOH and SOC monitoring normally requires digital data processor, which adds to the cost of the device and consumes power.
  • the Integration Circuit is used to obtain precise SOC and SOH data without implementing digital data processor on the device.
  • the Integrated Circuit determines SOC with over 99% accuracy and determines SOH with over 98% accuracy.
  • network operation is based on analysis of the discharge curve of the battery, as well as temperature, and/or data is presented as a time series. Some implementations use data from NASA Battery Usage dataset.
  • the network includes an input layer, 2 LSTM layers of 64 neurons each, and an output dense layer of 2 neurons (SOC and SOH values).
  • the resulting T-network include the following properties: 9 layers, approximately 1,200 neurons (e.g., 1,271 neurons), and approximately 50,000 connections (e.g., 51,776 connections).
  • the network operation is based on analysis of the discharge curve of the battery, as well as temperature.
  • the network is trained using Network IndRnn disclosed in the paper titled “State-of-Health Estimation of Li-ion Batteries inElectric Vehicle Using IndRNN under VariableLoad Condition” designed for processing data from NASA Battery Usage dataset.
  • the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries.
  • the IndRnn network contains an input layer with 18 neurons, a simple recurrent layer of 100 neurons and a dense layer of 1 neuron. [00452]
  • the resulting T-network had following properties: 4 layers, approximately 200 neurons (e.g., 201 neurons), and approximately 2,000 connections (e.g., 2,300 connections). Some implementations output only SOH with an estimation error of 1.3%. In some implementations, the SOC is obtained similar to how the SOH is obtained.
  • Example Analog Neuromorphic IC for Keyword spotting [00453] In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC can be used for keyword spotting. [00454] The input network is a neural network with 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio mel-spectrogram of size 49 times 10.
  • the network includes 5 convolutional layers, 4 depthwise convolutional layers, an average pooling layer, and a final dense layer.
  • the networks are pre-trained to recognize 10 short spoken keywords (yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go") from Google Speech Commands Dataset, with a recognition accuracy of 94.4%.
  • the Integration Circuit is manufactured based on Depthwise Separable Convolutional Neural Network (DS-CNN) for the voice command identification.
  • DS-CNN Depthwise Separable Convolutional Neural Network
  • the resulting T-network had following properties: 13 layers, approximately 72,000 neurons, and approximately 2.6 million connections.
  • Example DS-CNN Keyword Spotting Network [00457]
  • a keyword spotting network is transformed to a T-network, according to some implementations.
  • the network is a neural network of 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio spectrogram of size 49x10.
  • Network consists of 5 convolutional layers, 4 depthwise convolutional layers, average pooling layer and final dense layer.
  • Network is pre-trained to recognize 10 short spoken keywords (yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”) from Google Speech Commands Dataset https://ai.googleblog.com/2017/08/launching-speech-commands-dataset.html. There are 2 additional classes which correspond to ‘silence’ and ‘unknown’. Network output is a softmax of length 12. [00458] The trained neural network (input to the transformation) had a recognition accuracy of 94.4%, according to some implementations. In the neural network topology, each convolutional layer is followed with BatchNorm layer and ReLU layer, and ReLU activations are unbounded, and included around 2.5 million multiply-add operations.
  • Figures 26A-26K show example histograms 2600 for absolute weights for the layers 1 through 11, respectively, according to some implementations.
  • the weight distribution histogram (for absolute weights) was calculated for each layer.
  • the dashed lines in the charts correspond to a mean absolute weight value for the respective layer.
  • the average output absolute error (calculated over test set) of converted network vs original is calculated to be 4.1e-9.
  • Various examples for setting network limitations for the transformed network are described herein, according to some implementations. For signal limit, as ReLU activations used in the network are unbounded, and some implementations use a signal limit on each layer. This could potentially affect mathematical equivalence.
  • some implementations use a signal limit of 5 on all layers which corresponds to power voltage of 5 in relation to input signal range.
  • some implementations use a nominal set of 30 resistors [0.001, 0.003, 0.01, 0.03, 0.1, 0.324, 0.353, 0.436, 0.508, 0.542, 0.544, 0.596, 0.73, 0.767, 0.914, 0.985, 0.989, 1.043, 1.101, 1.149, 1.157, 1.253, 1.329, 1.432, 1.501, 1.597, 1.896, 2.233, 2.582, 2.844].
  • Some implementations select R- and R+ values (see description above) separately for each layer.
  • Some implementations convert the output layer as follows. Output layer is a dense layer that does not have ReLU activation. The layer has softmax activation which is not implemented in T-conversion and is left for digital part, according to some implementations. Some implementations perform no additional conversion.
  • Example Analog Neuromorphic IC for Obtaining Heartrate PPG is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue.
  • a PPG is often obtained by using a pulse oximeter which illuminates the skin and measures changes in light absorption.
  • PPG is often processed to determine heart rate in devices, such as fitness trackers.
  • Deriving heart rate (HR) from PPG signal is an essential task in edge devices computing.
  • PPG data obtained from device located on wrist usually allows to obtain reliable heartrate only when the device is stable. If a person is involved in physical exercise, obtaining heartrate from PPG data produces poor results unless combined with inertial sensor data.
  • an Integrated Circuit based on combination of Convolutional Neural Network and LSTM layers, can be used to precisely determine the pulse rate, basing on the data from photoplethysmography (PPG) sensor and 3-axis accelerometer.
  • the integrated circuit can be used to suppress motion artifacts of PPG data and to determine the pulse rate during physical exercise, such as jogging, fitness exercises, and climbing stairs, with an accuracy exceeding 90%
  • the input network is trained with PPG data from the PPG-Dalia dataset. Data is collected for 15 individuals performing various physical activities for a predetermined duration (e.g., 1-4 hours each).
  • the training data included wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electro-dermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from an ECG sensor with sampling around 2 Hz.
  • the original data was split into sequences of 1000 time steps (around 15 seconds), with a shift of 500 time steps, thus producing 16541 samples total.
  • the input network included 2 Conv1D layers with 16 filters each, performing time series convolution, 2 LSTM layers of 16 neurons each, and 2 dense layers of 16 and 1 neurons.
  • the network produces MSE error of less than 6 beats per minute over the test set.
  • the resulting T-network had following properties: 15 layers, approximately 700 neurons (e.g., 713 neurons), and approximately 12,000 connections (e.g., 12072 connections).
  • Example Processing PPG data with T-converted LSTM Network uses signal delay block which is added to each recurrent connection of GRU and LSTM neurons.
  • the delay block has an external cycle timer (e.g., a digital timer) which activates the delay block with a constant period of time dt. This activation produces an output of x(t-dt) where x(t) is input signal of delay block.
  • Such activation frequency can, for instance, correspond to network input signal frequency (e.g., output frequency of analog sensors processed by a T-converted network).
  • all delay blocks are activated simultaneously with the same activation signal.
  • Some blocks can be activated simultaneously on one frequency, and other blocks can be activated on another frequency. In some implementations, these frequencies have common multiplier, and signals are synchronized. In some implementations, multiple delay blocks are used over one signal producing additive time shift. Examples of delay blocks are described above in reference to Figure 13B shows two examples of delay blocks, according to some implementations.
  • the network for processing PPG data uses one or more LSTM neurons, according to some implementations. Examples of LSTM neuron implementations are described above in reference to Figure 13A, according to some implementations. [00473] The network also uses Conv1D, a convolution performed over time coordinate. Examples of Conv1D implementations are described above in reference to Figures 15A and 15B, according to some implementations.
  • PPG is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue.
  • a PPG is often obtained by using a pulse oximeter which illuminates the skin and measures changes in light absorption.
  • PPG is often processed to determine heart rate in devices such as fitness trackers.
  • Deriving heart rate (HR) from PPG signal is an essential task in edge devices computing.
  • Some implementations use PPG data from the Capnobase PPG dataset. The data contains raw PPG signal for 42 individuals of 8 min duration each, sampling 300 samples per second, and a reference heartrate data obtained from ECG sensor with sampling around 1 sample per second.
  • the input trained neural network NN-based allows for 1-3% accuracy in obtaining heartrate (HR) from PPG data.
  • HR heartrate
  • This section describes a relatively simple neural network in order to demonstrate how T-conversion and analog processing can deal with this task. This description is provided as an example, according to some implementations.
  • dataset is split into 4,670 training samples and 1,168 test samples.
  • the network included: 1 Conv1D layer with 16 filters and kernel of 20, 2 LSTM layers with 24 neurons each, 2 dense layers (with 24 and 1 neurons each).
  • test accuracy was found to be 2.1%.
  • the input network was T-transformed with following parameters: delay block with periods of 1, 5 and 10 time steps, and the following properties: 17 layers, 15,448 connections, and 329 neurons (OP3 neurons and multiplier blocks, not counting delay blocks).
  • Example Analog Neuromorphic Integrated Circuit for Object Recognition Based on Pulsed Doppler Radar Signal [00480] In some implementations, an Integration Circuit is manufactured, based on a multi-scale LSTM neural network, that can be used to classify the objects, based on pulse Doppler Radar signal.
  • the IC can be used to classify different objects, like humans, cars, cyclists, scooters, based on Doppler radar signal, removes clutter, and provides the noise to Doppler radar signal. In some implementations, the accuracy of classification of object with multi-scale LSTM network exceeded 90%.
  • Example Analog Neuromorphic IC for Human Activity Type Recognition Based on Inertial Sensor Data [00481]
  • a neuromorphic Integrated Circuit is manufactured, and can be used for human activity type recognition based on multi-channel convolutional neural networks, which have input signals from 3-axes accelerometers and possibly magnetometers and/or gyroscopes of fitness tracking devices, smart watches or mobile phones.
  • the multi-channel convolutional neural network can be used to distinguish between different types of human activities, such as walking, running, sitting, climbing stairs, exercising and can be used for activity tracking.
  • the IC can be used for detection of abnormal patterns of human activity, based on accelerometer data, convolutionally merged with heart rate data. Such IC can detect pre-stroke or pre heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy and others, according to some implementations.
  • the IC is based on a channel-wise 1D convolutional network discussed in the article “Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors.”
  • this network accepts 3-axis accelerometer data as input, sampled at up to 96Hz frequency.
  • the network is trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs,” etc.
  • the network included 3 channel- wise Conv networks with Conv layer of 12 filters and kernel of 64, followed by MaxPooling(4) layer each, and 2 common Dense layers of 1024 and N neurons respectively, where N is a number of classes.
  • the activity classification was performed with a low error rate (e.g., 3.12% error).
  • the resulting T-network had following properties: 10 layers, approximately 1,200 neurons (e.g., 1296 neurons), and approximately 20,000 connections (e.g., 20022 connections).
  • Example Transformation of Modular Net Structure for Generating Libraries [00484] A modular structure of converted neural networks is described herein, according to some implementations. Each module of a modular type neural network is obtained after transformation of (a whole or a part of) one or more trained neural network. In some implementations, the one or more trained neural networks is subdivided into parts, and then subsequently transformed into an equivalent analog network. Modular structure is typical for some of the currently used neural networks, and modular division of neural networks corresponds to a trend in neural network development.
  • Each module can have an arbitrary number of inputs or connections of input neurons to output neurons of a connected module, and an arbitrary number of outputs connected to input layers of a subsequent module.
  • a library of preliminary (or a seed list of) transformed modules is developed, including lithographic masks for manufacture of each module.
  • a final chip design is obtained as a combination of (or by connecting) preliminary developed modules.
  • Some implementations perform commutation between the modules.
  • the neurons and connections within the module are translated into chip design using ready-made module design templates. This significantly simplifies the manufacture of the chip, accomplished by just connecting corresponding modules. [00485]
  • Some implementations generate libraries of ready-made T-converted neural networks and/or T-converted modules.
  • a layer of CNN network is a modular building block
  • LSTM chain is another building block
  • Larger neural networks NNs also have modular structure (e.g., LSTM module and CNN module).
  • libraries of neural networks are more than by-products of the example processes, and can be sold independently.
  • a third-party can manufacture a neural network starting with the analog circuits, schematics, or designs in the library (e.g., using CADENCE circuits, files and/or lithography masks).
  • Some implementations generate T-converted neural networks (e.g., networks transformable to CADENCE or similar software) for typical neural networks, and the converted neural networks (or the associated information) are sold to a third-party.
  • a third-party chooses not to disclose structure and/or purpose of the initial neural network, but uses the conversion software (e.g., SDK described above) to converts the initial network into trapezia-like networks and passes the transformed networks to a manufacturer to the fabricate the transformed network, with a matrix of weights obtained using one of the processes described above, according to some implementations.
  • the conversion software e.g., SDK described above
  • corresponding lithographic masks are generated and a customer can train one of the available network architectures for his task, perform lossless transformation (sometimes called T transformation) and provide the weights to a manufacturer for fabricating a chip for the trained neural networks.
  • the modular structure concept is also used in the manufacture of multi-chip systems or the multi-level 3D chips, where each layer of the 3D chip represents one module.
  • the connections of outputs of modules to the inputs of connected modules in case of 3D chips will be made by standard interconnects that provide ohmic contacts of different layers in multi-layer 3D chip systems.
  • the analog outputs of certain modules is connected to analog inputs of connected modules through interlayer interconnects.
  • the modular structure is used to make multi-chip processor systems as well. A distinctive feature of such multi-chip assemblies is the analog signal data lines between different chips.
  • analog commutation schemes typical for compressing several analog signals into one data line and corresponding de-commutation of analog signals at receiver chip, is accomplished using standard schemes of analog signal commutation and de-commutation, developed in analog circuitry.
  • One main advantage of a chip manufactured according to the techniques described above, is that analog signal propagation can be broadened to multi-layer chips or multi-chip assemblies, where all signal interconnects and data lines transfer analog signals, without a need for analog-to-digital or digital-to-analog conversion. In this way, the analog signal transfer and processing can be extended to 3D multi-layer chips or multi-chip assemblies.
  • FIGS 32A-32E show a flowchart of a method 3200 for generating (3202) libraries for hardware realization of neural networks, according to some implementations.
  • the method is performed (3204) at the computing device 200 (e.g., using the library generation module 254) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (3206) a plurality of neural network topologies (e.g., the topologies 224), each neural network topology corresponding to a respective neural network (e.g., a neural network 220).
  • the method also includes transforming (3208) each neural network topology (e.g., using the neural network transformation module 226) to a respective equivalent analog network of analog components.
  • transforming (3230) a respective network topology to a respective equivalent analog network includes: (i) decomposing (3232) the respective network topology to a plurality of subnetwork topologies.
  • decomposing the respective network topology includes identifying (3234) one or more layers (e.g., LSTM layer, fully connected layer) of the respective network topology as the plurality of subnetwork topologies; (ii) transforming (3236) each subnetwork topology to a respective equivalent analog subnetwork of analog components; and (iii) composing (3238) each equivalent analog subnetwork to obtain the respective equivalent analog network.
  • the method also includes generating (3210) a plurality of lithographic masks (e.g., the masks 256) for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components.
  • each circuit is obtained by: (i) generating (3240) schematics for a respective equivalent analog network of analog components; and (ii) generating (3242) a respective circuit layout design based on the schematics (using special software, e.g., CADENCE).
  • the method further includes combining (3244) one or more circuit layout designs prior to generating the plurality of lithographic masks for fabricating the plurality of circuits.
  • the method further includes: (i) obtaining (3212) a new neural network topology and weights of a trained neural network; (ii) selecting (3214) one or more lithographic masks from the plurality of lithographic masks based on comparing the new neural network topology to the plurality of neural network topologies.
  • the new neural network topology includes a plurality of subnetwork topologies
  • selecting the one or more lithographic masks is further based on comparing (3216) each subnetwork topology with each network topology of the plurality of network topologies; (iii) computing (3218) a weight matrix for a new equivalent analog network based on the weights; (iv) generating (3220) a resistance matrix for the weight matrix; and (v) generating (3222) a new lithographic mask for fabricating a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithographic masks.
  • one or more subnetwork topologies of the plurality of subnetwork topologies fails to compare (3224) with any network topology of the plurality of network topologies, and the method further includes: (i) transforming (3226) each subnetwork topology of the one or more subnetwork topologies to a respective equivalent analog subnetwork of analog components; and generating (3228) one or more lithographic masks for fabricating one or more circuits, each circuit of the one or more circuits implementing a respective equivalent analog subnetwork of analog components.
  • Figures 33A-33J show a flowchart of a method 3300 for optimizing (3302) energy efficiency of analog neuromorphic circuits (that model trained neural networks), according to some implementations.
  • the method is performed (3204) at the computing device 200 (e.g., using the energy efficiency optimization module 264) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (3306) an integrated circuit (e.g., the ICs 262) implementing an analog network (e.g., the transformed analog neural network 228) of analog components including a plurality of operational amplifiers and a plurality of resistors.
  • the analog network represents a trained neural network (e.g., the neural networks 220), each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron.
  • the method also includes generating (3308) inferences (e.g., using the inferencing module 266) using the integrated circuit for a plurality of test inputs, including simultaneously transferring signals from one layer to a subsequent layer of the analog network.
  • the analog network has layered structure, with the signals simultaneously coming from previous layer to the next one.
  • the signals propagate through the circuit layer by layer; simulation at device level; time delays every minute.
  • the method also includes, while generating inferences using the integrated circuit, determining (3310) if a level of signal output of the plurality of operational amplifiers is equilibrated (e.g., using the signal monitoring module 268).
  • Operational amplifiers go through a transient period (e.g., a period that lasts less than 1 millisecond from transient to plateau signal) after receiving inputs, after which the level of signal is equilibrated and does not change.
  • the method also includes: (i) determining (3312) an active set of analog neurons of the analog network influencing signal formation for propagation of signals.
  • the active set of neurons need not be part of a layer/layers. In other words, the determination step works regardless of whether the analog network includes layers of neurons; and (ii) turning off power (3314) (e.g., using the power optimization module 270) for one or more analog neurons of the analog network, distinct from the active set of analog neurons, for a predetermined period of time.
  • some implementations switch off power (e.g., using the power optimization module 270) of operational amplifiers which are in layers behind an active layer (to where signal propagated at the moment), and which do not influence the signal formation on the active layer. This can be calculated based on RC delays of signal propagation through the IC. So all the layers behind the operational one (or the active layer) are switched off to save power. So the propagation of signals through the chip is like surfing - the wave of signal formation propagate through chip, and all layers which are not influencing signal formation are switched off.
  • signal propagates layer to layer, and the method further includes decreasing power consumption before a layer corresponding to the active set of neurons because there is no need for amplification before the layer.
  • determining the active set of analog neurons is based on calculating (3316) delays of signal propagation through the analog network.
  • determining the active set of analog neurons is based on detecting (3318) the propagation of signals through the analog network.
  • the trained neural network is a feed-forward neural network, and the active set of analog neurons belong to an active layer of the analog network, and turning off power includes turning off power (3320) for one or more layers prior to the active layer of the analog network.
  • the predetermined period of time is calculated (3322) based on simulating propagation of signals through the analog network, accounting for signal delays (using special software, e.g., CADENCE).
  • the trained neural network is (3324) a recurrent neural network (RNN), and the analog network further includes one or more analog components other than the plurality of operational amplifiers, and the plurality of resistors.
  • the method further includes, in accordance with a determination that the level of signal output is equilibrated, turning off power (3326) (e.g., using the power optimization module 270), for the one or more analog components, for the predetermined period of time.
  • the method further includes turning on power (3328) ) (e.g., using the power optimization module 270) for the one or more analog neurons of the analog network after the predetermined period of time.
  • determining if the level of signal output of the plurality of operational amplifiers is equilibrated is based on detecting (3330) if one or more operational amplifiers of the analog network is outputting more than a predetermined threshold signal level (e.g., power, current, or voltage).
  • a predetermined threshold signal level e.g., power, current, or voltage.
  • the method further includes repeating (3332) ) (e.g., by the power optimization module 270) the turning off for the predetermined period of time and turning on the active set of analog neurons for the predetermined period of time, while generating the inferences.
  • the method further includes, in accordance with a determination that the level of signal output is equilibrated, for each inference cycle (3334): (i) during a first time interval, determining (3336) a first layer of analog neurons of the analog network influencing signal formation for propagation of signals; and (ii) turning off power (3338) ) (e.g., using the power optimization module 270) for a first one or more analog neurons of the analog network, prior to the first layer, for the predetermined period of time; and during a second time interval subsequent to the first time interval, turning off power (3340) ) (e.g., using the power optimization module 270) for a second one or more analog neurons including the first layer of analog neurons and the first one or more analog neurons of the analog network, for the predetermined period.
  • a determination that the level of signal output is equilibrated for each inference cycle (3334): (i) during a first time interval, determining (3336) a first layer of analog neurons of the analog network influencing signal formation for propag
  • the one or more analog neurons consist (3342) of analog neurons of a first one or more layers of the analog network, and the active set of analog neurons consist of analog neurons of a second layer of the analog network, and the second layer of the analog network is distinct from layers of the first one or more layers.
  • Some implementations include means for delaying and/or controlling signal propagation from layer to layer of the resulting hardware-implemented neural network.
  • Example Transformation of MobileNet v.1 An example transformation of MobileNet v.1 into an equivalent analog network is described herein, according to some implementations.
  • MobileNet v1 architecture is depicted in the Table shown in Figure 34.
  • the first column 3402 corresponds to type of layer and stride
  • the second column 3404 corresponds to filter shape for the corresponding layer
  • the third column 3406 corresponds to input size for the corresponding layer.
  • the network consists of 27 convolutional layers, 1 dense layer, and has around 600 million multiply-accumulate operations for a 224x224x3 input image.
  • Output values are the result of softmax activation function which means the values are distributed in the range [0, 1] and the sum is 1.
  • the network is pre-trained for CIFAR-10 task (50,000 32x32x3 images divided into 10 non-intersecting classes). Batch normalization layers operate in ‘test’ mode to produce simple linear signal transformation, so the layers are interpreted as weight multiplier + some additional bias. Convolutional, AveragePooling and Dense layers are transformed using the techniques described above, according to some implementations.
  • Softmax activation function is not implemented in transformed network but applied to output of the transformed network (or the equivalent analog network) separately.
  • the resulting transformed network included 30 layers including an input layer, approximately 104,000 analog neurons, and approximately 11 million connections.
  • the average output absolute error (calculated over 100 random samples) of transformed network versus MobileNet v.1 was 4.9e-8.
  • the output signal on each layer of the transformed network is also limited by the value 6.
  • the weights are brought into accordance with a resistor nominal set. Under each nominal set, different weight values are possible. Some implementations use resistor nominal sets e24, e48 and e96, within the range of [0.1 - 1] Mega Ohm.
  • R- and R+ values are chosen separately for each layer from the set [0.05, 0.1, 0.2, 0.5, 1] Mega Ohm.
  • a value which delivers most weight accuracy is chosen.
  • all the weights (including bias) in the transformed network are ‘quantized’, i.e., set to the closest value which can be achieved with used resistors.
  • this reduced transformed network accuracy versus original MobileNet according to the Table shown below.
  • the Table shows mean square error of transformed network, when using different resistor sets, according to some implementations.
  • Some implementations provide a method for fabricating a neuromorphic Integrated Circuit for voice clarification, using techniques described above.
  • Various types of trained neural networks can be used for this purpose. For example, a neural network can be trained to identify only one voice, suppressing and removing everything else. In particular, the neural network can identify the voice that is the closest to the microphone. As another example, a neural network can be trained to identify several voices, suppressing and removing everything else. Voices can be identified and preserved regardless of their distance from the microphone(s). Alternatively, voices can be prioritized by their distances from the microphone(s) and given different weights in the output signal, based on their respective distances form the microphone.
  • voices can be identified and preserved regardless of their relative strength (e.g., volume).
  • voices can be prioritized by their relative strength and be given different weights in the output signal, based on their respective relative strength.
  • a neural network can process the signal that is originating from the microphone(s). Such a signal may include analog and/or digital signals.
  • a neural network can process an analog and/or a digital signal that is transmitted over a transmission media and received by the neural network. Such a signal can be transmitted across wireless or digital/internet networks for the purposes of phone communication. Such a signal can also be input after pre- and post-processing of the original voice(s), either before the signal is ready to be transmitted, or after the signal has been transmitted and delivered to the recipient.
  • a neural network can process a signal that is a mix of several voice signals, with associated noises. In particular, such a mix can be delivered to the recipient from several different sources. Such a signal can be pre- and post-processed by different methods for different components.
  • a neural network can process the signal that is a mix of several external voice signals, with associated noises, combined with the own voice(s) on the recipient side. In particular, such a mix can be delivered to the recipient from several different sources, including the recipient’s own voice overlapped with recipient’s own noises.
  • Such a signal can be pre- and/or post-processed by different methods for different components. The clarification of voice(s) can be performed for the combined signal.
  • a neural network can process a signal that includes voice(s) from the recipient side.
  • a signal can be processed before it is transmitted to the other party.
  • Such a signal can be processed by the neural network before it is pre- and/or post-processed by different methods prior to transmission.
  • Example Methods for Extracting Voice from Inbound or Outbound Analog noisy Signal [00514] Described herein are example techniques for the extraction of voice from a noisy signal, both inbound and outbound, where noise can be either stationary or non- stationary, using a neuromorphic analog Integrated Circuit. Such a circuit implements a noise suppression neural network at the hardware level.
  • the circuit design of the analog neuromorphic Integrated Circuit is realized by converting (using techniques described above) a noise suppression (or voice extraction) neural network.
  • a noise suppression (or voice extraction) neural network As described in the Background and Summary sections, the task of extracting the voice from noisy signal is of great importance for communication in smartphones, smartwatches, notebooks, or other voice transmitting devices.
  • noise cancellation or active noise suppression using dual microphone scheme, where the signal from one microphone is used to cancel noise at a main microphone. But these solutions do not cancel all noises, especially non-stationary ones, since not all noise is canceled in such combination of two microphones.
  • filters which can filter out stationary noise from inbound or outbound analog signal.
  • the Integrated Circuit is realized as a hardware solution and is represented by a set of operational amplifiers and resistors, connected in such a way that the resulting neuromorphic hardware chip operates similarly to the initial neural network (e.g., the neural network realized in software), with the absolute error not exceeding a maximum threshold percentage (e.g., 1% absolute) from the error corresponding to the software neural network.
  • the schematics of the Integration Circuit are obtained using techniques described above, thus ensuring full equivalency of analog neuromorphic hardware realization of the neural network and its initial software neural network model.
  • the analog Integrated Circuit may be used for voice extraction from noisy analog inbound or outbound signals, with low latency and low power consumption.
  • the hardware realization of a voice extraction neural network can be used to process both inbound and outbound noisy signals.
  • the Integrated Circuit has direct analog input and is placed adjacent to a microphone or a speaker of a smartphone, smartwatch, earbuds, notebook computer, or similar device.
  • the Integrated Circuit provides telecommunication voice transfer, extracting voice from noisy analog signals.
  • Such a solution suppresses both stationary and non-stationary noise from inbound or outbound analog signals (e.g., signals from a microphone or signals directed to a speaker or earbuds) and is characterized by excellent noise suppression, unlike conventional methods.
  • the resulting hardware realization of a voice extraction algorithm is characterized by low power operation, small latency, and small die area, which makes analog hardware realization an advantageous solution for noise reduction in smartphones, earbuds, notebook computers, tablets, or other voice transmitting devices, in comparison with software neural network voice extraction algorithms.
  • the small die area makes it possible to include the Integrated Circuit application in true wireless (TWS) earbuds or other miniature devices.
  • TWS true wireless
  • Such analog Integrated Circuits may also be used for two-way voice extraction (noise reduction) in Notebook PCs or Smartphones, where a neuromorphic analog integration circuit is installed both at the analog output of the microphone and at the analog input of the speaker or earbuds.
  • Example Neuromorphic Analog Integrated Circuit for Voice Clarity [00519] Some implementations obtain a convolutional neural network with 1D convolutions (e.g., as described in “Single Channel Speech Enhancement Using A Convolutional Neural Network,” by T. Kounovsky and J. Malek, 2017), an example of which is shown in Figure 35.
  • the architecture 3500 shown in Figure 35 performs Fourier transformation of incoming analog signal, to obtain input features 3502 that form a network input 3504. Subsequently, the architecture uses convolution 3506, maxpooling 3508, convolution 3510 and fully connected layers (layers 3512 and 3514), to obtain output 3516. An inverse Fourier transformation is applied on the output 3516 to obtain an analog output signal.
  • Some implementations convert this example network 3500 into a network of analog components using techniques described above and herein. Some implementations apply the techniques described above for fabricating a neuromorphic analog integrated circuit based on the network of analog components. Simulations have shown the resulting integrated circuit occupied a 30 square millimeter die area, consumed approximately 150 micro-Watts of power, and had a signal latency of 3 milliseconds. The resulting integrated circuit can be used for inbound and outbound voice extraction for smartphones, earbuds, notebook computers, smartwatches, or other telecommunication devices. In experiments, when the quality is measured according to PESQ criteria, an improvement of voice signal up to 30% was accomplished. The small power consumption allows the use of such integrated circuits in battery-powered devices.
  • the small die area allows installing the device into TWS earbuds or other miniature devices.
  • the network architecture shown in Figure 35 includes convolution, max pooling, and fully-connected layers. Transformation of convolutional layers is described above, according to some implementations. The following sections describe example transformation techniques for various components of the network shown in Figure 35, according to some implementations.
  • Example Transformation of Fully Connected (Dense) Layers [00521]
  • Figure 36 shows an example transformation 3600 of a fully-connected or dense layer, according to some implementations.
  • W is a weight matrix 3610, and each input is connected to every output.
  • Each weight W[i][j] corresponds to an edge or connection between the Input i and the Output j.
  • Bias b is a bias vector and f(x) is an element- wise activation function, typically ReLU.
  • FIG. 37 shows example transformations 3700 of a Conv1D layer of a Convolutional Neural Network, according to some implementations.
  • the Conv1D layer of a neural network is a convolution applied over a 1-dimensional (e.g., the X dimension 3708) sequence of data (shown as input 3702, which may include multiple input channels 3712) to obtain output 3704 (with output channels 3714).
  • Each element of an output tensor is a product of a particular subset of the input data 3702 and a slice of a weight tensor called the kernel 3706 (sometimes called a filter kernel or a filter).
  • the kernel 3706 sometimes called a filter kernel or a filter.
  • some implementations evaluate the output tensor shape and construct a layer of SNMs (e.g., the layer 3710) with the same shape (the number of SNMs equals the product of the tensor dimensions).
  • Each of these SNMs has incoming connections weighted according to a relevant slice of the weight tensor and is connected to a relevant subset of input SNMs of the previous layer.
  • Output-to-Input SNMs relations may depend on Conv1D layer parameters, such as kernel size, stride, padding, and filters.
  • the shape of the output layer SNMs is the same as output tensor shape in mathematical representation, where each SNM has a number of incoming weighted connections from the previous layer.
  • Example Transformation of MaxPooling Layers [00523]
  • Figure 38 shows example transformations 3800 of a MaxPooling layer of a Convolutional Neural Network, according to some implementations.
  • the subset of input values is defined by pool size and strides parameters (e.g., as specified by a machine learning framework where the original Convolutional Neural Network is defined).
  • Some implementations define dependency relations between each output value and a related subset of input values, and for each output value, build a maximum subtree (sometimes called a Max subtree), thereby converting the MaxPooling layer into a multilayer SNM structure according to following algorithm: a) Define a schematic with 2 layers and 2 SNMs (e.g., SNM 3808 and SNM 3810) performing a max operation (e.g., the Max2 operation 3802) over 2 Input elements (e.g., Input 13804 and Input 23806).
  • a max operation e.g., the Max2 operation 3802
  • 2 Input elements e.g., Input 13804 and Input 23806
  • b) Define a schematic with 3 layers and 3 SNMs (e.g., SNM 3820, SNM 3822, and SNM 3824) performing a max operation (e.g., the Max3 operation 3812) over 3 Input elements (e.g., Input 13814, Input 23816, and Input 33818).
  • c) Define a schematic with 3 layers and 4 SNMs (e.g., SNM 3836, SNM 3838, SNM 3840, and SNM 3842) performing a max operation (e.g., the Max4 operation 3826) over 4 Input elements (e.g., Input 13828, Input 23830, Input 33832, and Input 43834).
  • This tree is built in a manner that minimizes total tree layers and prioritizes the use of the Max4 schematic, according to some implementations. For instance, max(1,2,3,4,5,6,7,8,9) is transformed into max(max(1,2,3,4), max(5,6,7,8), 9) producing a structure of 6 layers with 11 SNMs.
  • An activation function other than ReLU can be applied over the output neuron.
  • Some implementations include two neural networks in a pipeline. Details on data, model, training, validation main results, for the two neural networks, are described herein.
  • a dataset contains labeled experiments with different types of activity (e.g., more than 50 types of activity), which correspond to one of several basic activity classes.
  • the basic activity classes include ‘SLS’ that indicates a resting state, ‘Walk’ that indicates walking, including on a treadmill, ‘Run’ that indicates running, including on a treadmill, and ‘Intensive’ that indicates various exercises from crossfit workouts, characterized by high load.
  • the two neural networks accomplish two tasks: creation of a descriptor (model converts a time series into a vector, embedding) and classification of base classes of activities (model matches the time series with one of four base classes).
  • the task of creating a descriptor includes training and testing a model (sometimes called a descriptor) that receives data with fixed-length (window) from one or more sensors (e.g., an accelerometer) as input and outputs a vector. Embeddings of windows with the same activity class are likely to be close and can be easily separated and a simple classifier can be built on top of this model.
  • the classification task includes training and testing the classifier for the base classes of activities. [00525]
  • a dataset may correspond to a set of experiments.
  • the dataset includes input for one or more users. For example, one person could participate in several experiments. In some implementations, each person may participate in only one experiment. Each experiment contains information, such as the values of the signal from an accelerometer and activity marks.
  • a sequence of windows (data sequences of a fixed length) is obtained. In some implementations, the sequence of windows contains data from 3 channels from the accelerometer, as well as an activity label for each window (or a mode of activity for each window).
  • Training a descriptor model may include two stages: training a neural network model for the descriptor and examining the quality of resulting embeddings. For the first stage, the dataset is divided into two parts: training and validation.
  • Figure 39 shows an example 3900 of splitting a dataset 3902 to a training dataset 2904 and a validation dataset 3906, based on experiments experiment_1, experiment_2, experiment_3, ..., experiment_N, according to some implementations. In this way, the model is tested on a previously unseen experiment, so that it is possible to objectively assess the quality of the model.
  • special types of activities are selected. These are activities, windows for which are excluded from the training dataset and included in the validation process.
  • FIG. 40 shows two graph plots 4000 and 4002 of activity labels, according to some implementations.
  • the graph plot 4000 corresponds to a mark up of segments for activity labels label_0, label_1, label_2, and label_3.
  • the x-axis in the two graph plots correspond to time periods, and the y-axis corresponds to activity labels.
  • the graph plot 4000 shows a mark up of segments. Segments marked 4004 correspond to activity label label_1 and segments marked 4006 correspond to activity label_2, in an experiment. There are four types of activities (or activity labels) in this experiment.
  • the activity types are selected in turn and periods with the selected activity are identified.
  • the label_2 activity type there are two such periods (shown in the graph plot 4002).
  • the two periods for label_2 activity are split into training 4008 and testing 4010 datasets for the selected activity (i.e., label_2).
  • Figures 41A and 41B show an example graph plot 4100 for splitting experiments to training or testing test data, according to some implementations.
  • First half of the periods is included in a training sample 4102, and second half of the periods is included in a test sample 4104. Since the segment size is 125, and the step is 25, the segments have intersections.
  • the techniques described herein allow minimizing the number of intersections between segments from train and test samples. Intersections arise regardless, however.
  • the experiment has an "unknown" label (activity near bottom of they-axis). If an intersection is found between training and testing samples, then the intersecting segments are deleted 4106. Segments belonging to the unknown activity type are prioritized for deletion. As in Figure 40, the y-axis in Figures 41A and 41B show activity labels.
  • Example Model the model consists of three parts: an encoder, a decoder, and a classifier head.
  • the process of training the model is divided into two stages: training the encoder and decoder (sometimes called an autoencoder) and training the classifier.
  • the autoencoder is trained, then the autoencoder is frozen and the classifier is trained, which receives embedding from the encoder as input, according to some implementations.
  • Figures 42A, 42B, and 42C show schematic diagrams of an example encoder 4200, an example decoder 4222, and an example classifier 4244, respectively, according to some implementations.
  • Example Descriptor model [00529] As a first step, the autoencoder is trained.
  • the autoencoder includes an encoder (e.g., the encoder 4200) and a decoder (e.g., the decoder 4222).
  • the autoencoder includes one-dimensional convolutions (Conv1d in the encoder and ConvTranspose1d in the decoder), and after each convolutional layer, BatchNorm is used.
  • An example kernel size in Conv1d and ConvTranspose1d layers is 5.
  • the first 4 layers of convolutional layers have stride 1, the remaining layers have stride 2. After each layer, ReLu is used as the activation function.
  • the output of the encoder is a vector of length 16.
  • each embedding is a string of 16 bytes. Each of these 16 bytes encodes a feature of a user’s movement. In other words, the 16 bytes together represent a specific movement, and represent a digital fingerprint of the movement.
  • an example embedding includes the following values: -2.3534, -28.4428, 10.2809, - 10.5756, -1.0527, -7.4458, 1.3814, -1.9068, 10.5118, 1.2902, 26.5022, 1.9261, -2.9055, 3.9552, -0.3831, and 13.4464.
  • Each value represents a feature for a specific movement.
  • the encoder 4200 includes 3 input channels 4202 (of length 125 each, acc_x, acc_y, and acc_z for the three input channels), a one-dimensional convolution 4204 for 16 channels with stride 1, a one-dimensional convolution 4206 for 16 channels with stride 1, a one-dimensional convolution 4208 for 16 channels with stride 1, a one-dimensional convolution 4212 for 16 channels with stride 1, a one-dimensional convolution 4212 for 32 channels with stride 2, a one-dimensional convolution 4214 for 64 channels with stride 2, a one-dimensional convolution 4216 for 128 channels with stride 2, a flatten layer 4218, and a layer 4220 that generates linear output with length 16 (embedding).
  • the decoder 4222 includes input (embedding) 4224 for 1 channel of length 16, a linear layer 4226, a layer 4228 to reshape to 128 channels, a convolution or transpose layer 4230 for 64 channels with stride 2, a convolution or transpose layer 4232 for 32 channels with stride 2, a convolution or transpose layer 4234 for 16 channels with stride 2, a convolution or transpose layer 4236 for 16 channels with stride 1, a convolution or transpose layer 4238 for 16 channels with stride 1, a convolution or transpose layer 4240 for 16 channels with stride 1, and a convolution or transpose layer 4230 for 3 channels of length 125 (which is the reconstructed input).
  • input embedding
  • Example Base Class Classifier model [00532] Next, the classifier of the base classes is trained.
  • the encoder trained in the first step forms an embedding, which is input to a multilayer perceptron, according to some implementations. Some implementations use linear layers with ReLU at the output.
  • the model outputs a vector of length 4 (e.g., a classification into 4 basic activity classes). If the window was marked as unknown activity type, then it received the label 1 and did not participate in training the model and calculating metrics. This is due to the fact that it cannot be said unambiguously which base class of activity took place on a given window.
  • the classifier 4244 includes input (embedding) 4246 for 1 channel of length 16, a linear layer 4248 of length 256, a linear layer 4250 of length 128, a linear layer 4252 of length 64, a linear layer 4254 of length 32, and a linear layer 4256 of length 4 (which is the predicted class).
  • Example Training [00534] In some experiments, the training process of the model was carried out on a V100 video card, the consumption of RAM during the training process reached 40GB, the average time of one epoch was 2 minutes 15 seconds. The training was split in two stages: training the autoencoder (for descriptor task) and training the classifier of the basic classes.
  • Training process consisted of 40 epochs for training for the descriptor and 40 epochs for training the classifier, learning rate was 0.001, and for descriptor learning rate scheduler was used (every 10 epochs learning rate was decreased by 2 times).
  • the main metric used to assess the quality of models is F2 score. This metric can be calculated using Equation (1) shown below.
  • P indicates Precision
  • R indicates Recall.
  • ⁇ 2_ ⁇ ⁇ ⁇ ⁇ ⁇ ( ⁇ , ⁇ ) 5 ⁇ ⁇ ⁇ ⁇ /(4 ⁇ ⁇ + ⁇ ) (1)
  • MSE loss an example of which is shown in Equation (2) below
  • Equation (2) T is true vector
  • P is predicted vector
  • N vector length
  • P is the predicted vector of length
  • P is the predicted vector of length C
  • C is a number of classes
  • class is an index of truth class.
  • ⁇ ⁇ ⁇ ⁇ ⁇ ( ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ) ⁇ log Example Results Descriptor model
  • the embeddings were again divided into training or testing using the algorithm described above for training and testing a simple classifier of activity types.
  • KNN K-Nearest Neighbors
  • This classifier was trained separately for each special type of activity - a binary classification was performed.
  • the predictions from the binary classifier are smoothed using a median filter with a window of 15 predictions (equivalent to 15 seconds).
  • the values of the metrics on the training and testing samples of this classifier are presented below in Tables 1 and 2, respectively.
  • Table 1 Training scores for KNN model, trained on embeddings
  • Table 2 Test scores for KNN model, trained on embeddings
  • Figures 43A-43F, 44A-44D, 46A-46F, 47A-47F, and 49A-49D show graphs of results, according to some implementations. In these graphs, the Y-axis corresponds to the type of activity, and the X-axis corresponds to time. Thin lines 4308 indicate the true value of the type of activity, thick or solid lines 4310 indicate prediction of the KNN model, pattern 4312 (with grains) under the graph indicate which sections were assigned to train, and pattern 4314 with stripes under the graph indicate test samples, respectively, for the classifier.
  • Figures 43A-43F show graph plots of time series of predictions of the KNN classifier for a set of experiments, according to some implementations.
  • Figures 43A and 43B show a graph plot 4300 that corresponds to data for crossfit_airbike. Note predictions 4302-2, 4302-4, 4302-6, and 4302-8 of crossfit_airbike.
  • Figures 43C and 43D show a graph plot 4304 that corresponds to data for crossfit_row. Note predictions 4306-2, 4306-4, 4306-6, 4306-8, and 4306-10 of crossfit_row.
  • Figures 43E and 43F show a graph plot 4308 that corresponds to data for crossfit_ski. Note predictions 4310-2, 4310-4, 4310-6, and 4310-8 of crossfit_ski.
  • Figures 44A-44D show additional graph plots of time series of predictions of the KNN classifier, according to some implementations.
  • Figures 44A and 44B show a graph plot 4400 that corresponds to data for crossfit_row. Note predictions 4402-2, 4402-4, 4402-6, and 4402-8 of crossfit_row.
  • Figures 44C and 44D show a graph plot 4404 that corresponds to data for crossfit_situp. Note predictions 4406-2, 4406-4, 4406-6, 4406-8, and 4406-10 of crossfit_situp.
  • FIGS. 45A and 45B show a table 4500 that presents predictions for the base class activity classifier model on validation experiments described earlier, according to some implementations. The table shows count of predictions of each activity types and base activity classes. Scale 4502 is shown on the right.
  • Table 3 shown below lists average metrics for the classification task for base activity classes. These metrics were first calculated for each experiment, after which they were averaged between experiments. For this reason, the value of these metrics differs from the metrics shown in Figures 45A and 45B, where they were calculated to add all the experiments together. [00544] Table 3: Mean base class metrics across valid experiments [00545] Figures 46A-46F show graph plots of time series of predictions of the KNN classifier for a set of experiments, according to some implementations.
  • Figures 46A and 46B show a graph plot 4600 that corresponds to data for crossfit_airbike. Note predictions 4602-2 and 4602-4 of crossfit_airbike. For training, accuracy was 0.9942, F1 score was 0.9286, F2 score was 0.9028, precision was 0.975, and recall was 0.8864; and for test, accuracy was 0.9815, F1 score was 0.8598, F2 score was 0.9388, precision was 0.7541, and recall was 1.0.
  • Figures 46C and 46D show a graph plot 4604 that corresponds to data for crossfit_row. Note predictions 4606-2, and 4606-4 of crossfit_row.
  • FIGS. 46E and 46F show a graph plot 4608 that corresponds to data for crossfit_ski. Note predictions 4610 of crossfit_ski.
  • accuracy was 0.9832, F1 score was 0.557, F2 score was 0.4622, precision was 0.8462, and recall was 0.4151; and for test, accuracy was 0.9766, F1 score was 0.7246, F2 score was 0.7062, precision was 0.7576, and recall was 0.6944.
  • Figures 47A-47F show graph plots of time series of predictions of the KNN classifier for a set of experiments, according to some implementations.
  • Figures 47A and 47B show a graph plot 4700 that corresponds to data for crossfit_row. Note prediction 4702-2 of crossfit_row. For training, accuracy was 0.9988, F1 score was 0.9829, F2 score was 0.9729, precision was 1.0, and recall was 0.9663; and for test, accuracy was 1.0, F1 score was 1.0, F2 score was 1.0, precision was 1.0, and recall was 1.0.
  • Figures 47C and 47D show a graph plot 4704 that corresponds to data for crossfit_ski.
  • Figures 46A-46F show a few examples of false recognition of incorrect activity.
  • the system predicts crossfit_airbike 4602-4 when the person was at rest (the type of activity is standing). This may be due to arm movements during rest between exercises (arm warm-up).
  • crossfit_warmup e.g., in Figure 46C, the system predicts crossfit_row 4606-2 instead of crossfit_warmup, and in Figure 46D, the system predicts crossfit_row 4606-2 instead of standing).
  • Figure 47F the system fails to predict crossfit-situp 4712 in some instances.
  • some implementations use larger training datasets (e.g., use a size large enough to correct such errors noticed during testing). For example, there are public datasets that contain dozens of hours of data. For commercial use, practical applications may use hundreds to thousands of hours of data, and it is possible to collect such data in a practical way.
  • Figure 48 shows a visualization 4800 for t-distributed stochastic neighbor embedding (t-SNE) by exercise sets (sets 4802 and 4804) for embedding different sets of the crossfit_situp exercise in experiment shown in Figures 47E and 47F, according to some implementations.
  • a set refers to a set of activities for an exercise, performed for a duration, before a user rests.
  • the last two sets 4802 are far from the previous ones 4804. Nevertheless, embeddings are well grouped into clusters, so with an increase of size of the dataset for training KNN, the quality of classification this type of activity is likely to improve.
  • Figures 49A-49D show additional graph plots of time series of predictions of the KNN classifier, according to some implementations.
  • Figures 49A and 49B show a graph plot 4900 that corresponds to data for crossfit_ski, without smoothing. Note predictions 4902 of crossfit_ski.
  • Figures 49C and 49D show a graph plot 4904 that corresponds to data for crossfit_ski, with smoothing. Note predictions 4906 of crossfit_ski.
  • Figures 49A-49D show how the presence of smoothing affects the quality of the classification. These graphs show the time series for the crossfit_ski activity binary classifier. When using the median filter (or smoothing, as shown in graphs 49C and 49D), the first set of the exercise 4908 is lost.
  • FIG. 50 shows an example human activity recognition device 5000, according to some implementations.
  • the device includes an integrated circuit 5002 for human activity recognition.
  • the integrated circuit includes an analog network of analog components configured to implement a trained neural network model.
  • the neural network model is trained to generate a plurality of descriptors 5006 for a plurality of predefined human activities based on a plurality of features extracted from a plurality of electrical signals (e.g., signals 5012 and/or signals 5008) from one or more sensors (e.g., external sensors 5010 and/or internal sensors 5008).
  • the device also includes one or more digital components 5004 (e.g., CPUs, DSPs, and associated I/O components) configured to classify human activity as one of the plurality of predefined human activities according to the plurality of descriptors 5006 generated by the integrated circuit 5002.
  • the human activity recognition device 5000 further includes the one or more sensors (e.g., the internal sensors 5008) configured to collect the plurality of electrical signals during the human activity.
  • the trained neural network model (implemented by the integrated circuit 5002) is an autoencoder that includes an encoder (e.g., the encoder 4200) and a decoder (e.g., the decoder 4222).
  • the one or more digital components 5004 implement a trained machine learning classifier that is a KNN (K-Nearest Neighbors) classifier (e.g., the classifier 4244) which can be retrained.
  • KNN K-Nearest Neighbors
  • the trained machine learning classifier is trained separately for each of the plurality of predefined human activities using binary classification.
  • the one or more digital components 5004 are further configured to smooth the output of the trained machine learning classifier to obtain a basic class of activity.
  • the one or more sensors include one or more of IMUs, cameras, microphones, and biofeedback devices.
  • the integrated circuit 5002 is fabricated by steps comprising: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model.
  • Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating the integrated circuit, according to the schematic model, using a lithographic process. Examples of related neural network topologies, trained neural networks, analog networks, conversion methods, and fabrication steps, are shown and described above in reference to Figures 1-38, according to some implementations.
  • generating the schematic model includes generating a resistance matrix for the weight matrix.
  • Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • FIG 51 shows a flowchart of an example method 5100 for recognizing (5102) human activities, according to some implementations.
  • the method is performed (5104) by a human activity recognition device (e.g., the device 5000 described above in reference to Figure 50).
  • the method includes using (5106) one or more sensors to track activity of a user, including obtaining a plurality of electrical signals from the one or more sensors. Window length, step size, and/or sampling frequency, for obtaining the electrical signals, may be adjusted, depending on the human activities, according to some implementations.
  • the method also includes forming (5108) a feature vector by extracting a plurality of features from the plurality of electrical signals.
  • the features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities.
  • raw electrical signals from the sensors may be input to a deserializer, and may be sampled (e.g., at 20-25 Hz), to extract features from the electrical signals.
  • the processed signals may be scaled and/or shifted.
  • An accelerometer generates digital signals, which are converted to analog signals before applying transformations. Some implementations select only a single axis or a pair of axes of acceleration data, when extracting features for forming the feature vectors. Some implementations normalize acceleration data, and/or create segments from the acceleration data (sample size may define the number of segments per feature vector). In some implementations, the feature vectors are formed from analog raw signals using a deserializer and sampling using a hold array, where the number of samples and holds equals the number of inputs of an analog neural network.
  • the method also includes applying (5110) an analog neurocomputing hardware device to the feature vector (in other words, inputting the feature vector to the analog neurocomputing hardware) to generate an embedding vector that specifies a descriptor.
  • the analog neurocomputing hardware device implements the trained neural network model.
  • the trained neural network model is an autoencoder, which includes an encoder and a decoder.
  • An autoencoder is also suitable for keyword spotting, and the exact keywords can be defined after the network is trained.
  • Pictures need 2D convolutions while sound can be done with 1D convolutions, which resemble the example autoencoder architecture described above in reference to Figures 42A and 42B, according to some implementations.
  • the window size may be adjusted depending on the application.
  • the method also includes applying (5112) a trained machine learning classifier to the embedding vector to classify the activity of the user as one of the predefined human activities.
  • the trained machine learning classifier is a KNN (K-Nearest Neighbors) classifier.
  • the number of neighbors for the KNN classifier equals five.
  • the trained machine learning classifier is trained separately for each of the predefined human activities using binary classification. Some implementations also smooth the output of the trained machine learning classifier to obtain a basic class of activity.
  • Some implementations perform post-processing of the output of the human activity recognition device in an I/O device 5016 which may be a consumer device, such as a smart watch.
  • Smoothing the activity labels may include using average(X) predictions within some time range instead of taking just one prediction during each time step, or using some other more complicated formula. Some implementations perform prediction every second. A user may not need to know the activity every second. For example, the user may want to know that information during a specific time of a day or in certain time periods (e.g., from 1:00 till 1:45, when the user is typically running). Smoothing may be used for that purpose, according to some implementations.
  • the analog neurocomputing hardware device is fabricated by steps including: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model (each element of the weight matrix represents one or more connections between analog components of the equivalent analog network); generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating an integrated circuit, according to the schematic model, using a lithographic process. Examples of these steps are described above in reference to Figures 1-38, according to various implementations.
  • generating the schematic model includes generating a resistance matrix for the weight matrix, where each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the trained machine learning classifier is implemented using one or more digital components and the trained machine learning classifier can be retrained for new users.
  • Figure 52 shows a flowchart of another example method 5200 for recognizing (5202) human activities, according to some implementations. The method is performed (5204) by a human activity recognition device 5000 described above in reference to Figure 50. The method includes obtaining (5206) a sequence of electrical signals from one or more sensors that track activity of a user.
  • the method also includes forming (5208) a plurality of feature vectors by extracting features from the sequence of electrical signals.
  • the features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities.
  • the method also includes applying (5210) the analog neurocomputing hardware device to the plurality of feature vectors to generate a plurality of embedding vectors, each of which specifies a corresponding descriptor.
  • the method also includes using (5212) the plurality of embedding vectors to classify the activity of the user as one of the predefined human activities.
  • the method further includes: receiving (5214), from the user, a set of descriptors that describes specific physical activities; and using the set of descriptors and the plurality of embedding vectors to classify the activity of the user as one of the specific physical activities.
  • the method further includes generating statistics of personal daily routines of the user based on classifying the activity of the user as one of the specific physical activities.
  • the method further includes: storing (5216), for the user, the plurality of embedding vectors as describing a specific activity; and using the plurality of embedding vectors for classifying subsequent activities of the user as the specific activity.
  • the method further includes: receiving (5218), from a trainer (not the user), a set of descriptors that describes a specific activity; and providing feedback to the user if the activity matches the specific activity based on the plurality of embedding vectors and the set of descriptors.
  • Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines [00566] Sensors that monitor rotation or reciprocating parts of machines typically generate signals in the range of 0 to 20 kHz. A wideband connection to a control unit is necessary. This makes the sensors consume a lot of power, making it almost impossible to use batteries. An apparatus based on analog neuromorphic hardware is suitable for this application.
  • an architecture contains artificial neurons (the nodes performing computations) and axons (the connections with weights between the nodes) implemented using circuitry elements.
  • the neurons may be implemented as operational amplifiers, and axons may be implemented using thin-film resistors.
  • the devices according to the techniques described here simplify chip layout.
  • the techniques work well for Convolutional Neural Networks (CNN), where connections are very sparse, as well as Recurrent Neural Networks (RNNs), transformers, and autoencoders.
  • CNN Convolutional Neural Networks
  • RNNs Recurrent Neural Networks
  • Techniques for converting a trained and optimized neural network model into a chip structure are described above.
  • the chip architecture offers area utilization close to 100% and may use 8 bits per weight of a trained neural network. Such techniques enable faster time to market, lower technical risks, and better performance.
  • processors according to the techniques described herein may use hybrid cores (using analog and digital hardware) similar to human brain data processing.
  • Some implementations combine a fixed weights method, which includes complete separation between inference and training, with a fixed chip structure, similar to the human visual nerve and retina, and a flexible function, which differs depending on the application.
  • the flexible function is responsible for further classification of the received embeddings.
  • machine learning it is well-known that, after several hundred training cycles (also known as epochs), a deep convolutional neural network maintains fixed weights and structure for the first 80-90% of the layers, and in the following cycles only the few last layers responsible for classification continue to change weights. This property is also used in transfer learning techniques. This property is used for implementing hybrid analog/digital hardware.
  • a fixed neural network is responsible for pattern detection (e.g., for generating embeddings), and it is combined with a flexible algorithm responsible for the pattern interpretation.
  • the flexible algorithm can be implemented in digital or analog hardware, and may include additional flexible neural networks.
  • the hybrid core includes (i) a fixed neuromorphic analog core that has ultra-low power consumption and low latency, for generating embeddings, and (ii) a flexible digital core for final classification.
  • Embeddings are representations containing densely packed information about sensory input, and are formed by a neural network or biological nervous system. Embeddings are formed in hidden layers of a neural network, and contain the most significant information about input data.
  • Embeddings may be used as input data for further efficient processing, for data classification, and interpretation.
  • a classification task may be accomplished in two stages.
  • a neural network G is applied to an input vector x from R N to generate an embedding v, where v is a vector with M dimensions and M ⁇ N.
  • y C(v, WC)
  • C is the final classifier
  • WC trainable parameters of C
  • y is a classification result for input vector x.
  • 80-90% of neural network weights do not change after several epochs of training a network and can be subjected to so-called transfer learning.
  • the neural network weights can be a part of new neural network with fixed weights (weights corresponding to 80% of the original neural network).
  • the rest of the neural network (e.g., 20% of the layers) can be trained separately and may be implemented using a flexible hardware system (without fixed weights).
  • This combines fixed parts of a neural network, placed as resistors into a neuromorphic analog signal processor, with flexible parts of the neural network realized in digital circuitry (e.g., in the MCU of the device, at a RISC-V processor, at an FPGA, or at a CPU), which may be integrated (e.g., on the same chip as the neuromorphic analog signal processor).
  • the flexible part of a network may be implemented by means of standard technologies and may be developed using programming languages such as Python, C/C++, Assembler and specialized frameworks (e.g., TensorFlow or Torch) and may run on conventional digital computing units, such as CPUs, GPUs, RISCs, FPGAs. or similar devices, depending on the target device and/or application.
  • the digital computing units may also perform the role of a digital controller, provide signals to interfaces and multiplexing power signals within the neuromorphic analog signal processor.
  • the flexible part may be implemented using compute-in-memory and programmable memory tiles (e.g., SST superflash memory), memristors or other types of programmable memory.
  • the flexible part may also be implemented using a classification neural network or classification algorithms at a CPU of the system, using external computation capabilities, since classification is rather simple and not resource intensive. Since the fixed part is computationally intensive and the flexible part is not, the two parts of a neural network may be logically distributed.
  • vibrational sensors are commonly used to measure vibrations in machinery, tracks, railway cars, wind turbines, and oil and gas pumps.
  • the signals due to the vibrations may be transferred wirelessly to analytic equipment.
  • the signals include a large amount of data.
  • the data flow may shorten the battery life of operating sensor nodes.
  • Analog neuromorphic hardware helps reduce the data flow from vibration sensors (sometimes up to 99.9%).
  • Some implementations use an encoder-decoder approach, and transmit, via long-range technology (sometimes referred to as LoRa or similar low power technology), embeddings extracted from the initial data.
  • the autoencoder systems and embeddings may create new classes, describing new signals of vibration sensors, even if they were not trained to recognize these types of signal patterns.
  • Analog neuromorphic hardware based on the disclosed technology may be used to implement an encoder neural network, which obtains a range of different vibration signals from various vibration sensors. The output of the analog hardware may be analyzed by a digital system to recognize machine malfunctions.
  • Some implementations detect a predetermined range of frequencies (e.g., up to 20 kHz or up to 60 kHz) and may use an extended sampling rate (e.g., up to 41 kHz). This allows early failure prediction (e.g., predictions up to one month or more in advance) for rotating or moving parts of machinery.
  • the use of the embeddings reduces the amount of data sent to a cloud infrastructure, and addresses the fundamental problem of low bandwidth required by internet-of-things (IoT) systems.
  • Compressed sensing is one of the technologies used for condition classification of rolling element bearings in rotating machines. This technology allows sampling below the Nyquist sampling rate. It is one of several methods for fault detection and classification in the compressed domain (i.e., without reconstructing the original signal).
  • Compressed sensing may be used for intelligent condition monitoring for bearing faults from highly compressed measurements using sparse over-complete features.
  • Compressed sensing may be used to produce highly compressed measurements of an original bearing dataset.
  • a deep neural network (DNN) with an unsupervised feature learning algorithm based on a sparse autoencoder may be used for learning overcomplete sparse representations of these compressed datasets.
  • the fault classification may be achieved using a softmax regression layer.
  • An autoencoder may be viewed as a source of embeddings that are subsequently processed by a classifier (a softmax regression layer).
  • FIG. 53 shows an example sparse autoencoder 5300 for bearing fault classification. Bearing vibration signals 5312 are acquired (in a data acquisition step 5310) using a compressive sampling mechanism to obtain compressed measurements.
  • the measurements are input into a first learning stage 5308, which implements a first encoder for obtaining feature representations v1 having d1 dimensions, with d1 > m (m is the number of input dimensions).
  • the feature representations v1 are input into a second learning stage 5306, which is another encoder for obtaining feature representations v 2 having d 2 dimensions, with d 2 > d 1 .
  • the feature representations v 2 are subsequently input into a softmax regression layer 5304 for classifying bearing health conditions to obtain classification results 5302.
  • the first learning stage 5308 and the second learning stage 5306 are implemented using an analog neuromorphic core using fixed weights and the softmax regression layer 5304 is implemented using a digital circuit.
  • a ResNet Convolutional Neural Network may be used for fault analysis, as shown in Figure 54.
  • a deep ResNet framework with global average pooling (GAP) (sometimes referred to as average pooling) for feature learning and fault diagnosis of rolling bearing has been proposed. See H. Hao, et al., Research on Intelligent Fault Diagnosis of Rolling Bearing Based on Improved Deep Residual Network, 2021. An example of this is shown in Figure 54.
  • the structure of the network 5400 is composed of an input layer 5402, a convolution layer 5404, a max-pooling layer 5406, eight residual blocks 5408, 5410, 5412, 5414, 5416, 5418, 5420, and 5422, followed by a global average pooling (GAP) layer 5424 and a softmax output layer 5426.
  • the network may be used to process fault data of a rolling bearing.
  • a one-dimensional time-domain signal of the rolling bearing fault signal may be input to the network 5400 to obtain a probability distribution of each failure type.
  • the ResNet CNN may be used for bearing fault classification, in which case the output of global average pooling (GAP) layer may be embeddings classified by the softmax output layer.
  • GAP global average pooling
  • Residual blocks may include Conv1D, Conv2D, BatchNorm, ReLU, and/or add layers.
  • the ResNet CNN may include a dense layer or fully-connected layer, in addition to the GAP layer.
  • a stacked noise reduction autoencoder may be utilized for abstracting characteristics from the original vibration signals, and then, the characteristics may be provided as input for a backpropagation (BP) network classifier as described in Y. Gu, et al., A Denoising Autoencoder-Based Bearing Fault Diagnosis System for Time-Domain Vibration Signals, 2021. The results output by this classifier represent different fault categories.
  • BP backpropagation
  • FIG 55 shows a flowchart for an example method 5500 for deep autoencoder training for bearing fault classification.
  • a deep autoencoder may include four layers.
  • the size of the first layer may be identical to the feature dimension size of the training data, which is 2048.
  • the shape of the entire DAE may be, for example, 2048-500- 500-10-500-500-2048, where the structure of the encoder and the structure of the decoder are symmetric.
  • the classification may be performed by removing the decoder from the pretrained DAE and by adding a soft-max layer for the back-propagation network.
  • original vibration signal data in a training set may be organized (5510).
  • the organized data may be used to build (5512) a stacked DAE with N hidden layers.
  • the DAE may be trained for N iterations (steps 5514, 5516, and 5518).
  • the DAE may be finetuned in a next step 5506.
  • Dimensionality of the output layer of the DAE may be determined (5520). The number of dimensions may be equal to the number of possible health conditions of a machine.
  • the parameters of the DAE may be fine-tuned (5522) through a BP network.
  • the trained model may be applied (5524) to test data and diagnosis results 5508 may be output (5526).
  • analog neuromorphic hardware may be used for predictive maintenance of rotating and reciprocating parts of machines.
  • the hardware may be used to generate embeddings (descriptors) from an analog 20 kHz bandwidth signal, which may be packed into an embedding having a size of 1000 bits, and then transferred by a low bandwidth channel (e.g., LoRa or other LPWAN (Low-Power Wide-Area Network) technology).
  • a low bandwidth channel e.g., LoRa or other LPWAN (Low-Power Wide-Area Network) technology.
  • Some implementations separately generate embeddings using analog neuromorphic hardware using fixed weights, according to the techniques described above. Subsequently, the signal obtained may be analyzed in a cloud or an edge cloud.
  • the cloud’s infrastructure may use a neural network (or layers of a neural network) that uses flexible weights (e.g., the neural network may be retrained). In this way, it is possible to apply fixed weights using an analog neuromorphic hardware for predictive maintenance tasks.
  • Some implementations train a neural network (or a portion thereof) so that the trained neural network uses analog neuromorphic hardware according to techniques described above, and generates descriptors for vibrational or reciprocating signals, which enables distinguishing abnormal operation of machines from normal operation.
  • Some implementations use a hybrid core including analog neuromorphic hardware according to techniques described above. The analog neuromorphic hardware implements a fixed part, in a chip that is close to the sensors.
  • an apparatus or a system that includes the hybrid core may be used for control of bearings or any rotating parts. Bearings or other rotating parts (or any movable part) may generate vibrations caused by rotation during operation.
  • the movable part may include any rotating or reciprocating part, a transverse moving part, feed mechanisms, or auxiliary parts of a machine. Periodicity in the vibrations may be used for predicting machine maintenance problems.
  • Some implementations use a vibration wireless sensor node, which is a device that measures the amplitude and frequency of vibration in equipment.
  • Such nodes may be installed on or near parts of a machine and may consist of a vibration sensor (with analog or digital output), an analog-digital converter (ADC), a communication module, and a battery. Such devices may work according to schedule, but may be time-limited because of high power consumption of a wide band communication module.
  • the communication module may transmit a large amount of raw data to analytic equipment for analysis and interpretation.
  • analog neuromorphic hardware or a hybrid core containing such analog hardware may be used as a part of a sensor node, which receives raw data coming from a sensor.
  • Such hardware may be used to pre- process raw signals and to extract embeddings. Pre-processed data may be 99.9% less when compared to the original input data size.
  • LPWAN technology like LoRa can be used instead of traditional wide band communication technology (for example Wi-Fi or Bluetooth).
  • Wi-Fi wireless fidelity
  • Bluetooth wireless fidelity
  • signals may be analyzed to determine the source of the vibration, make failure predictions, and suggest appropriate maintenance.
  • Use of ultra-low power chips provides much longer life from a battery.
  • energy harvesting technology may be used to collect power from the vibration source. In this way, the techniques described here may be used to collect and transmit vibration data much more frequently and it is possible to make earlier and more precise analytical predictions.
  • hybrid hardware may be used for the control of vibration in tires.
  • a smart tire sensor node is a device that includes a vibration sensor (with analog or digital output), a neuromorphic analog signal processor, a LoRa communication module (or other LPWAN), and a battery with or without an energy harvesting module.
  • This device may be disposed in or on a tire and may collect vibration data while a vehicle is moving.
  • Initial data coming to the neuromorphic analog signal processor or chip may be pre-processed and/or transformed into embeddings. This reduces data volume by as much as 99.9%.
  • the reduced data may be transmitted by an LPWAN module to a gateway and then to an application (e.g., an application hosted in a cloud) for further analytics and interpretation.
  • FIG. 56 shows an example system 5600 for predictive maintenance of a machine, according to some implementations.
  • the system may include a hardware apparatus 5620.
  • the hardware apparatus 5620 may include a vibration sensor 5608 configured to sense vibrations 5612 of a vibration source 5610 (e.g., a machine part) and an analog circuit 5602 comprising a plurality of operational amplifiers and a plurality of resistors.
  • the analog circuit 5602 may be coupled to the vibration sensor 5608, and configured to (i) receive an analog signal 5614 from the vibration sensor 5608 and (ii) compute an output 5616 based on the analog signal, by performing a portion of a trained neural network (e.g., the example ResNet CNN shown in Figure 54).
  • the hardware apparatus 5620 further includes a transceiver 5606 coupled to the analog circuit 5602 and configured to receive the output 5616 from the analog circuit 5602 and transmit the output 5616 over a low power wide area network (LPWAN) 5618.
  • LPWAN low power wide area network
  • the vibration sensor 5608 is disposed adjacent to a movable part (e.g., a rotating or reciprocating part) of a machine 5610 (sometimes called a vibration source), and the vibration sensor 5608 is configured to collect vibration signals 5612 from the rotating or reciprocating part.
  • the rotating part includes a ball bearing of the machine.
  • the output 5616 of the analog circuit 5602 represents embeddings used for at least one of: defining a source of vibration, predicting failures of a machine coupled to the vibration source, or generating suggestions for maintenance of the machine.
  • the vibration sensor 5608 is disposed in or on a tire and is configured to collect vibration data for the tire, which is an example of a vibration source.
  • the output 5616 represents embeddings used to predict at least one of: a road surface, a physical condition, a tire condition, a suspension condition, or a time-to-failure of the vibration source.
  • the vibration sensor 5608 is configured to sample signals in a range of 0 to 20 kilohertz (kHz).
  • kHz kilohertz
  • the vibration sensor 5608 is configured to sample signals up to 41 kHz.
  • the vibration sensor 5608 is configured to sample signals below a Nyquist sampling rate for fault detection and classification in a compressed domain. [00598] In some implementations, the vibration sensor 5608 is configured to sample signals for compressed sensing (CS) for condition classification of rolling element bearings in rotating machines. [00599] In some implementations, the trained neural network comprises a plurality of layers of neurons including a first set of layers (e.g., all the layers in the ResNet CNN shown in Figure 54 except the softmax layer) and a second set of layers (e.g., the softmax layer) and the analog circuit 5602 is configured to implement the first set of layers. In some implementations, the second set of layers consists of a last layer of the trained neural network.
  • the trained neural network includes a deep neural network for unsupervised learning based on a sparse autoencoder (e.g., the example sparse autoencoder shown and described above in reference to Figure 53).
  • the trained neural network comprises a ResNet Convolutional Neural Network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings.
  • the vibration sensor 5608 is configured to sample and output a one-dimensional time domain signal of a rolling bearing fault signal.
  • the trained neural network comprises a stacked noise reduction autoencoder.
  • the analog circuit 5602 is configured to be powered by vibrations of the vibration source 5610.
  • the hardware apparatus 5620 further includes a power harvesting circuit configured to harvest power from vibrations 5612 of the vibration source 5610 and supply power to the analog circuit 5602.
  • the plurality of operational amplifiers is configured to implement neurons of the portion of the trained neural network
  • the plurality of resistors is configured to implement axons or connections between neurons of the portion of the trained neural network.
  • the analog circuit 5602 is configured to implement an optimized neural network corresponding to the trained neural network.
  • values of the plurality of resistors are based on weights of connections of the trained neural network.
  • the plurality of resistors is configured to connect the plurality of operational amplifiers.
  • the analog circuit 5602 comprises resistors in the backend-of-the-line (BEOL).
  • the trained neural network is an autoencoder comprising an encoder portion and a decoder portion. The encoder portion reconstructs an input vector at an output layer after nonlinear transformations performed by hidden layers.
  • the analog circuit 5602 corresponds to the encoder portion of the autoencoder.
  • the encoder portion comprises the hidden layers.
  • the analog circuit 5602 is configured to compute a representation of the input vector in fewer dimensions than an input space of the input vector. [00612] In some implementations, the analog circuit 5602 is configured to generate compressed data that encodes vibration sensor data based on vibration features from the vibration sensor 5608. [00613] In another aspect, the system 5600 includes the hardware apparatus 5620, the transceiver 5606, and a digital circuit 5604. The hardware apparatus 5620 includes the vibration sensor 5608 configured to sense vibrations of the vibration source 5610 of a machine. The hardware apparatus 5620 also includes the analog circuit 5602 comprising a plurality of operational amplifiers and a plurality of resistors.
  • the analog circuit 5602 is coupled to the vibration sensor 5608 and configured to receive an analog signal 5614 from the vibration sensor 5608 and compute an output 5616 based on the analog signal 5614, by performing a portion of a trained neural network.
  • the transceiver 5606 is coupled to the analog circuit 5602 and configured to receive the output 5616 from the analog circuit 5602 and transmit the output 5616 over a low power wide area network (LPWAN) 5618.
  • LPWAN low power wide area network
  • the analog signal is converted before transmitting.
  • An analog-to-digital converter (ADC) may be included in the hardware apparatus 5620, or may be included as part of the transceiver 5606.
  • the digital circuit 5604 is communicatively coupled to the transceiver 5606 of the hardware apparatus 5620 via the LPWAN.
  • the digital circuit 5604 is configured to receive the output from the analog circuit 5602 (via the transceiver 5606) and predict the state 5624 of the machine for maintenance (e.g., by a device 5622 for maintaining the machine), based on the output.
  • the digital circuit 5604 includes one or more digital computing units, including one or more of: CPUs, GPUs, RISCs, FPGAs, and ASICs.
  • the digital circuit 5604 comprises a processor configured to perform data classification.
  • the data classification is performed by a neural network that is distinct from the trained neural network.
  • the data classification is performed using k-nearest neighbors (k-NN).
  • the output of the analog circuit 5602 represents embeddings and the digital circuit 5604 is configured to use the embeddings to classify the analog signal 5614.
  • a method is provided for vibration sensing. The method includes sensing vibrations of a vibration source using a vibration sensor (e.g., the vibration sensor 5608) to obtain an analog signal (e.g., the analog signal 5614). The method also includes computing an output (e.g., the output 5616) based on the analog signal, by performing a portion of a trained neural network, using an analog circuit (e.g., the analog circuit 5602) comprising a plurality of operational amplifiers and a plurality of resistors.
  • the method also includes transmitting the output over a low power wide area network (LPWAN) 5618 using a transceiver 5606.
  • LPWAN low power wide area network
  • transceiver 5606 The terminology used in the description of the invention herein is for the purpose of describing particular implementations only and is not intended to be limiting of the invention.
  • LPWAN low power wide area network
  • the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.
  • the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Acoustics & Sound (AREA)
  • Feedback Control In General (AREA)

Abstract

Systems, methods, and devices are provided for predictive maintenance of machines. An example apparatus includes a vibration sensor configured to sense vibrations of a vibration source and an analog circuit. The analog circuit comprises a plurality of operational amplifiers and a plurality of resistors. The analog circuit is coupled to the vibration sensor and configured to: receive an analog signal from the vibration sensor; and compute an output based on the analog signal, by performing a portion of a trained neural network.

Description

Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application is a continuation of and claims priority to U.S. Patent Application No. 17/902,757, entitled “Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines,” filed September 2, 2022, which is a continuation-in-part of US Application No.17/744,565, filed May 13, 2022, entitled “Systems and Methods for Human Activity Recognition Using Analog Neuromorphic Computing Hardware,” which is a continuation-in-part of US Application No.17/189,109, filed March 1, 2021, entitled “Analog Hardware Realization of Neural Networks,” which is a continuation of PCT Application PCT/RU2020/000306, filed June 25, 2020, entitled “Analog Hardware Realization of Neural Networks,” each of which is incorporated by reference herein in its entirety. US Application No.17/189,109 is also a continuation-in-part of PCT Application PCT/EP2020/067800, filed June 25, 2020, entitled “Analog Hardware Realization of Neural Networks,” which is incorporated by reference herein in its entirety. TECHNICAL FIELD [0002] The disclosed implementations relate generally to neural networks, and more specifically to systems and methods for predictive maintenance of machines using a neuromorphic analog signal processor. BACKGROUND [0003] Conventional hardware has failed to keep pace with innovation in neural networks and the growing popularity of machine learning based applications. The complexity of neural networks continues to outpace CPU and GPU computational power as digital microprocessor advances are plateauing. Neuromorphic processors based on spike neural networks, such as Loihi and True North, are limited in their applications. For GPU-like architectures, power and speed of such architectures are limited by data transmission speed. Data transmission can consume up to 80% of chip power, and can significantly impact the speed of calculations. Edge applications demand low power consumption, but there are currently no known performant hardware implementations that consume less than 50 milliwatts of power. [0004] Memristor-based architectures that use cross-bar technology remain impractical for manufacturing recurrent and feed-forward neural networks. For example, memristor-based cross-bars have a number of disadvantages, including high latency and leakage of currents during operation, which make them impractical. Also, there are reliability issues in manufacturing memristor-based cross-bars, especially when neural networks have both negative and positive weights. For large neural networks with many neurons, at high dimensions, memristor-based cross-bars cannot be used for simultaneous propagation of different signals, which in turn complicates summation of signals, when neurons are represented by operational amplifiers. Furthermore, memristor-based analog integrated circuits have a number of limitations, such as a small number of resistive states, first cycle problems when forming memristors, the complexity with channel formation when training the memristors, unpredictable dependency on dimensions of the memristors, slow operations of memristors, and drift of state of resistance. [0005] Additionally, the training process required for neural networks presents unique challenges for hardware realization of neural networks. A trained neural network is used for specific inferencing tasks, such as classification. Once a neural network is trained, a hardware equivalent is manufactured. When the neural network is retrained, the hardware manufacturing process is repeated, driving up costs. Although some reconfigurable hardware solutions exist, such hardware cannot be easily mass produced, and costs a lot more than hardware that is not reconfigurable (e.g., five times the cost). Further, edge environments, such as smart-home applications, do not require re-programmability as such. For example, 85% of all applications of neural networks do not require any retraining during operation, so on-chip learning is not that useful. Furthermore, edge applications include noisy environments, which can cause reprogrammable hardware to become unreliable. [0006] One such application is predictive maintenance of machines. Vibrational sensors are commonly used to measure vibrations in machinery, tracks, railway cars, wind turbines, and oil and gas pumps. Sensors that monitor rotation or reciprocating parts of machines typically generate signals in the range from 0 to 20 kHz. This makes the sensors consume a lot of power, making it almost impossible to use batteries. Signals due to the vibrations may be transferred wirelessly to an equipment for analysis. A wideband connection to a control unit is typically necessary because the signals can include a large amount of data flow. This data flow may shorten the battery life of operating sensor nodes. SUMMARY [0007] Accordingly, there is a need for methods, circuits and/or interfaces that address at least some of the deficiencies identified above. Analog circuits that model trained neural networks and manufactured according to the techniques described herein, can provide improved performance per watt advantages, can be useful in implementing hardware solutions in edge environments, and can tackle a variety of applications, such as drone navigation and autonomous cars. The cost advantages provided by the proposed manufacturing methods and/or analog network architectures are even more pronounced with larger neural networks. Also, analog hardware implementations of neural networks provide improved parallelism and neuromorphism. Moreover, neuromorphic analog components are not sensitive to noise and temperature changes, when compared to digital counterparts. [0008] Chips manufactured according to the techniques described herein provide orders of magnitude in improvement over conventional systems in size, power, and performance, and are ideal for edge environments, including for retraining purposes. Such analog neuromorphic chips can be used to implement edge computing applications or in Internet-of-Things (IoT) environments. Due to the analog hardware, initial processing (e.g., formation of descriptors for image recognition), that can consume over 80-90% of power, can be moved onto a chip, thereby decreasing energy consumption and network load, which can open up new markets for applications. [0009] Various edge applications can benefit from the use of such analog hardware. For example, for video processing, the techniques described herein can be used to include direct connection to a CMOS sensor without a digital interface. Various other video processing applications include road sign recognition for automobiles, camera-based true depth and/or simultaneous localization and mapping for robots, room access control without a server connection, and always-on solutions for security and healthcare. Such chips can be used for data processing from radars and lidars, and for low-level data fusion. Such techniques can be used to implement battery management features for large battery packs, sound/voice processing without connection to data centers, voice recognition on mobile devices, wake up speech instructions for IoT sensors, translators that translate one language to another, large sensors arrays of IoT with low signal intensity, and/or configurable process control with hundreds of sensors. In particular, such techniques can be used for implementing human activity recognition. Substituting conventional software algorithms with analog neural network implementations can help offload computation from device micro-controllers so that the total power consumption of the device is decreased substantially (e.g., power consumption decreases by over 40%). Moreover, techniques described herein can be used to distinguish a large number of human activities, and such techniques can be used for developing personalized solutions for different users or classes of users. For example, neural network determination of human activity, or generating descriptors, allows recognition of more than fifty different human activities. Neural networks allow personalization of activities, and descriptors can be generated as unique digital fingerprint for specific users. [0010] Neuromorphic analog chips can be mass produced after standard software- based neural network simulations/training, according to some implementations. A client’s neural network can be easily ported, regardless of the structure of the neural network, with customized chip design and production. Moreover, a library of ready to make on-chip solutions (network emulators) are provided, according to some implementations. Such solutions require only training, one lithographic mask change, following which chips can be mass produced. For example, during chip production, only part of the lithography masks need to be changed. [0011] The techniques described herein can be used to design and/or manufacture an analog neuromorphic integrated circuit that is mathematically equivalent to a trained neural network (either feed-forward or recurrent neural networks). According to some implementations, the process begins with a trained neural network that is first converted into a transformed network comprised of standard elements. Operations of the transformed network are simulated using software with known models representing the standard elements. The software simulation is used to determine the individual resistance values for each of the resistors in the transformed network. Lithography masks are laid out based on the arrangement of the standard elements in the transformed network. Each of the standard elements is laid out in the masks using an existing library of circuits corresponding to the standard elements to simplify and speed up the process. In some implementations, the resistors are laid out in one or more masks separate from the masks including the other elements (e.g., operational amplifiers) in the transformed network. In this manner, if the neural network is retrained, only the masks containing the resistors, or other types of fixed-resistance elements, representing the new weights in the retrained neural network need to be regenerated, which simplifies and speeds up the process. The lithography masks are then sent to a fab for manufacturing the analog neuromorphic integrated circuit. [0012] In one aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method incudes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. [0013] In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. [0014] In some implementations, the method further includes obtaining new weights for the trained neural network, computing a new weight matrix for the equivalent analog network based on the new weights, and generating a new resistance matrix for the new weight matrix. [0015] In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: for each layer of the one or more layers of neurons: (i) identifying one or more function blocks, based on the respective mathematical function, for the respective layer. Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function; and (ii) generating a respective multi-layer network of analog neurons based on arranging the one or more function blocks. Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multi-layer network is connected to one or more analog neurons of a second layer of the multi-layer network. [0016] In some implementations, the one or more function blocks include one or more basic function blocks selected from the group consisting of: (i) a weighted summation block with a block output ^^^௨௧ =
Figure imgf000008_0001
^^^ ^^ + ^^ ^^ ^^ ^^൯. ^^ ^^ ^^ ^^ is Rectified Linear Unit (ReLU) activation function or a similar activation function, ^^^ represents an i-th input, ^^^ represents a weight corresponding to the i-th input, and ^^ ^^ ^^ ^^ represents a bias value, and ^^ is a summation operator; (ii) a signal multiplier block with a block output ^^^௨௧ = ^^ ^^ ^^ ^^ ^^. ^^^ . ^^^. ^^^ represents an i-th input and ^^^ represents a j-th input, and ^^ ^^ ^^ ^^ ^^ is a predetermined coefficient; (iii) a sigmoid activation block with a block output ^^^௨௧
Figure imgf000008_0002
Figure imgf000008_0003
^^represents an input, and A and B are predetermined coefficient values of the sigmoid activation block; (iv) a hyperbolic tangent activation block with a block output ^^^௨௧ = ^^ ∗ tanh ( ^^ ∗ ^^^^). ^^^^ represents an input, and A and B are predetermined coefficient values; and (v) a signal delay block with a block output ^^( ^^) = ^^( ^^ − ^^ ^^). ^^ represents a current time-period, ^^( ^^ − ^^ ^^) represents an output of the signal delay block for a preceding time period ^^ − ^^ ^^, and ^^ ^^ is a delay value. [0017] In some implementations, identifying the one or more function blocks includes selecting the one or more function blocks based on a type of the respective layer. [0018] In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network. [0019] In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting the one or more weights such that combining the one or more intermediate functions results in the mathematical function. [0020] In some implementations, the method further includes: (i) generating equivalent digital network of digital components for one or more output layers of the neural network topology; and (ii) connecting output of one or more layers of the equivalent analog network to the equivalent digital network of digital components. [0021] In some implementations, the analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. [0022] In some implementations, selecting component values of the analog components includes performing a gradient descent method to identify possible resistance values for the plurality of resistors. [0023] In some implementations, the neural network topology includes one or more GRU or LSTM neurons, and transforming the neural network topology includes generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons. [0024] In some implementations, the one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology. [0025] In some implementations, the neural network topology includes one or more layers of neurons that perform unlimited activation functions, and transforming the neural network topology includes applying one or more transformations selected from the group consisting of: (i) replacing the unlimited activation functions with limited activation; and (ii) adjusting connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized. [0026] In some implementations, the method further includes generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix. [0027] In some implementations, the method further includes: (i) obtaining new weights for the trained neural network; (ii) computing a new weight matrix for the equivalent analog network based on the new weights; (iii) generating a new resistance matrix for the new weight matrix; and (iv) generating a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix. [0028] In some implementations, the trained neural network is trained using software simulations to generate the weights. [0029] In another aspect, a method for hardware realization of neural networks is provided, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes calculating one or more connection constraints based on analog integrated circuit (IC) design constraints. The method also includes transforming the neural network topology to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints. The method also includes computing a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network. [0030] In some implementations, transforming the neural network topology to the equivalent sparsely connected network of analog components includes deriving a possible input connection degree
Figure imgf000010_0001
and output connection degree ^^^, according to the one or more connection constraints. [0031] In some implementations, the neural network topology includes at least one densely connected layer with ^^ inputs and ^^ outputs and a weight matrix ^^. In such cases, transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with ^^ inputs, ^^ outputs, and
Figure imgf000010_0002
^^^ − 1 layers, such that input connection degree does not exceed ^^^, and output connection degree does not exceed ^^^. [0032] In some implementations, the neural network topology includes at least one densely connected layer with ^^ inputs and ^^ outputs and a weight matrix ^^. In such cases, transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with ^^ inputs, ^^ outputs, and ^^ ≥ ^^ ^^ ^^൫⌈ ^^ ^^ ^^ே^ ^^⌉, ⌈ ^^ ^^ ^^ே^ ^^⌉൯ layers. Each layer m is represented by a corresponding weight matrix ^^^, where absent connections are represented with zeros, such that input connection degree does not exceed ^^^, and output connection degree does not exceed ^^^. The equation ^^ =
Figure imgf000010_0003
^^^ is satisfied with a predetermined precision. [0033] In some implementations, the neural network topology includes a single sparsely connected layer with ^^ inputs and ^^ outputs, a maximum input connection degree of ^^^, a maximum output connection degree of ^^^, and a weight matrix of ^^, where absent connections are represented with zeros. In such cases, transforming the single sparsely connected layer includes constructing the equivalent sparsely connected network with ^^ inputs, ^^ outputs, ^^ ≥ ^^ ^^ ^^൫⌈ ^^ ^^ ^^ே^ ^^^⌉, ⌈ ^^ ^^ ^^ே^ ^^^⌉൯ layers, each layer ^^ represented by a corresponding weight matrix ^^^, where absent connections are represented with zeros, such that input connection degree does not exceed ^^^, and output connection degree does not exceed ^^^. The equation ^^ = ∏ ^ୀ^..ெ ^^^ is satisfied with a predetermined precision. [0034] In some implementations, the neural network topology includes a convolutional layer with ^^ inputs and ^^ outputs. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the convolutional layer into a single sparsely connected layer with ^^ inputs, ^^ outputs, a maximum input connection degree of ^^^, and a maximum output connection degree of ^^^. ^^^ ≤ ^^^ and ^^^ ≤ ^^^. [0035] In some implementations, generating a schematic model for implementing the equivalent sparsely connected network utilizing the weight matrix. [0036] In some implementations, the neural network topology includes a recurrent neural layer. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes transforming the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections. [0037] In some implementations, the neural network topology includes a recurrent neural layer. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with ^^ inputs and ^^ output and a weight matrix ^^, where absent connections are represented with zeros. [0038] In some implementations, the neural network topology includes K inputs, a weight vector ^^ ∈ ^^^, and a single layer perceptron with a calculation neuron with an activation function F. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating a number of layers m for the equivalent sparsely connected network using the equation ^^ = ⌈ ^^ ^^ ^^ே ^^⌉; and (iii) constructing the equivalent sparsely connected network with the K inputs, m layers and the connection degree N. The equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes calculating a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation ^^ = ^^
Figure imgf000012_0001
[0039] In some implementations, the neural network topology includes K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating number of layers m for the equivalent sparsely connected network using the equation ^^ = ⌈ ^^ ^^ ^^ே ^^⌉; (iii) decomposing the single layer perceptron into L single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network of the L single layer perceptron networks: (a) constructing a respective equivalent pyramid-like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (b) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network of the L single layer perceptron networks: (i) setting a weight vector ^^ = ^^^, ith row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network; and (ii) calculating a weight vector Wi for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation ^^ = ^^
Figure imgf000013_0001
[0040] In some implementations, the neural network topology includes K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons Li and corresponding weight matrices Vi that includes a row of weights for each calculation neuron of the Li calculation neurons. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing the multi-layer perceptron into ^^ =
Figure imgf000013_0002
single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the ^^ calculation neurons. Decomposing the multi-layer perceptron includes duplicating one or more input of the K inputs that are shared by the ^^ calculation neurons; (iii) for each single layer perceptron network of the Q single layer perceptron networks: (a) calculating a number of layers m for a respective equivalent pyramid- like sub-network using the equation ^^ = ^ ^^ ^^ ^^ே ^^^,^^. Ki, j is number of inputs for the respective calculation neuron in the multi-layer perceptron; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with Ki, j inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*Ki, j inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes: for each single layer perceptron network of the Q single layer perceptron networks: (i) setting a weight vector ^^ = ^^^ ^ , the ith row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the multi-layer perceptron; and (ii) calculating a weight vector Wi for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes Ki, j equations with riables, and S is computed using the equation ^^ = ^^ ே^ S va ି^ ^,^ ^ே^షభ(ேି^)^. [0041] In some implementations, the neural network topology includes a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons Li and corresponding weight matrices Vi that includes a row of weights for each calculation neuron of the Li calculation neurons. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing the CNN into ^^ =
Figure imgf000014_0001
single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the ^^ calculation neurons. Decomposing the CNN includes duplicating one or more input of the K inputs that are shared by the ^^ calculation neurons; (iii) for each single layer perceptron network of the Q single layer perceptron networks: (a) calculating number of layers m for a respective equivalent pyramid- like sub-network using the equation ^^ = ^ ^^ ^^ ^^ே ^^^,^^. j is the corresponding layer of the respective calculation neuron in the CNN, and Ki, j is number of inputs for the respective calculation neuron in the CNN; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with Ki, j inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*Ki, j inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network of the Q single layer perceptron networks: (i) setting a weight vector ^^ = ^^^ th ^ , the i row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the CNN; and (ii) calculating weight vector Wi for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes Ki, j equations with S variables, and S is ^ computed using the equation ^^ = ^^ ே ି^ ^,^ ^ே^షభ(ேି^)^. [0042] In some implementations, the neural network topology includes K inputs, a layer Lp with K neurons, a layer Ln with L neurons, and a weight matrix ^^ ∈ ^^^×^, where R is the set of real numbers, each neuron of the layer Lp is connected to each neuron of the layer Ln, each neuron of the layer Ln performs an activation function F, such that output of the layer Ln is computed using the equation ^^^ = ^^( ^^. ^^) for an input ^^. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes performing a trapezium transformation that includes: (i) deriving a possible input connection degree ^^ூ > 1 and a possible output connection degree ^^ை > 1, according to the one or more connection constraints; (ii) in accordance with a determination that ^^ ⋅ ^^ < ^^ ⋅ ^^ூ + ^^ ⋅ ^^ை, constructing a three-layered analog network that includes a layer LAp with K analog neurons performing identity activation function, a layer LAh with ^^ = analog neurons performing identity activation function, and a layer LAo
Figure imgf000015_0001
with L analog neurons performing the activation function F, such that each analog neuron in the layer LAp has NO outputs, each analog neuron in the layer LAh has not more than NI inputs and NO outputs, and each analog neuron in the layer LAo has NI inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes generating a sparse weight matrices ^^^ and ^^^ by solving a matrix equation ^^^. ^^^ = ^^ that includes ^^ ⋅ ^^ equations in ^^ ⋅ ^^ை + ^^ ⋅ ^^ூ variables, so that the total output of the layer LAo is calculated using the equation ^^^ = ^^( ^^^. ^^^. ^^). The sparse weight matrix ^^^ ∈ ^^^×ெ represents connections between the layers LAp and LAh, and the sparse weight matrix ^^^ ∈ ^^ெ×^ represents connections between the layers LAh and LAo,. [0043] In some implementations, performing the trapezium transformation further includes: in accordance with a determination that ^^ ⋅ ^^ ≥ ^^ ⋅ ^^ூ + ^^ ⋅ ^^ை: (i) splitting the layer Lp to obtain a sub-layer Lp1 with K’ neurons and a sub-layer Lp2 with (K - K’) neurons such that ^^′ ⋅ ^^ ≥ ^^ ⋅ ^^ூ + ^^′ ⋅ ^^ை; (ii) for the sub-layer Lp1 with K’ neurons, performing the constructing, and generating steps; and (iii) for the sub-layer Lp2 with K - K’ neurons, recursively performing the splitting, constructing, and generating steps. [0044] In some implementations, the neural network topology includes a multilayer perceptron network. In such cases, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. [0045] In some implementations, the neural network topology includes a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non-linear function calculation. In such cases, the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation. [0046] In some implementations, the neural network topology includes a long short- term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations. In such cases, the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations. [0047] In some implementations, the neural network topology includes a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers and (ii) one or more fully-connected layers. In such cases, the method further includes: (i) transforming the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and (ii) for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. [0048] In some implementations, the neural network topology includes K inputs, L output neurons, and a weight matrix ^^ ∈ ^^^×^, where R is the set of real numbers, each output neuron performs an activation function F. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes performing an approximation transformation that includes: (i) deriving a possible input connection degree ^^ூ > 1 and a possible output connection degree ^^ை > 1, according to the one or more connection constraints; (ii) selecting a parameter p from the set {0, 1, … ,
Figure imgf000016_0001
− 1}; (iii) in accordance with a determination that p > 0, constructing a pyramid neural network that forms first p layers of the equivalent sparsely connected network, such that the pyramid neural network has ^^^ = ^ ^^/ ^^^ ூ ^ neurons in its output layer. Each neuron in the pyramid neural network performs identity function; and (iv) constructing a trapezium neural network with ^^^ inputs and ^^ outputs. Each neuron in the last layer of the trapezium neural network performs the activation function F and all other neurons perform identity function. In such cases, computing the weight matrix for the equivalent sparsely connected network includes: (i) generating weights for the pyramid neural network including (a) setting weights of every neuron i of the first layer of the pyramid neural network according to following rule: (a) ^^ (^) ^^^ = ^^. ^^ is a non-zero constant and ^^^ = ( ^^ − 1) ^^ூ + 1; and (b) for all weights j of the neuron except k୧; and (b) setting all other weights
Figure imgf000017_0001
of the pyramid neural network to 1; and (ii) generating weights for the trapezium neural network including (a) setting weights of each neuron i of the first layer of the trapezium neural network according to the equation
Figure imgf000017_0002
and (b) setting other weights of the neural network to 1. [0049] In some implementations, the neural network topology includes a multilayer perceptron with the ^^ inputs, ^^ layers, and ^^^ୀ^,ௌ calculation neurons in i-th layer, and a weight matrix ^^^ୀ^,ௌ ∈ ^^^^×^^షభ for the i-th layer, where ^^^ = ^^. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer ^^ of the ^^ layers of the multilayer perceptron: (i) constructing a respective pyramid-trapezium network PTNNXj by performing the approximation transformation to a respective single layer perceptron consisting of ^^^ି^inputs, ^^^ output neurons, and a weight matrix ^^^; and (ii) constructing the equivalent sparsely connected network by stacking each pyramid trapezium network. [0050] In another aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. The method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. [0051] In some implementations, generating the resistance matrix for the weight matrix includes: (i) obtaining a predetermined range of possible resistance values { ^^^^^, ^^^^௫} and selecting an initial base resistance value ^^^^^^ within the predetermined range; (ii) selecting a limited length set of resistance values, within the predetermined range, that provide most uniform distribution of possible weights ^^ ^ ^,^ =
Figure imgf000018_0001
− ோೕ^ within the range [− ^^^^^^, ^^^^^^] for all combinations of { ^^^, ^^^} within the limited length set of resistance values; (iii) selecting a resistance value ^^ା = ^^ି, from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias ^^^^௫ of each neuron or for each layer of the equivalent analog network, such that ^^ା = ^^ି is the closest resistor set value to ^^^^^^ ∗ ^^^^௫; and (iv) for each element of the weight matrix, selecting a respective first resistance value ^^^ and a respective second resistance value ^^ଶ that minimizes an error according to for all possible values of ^^ and ^^ within
Figure imgf000018_0002
^ ଶ the predetermined range of possible resistance values. w is the respective element of the weight matrix, and ^^^^^ is a predetermined relative tolerance value for resistances. [0052] In some implementations, the predetermined range of possible resistance values includes resistances according to nominal series E24 in the range 100 KΩ to 1 MΩ. [0053] In some implementations, ^^ା and ^^ି are chosen independently for each layer of the equivalent analog network. [0054] In some implementations, ^^ା and ^^ି are chosen independently for each analog neuron of the equivalent analog network. [0055] In some implementations, a first one or more weights of the weight matrix and a first one or more inputs represent one or more connections to a first operational amplifier of the equivalent analog network. In such cases, the method further includes, prior to generating the resistance matrix: (i) modifying the first one or more weights by a first value; and (ii) configuring the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function. [0056] In some implementations, the method further includes: (i) obtaining a predetermined range of weights; and (ii) updating the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input. [0057] In some implementations, the trained neural network is trained so that each layer of the neural network topology has quantized weights. [0058] In some implementations, the method further includes retraining the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network. [0059] In some implementations, the method further includes retraining the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. [0060] In another aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. The method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix. The method also includes pruning the equivalent analog network to reduce number of the plurality of operational amplifiers or the plurality of resistors, based on the resistance matrix, to obtain an optimized analog network of analog components. [0061] In some implementations, pruning the equivalent analog network includes substituting, with conductors, resistors corresponding to one or more elements of the resistance matrix that have resistance values below a predetermined minimum threshold resistance value. [0062] In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value. [0063] In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the weight matrix that are approximately zero. [0064] In some implementations, pruning the equivalent analog network further includes removing one or more analog neurons of the equivalent analog network without any input connections. [0065] In some implementations, pruning the equivalent analog network includes: (i) ranking analog neurons of the equivalent analog network based on detecting use of the analog neurons when making calculations for one or more data sets; (ii) selecting one or more analog neurons of the equivalent analog network based on the ranking; and (iii) removing the one or more analog neurons from the equivalent analog network. [0066] In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring propagation of analog signals by using the model to generate calculations for the one or more data sets. [0067] In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring output signals of the model by using the model to generate calculations for the one or more data sets. [0068] In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring power consumed by the analog neurons by using the model to generate calculations for the one or more data sets. [0069] In some implementations, the method further includes subsequent to pruning the equivalent analog network, and prior to generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network, recomputing the weight matrix for the equivalent analog network and updating the resistance matrix based on the recomputed weight matrix. [0070] In some implementations, the method further includes, for each analog neuron of the equivalent analog network: (i) computing a respective bias value for the respective analog neuron based on the weights of the trained neural network, while computing the weight matrix; (ii) in accordance with a determination that the respective bias value is above a predetermined maximum bias threshold, removing the respective analog neuron from the equivalent analog network; and (iii) in accordance with a determination that the respective bias value is below a predetermined minimum bias threshold, replacing the respective analog neuron with a linear junction in the equivalent analog network. [0071] In some implementations, the method further includes reducing number of neurons of the equivalent analog network, prior to generating the weight matrix, by increasing number of connections from one or more analog neurons of the equivalent analog network. [0072] In some implementations, the method further includes pruning the trained neural network to update the neural network topology and the weights of the trained neural network, prior to transforming the neural network topology, using pruning techniques for neural networks, so that the equivalent analog network includes less than a predetermined number of analog components. [0073] In some implementations, the pruning is performed iteratively taking into account accuracy or a level of match in output between the trained neural network and the equivalent analog network. [0074] In some implementations, the method further includes, prior to transforming the neural network topology to the equivalent analog network, performing network knowledge extraction. [0075] In another aspect, an integrated circuit is provided, according to some implementations. The integrated circuit includes an analog network of analog components fabricated by a method that includes: (i) obtaining a neural network topology and weights of a trained neural network; (ii) transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron; (iii) computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection; (iv) generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix; (v) generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix; and (vi) fabricating the circuit based on the one or more lithographic masks using a lithographic process. [0076] In some implementations, the integrated circuit further includes one or more digital to analog converters configured to generate analog input for the equivalent analog network of analog components based on one or more digital. [0077] In some implementations, the integrated circuit further includes an analog signal sampling module configured to process 1-dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit. [0078] In some implementations, the integrated circuit further includes a voltage converter module to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers. [0079] In some implementations, the integrated circuit further includes a tact signal processing module configured to process one or more frames obtained from a CCD camera. [0080] In some implementations, the trained neural network is a long short-term memory (LSTM) network. In such cases, the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing. [0081] In some implementations, the integrated circuit further includes one or more analog to digital converters configured to generate digital signal based on output of the equivalent analog network of analog components. [0082] In some implementations, the integrated circuit further includes one or more signal processing modules configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications. [0083] In some implementations, the trained neural network is trained, using training datasets containing signals of arrays of gas sensors on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected. In such cases, the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections. [0084] In some implementations, the trained neural network is trained, using training datasets containing thermal aging time series data for different MOSFETs, for predicting remaining useful life (RUL) of a MOSFET device. In such cases, the neural network topology includes 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons, and (v) between 123,000 and 124,000 connections. [0085] In some implementations, the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS). In such cases, the neural network topology includes an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons, and (v) between 51,000 and 52,000 connections. [0086] In some implementations, the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS). In such cases, the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons, and (v) between 2,200 and 2,400 connections. [0087] In some implementations, the trained neural network is trained, using training datasets containing speech commands, for identifying voice commands. In such cases, the neural network topology is a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections. [0088] In some implementations, the trained neural network is trained, using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor, for determining pulse rate during physical exercises based on PPG sensor data and 3- axis accelerometer data. In such cases, the neural network topology includes two Conv1D layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively. In such cases, the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons, and (vi) between 12,000 and 12,500 connections. [0089] In some implementations, the trained neural network is trained to classify different objects based on pulsed Doppler radar signal. In such cases, the neural network topology includes multi-scale LSTM neural network. [0090] In some implementations, the trained neural network is trained to perform human activity type recognition, based on inertial sensor data. In such cases, the neural network topology includes three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes. In such cases, the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons, and (vi) between 20,000 and 21,000 connections. [0091] In some implementations, the trained neural network is further trained to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation. [0092] In another aspect, a method is provided for generating libraries for hardware realization of neural networks. The method includes obtaining a plurality of neural network topologies, each neural network topology corresponding to a respective neural network. The method also includes transforming each neural network topology to a respective equivalent analog network of analog components. The method also includes generating a plurality of lithographic masks for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components. [0093] In some implementations, the method further includes obtaining a new neural network topology and weights of a trained neural network. The method also includes selecting one or more lithographic masks from the plurality of lithographic masks based on comparing the new neural network topology to the plurality of neural network topologies. The method also includes computing a weight matrix for a new equivalent analog network based on the weights. The method also includes generating a resistance matrix for the weight matrix. The method also includes generating a new lithographic mask for fabricating a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithographic masks. [0094] In some implementations, the new neural network topology includes a plurality of subnetwork topologies, and selecting the one or more lithographic masks is further based on comparing each subnetwork topology with each network topology of the plurality of network topologies. [0095] In some implementations, one or more subnetwork topologies of the plurality of subnetwork topologies fails to compare with any network topology of the plurality of network topologies. In such cases, the method further includes: (i) transforming each subnetwork topology of the one or more subnetwork topologies to a respective equivalent analog subnetwork of analog components; and (ii) generating one or more lithographic masks for fabricating one or more circuits, each circuit of the one or more circuits implementing a respective equivalent analog subnetwork of analog components. [0096] In some implementations, transforming a respective network topology to a respective equivalent analog network includes: (i) decomposing the respective network topology to a plurality of subnetwork topologies; (ii) transforming each subnetwork topology to a respective equivalent analog subnetwork of analog components; and (iii) composing each equivalent analog subnetwork to obtain the respective equivalent analog network. [0097] In some implementations, decomposing the respective network topology includes identifying one or more layers of the respective network topology as the plurality of subnetwork topologies. [0098] In some implementations, each circuit is obtained by: (i) generating schematics for a respective equivalent analog network of analog components; and (ii) generating a respective circuit layout design based on the schematics. [0099] In some implementations, the method further includes combining one or more circuit layout designs prior to generating the plurality of lithographic masks for fabricating the plurality of circuits. [00100] In another aspect, a method is provided for optimizing energy efficiency of analog neuromorphic circuits, according to some implementations. The method includes obtaining an integrated circuit implementing an analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. The analog network represents a trained neural network, each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. The method also include generating inferences using the integrated circuit for a plurality of test inputs, including simultaneously transferring signals from one layer to a subsequent layer of the analog network. The method also includes, while generating inferences using the integrated circuit: (i) determining if a level of signal output of the plurality of operational amplifiers is equilibrated; and (ii) in accordance with a determination that the level of signal output is equilibrated: (a) determining an active set of analog neurons of the analog network influencing signal formation for propagation of signals; and (turning off power for one or more analog neurons of the analog network, distinct from the active set of analog neurons, for a predetermined period of time. [00101] In some implementations, determining the active set of analog neurons is based on calculating delays of signal propagation through the analog network. [00102] In some implementations, determining the active set of analog neurons is based on detecting the propagation of signals through the analog network. [00103] In some implementations, the trained neural network is a feed-forward neural network, and the active set of analog neurons belong to an active layer of the analog network, and turning off power includes turning off power for one or more layers prior to the active layer of the analog network. [00104] In some implementations, the predetermined period of time is calculated based on simulating propagation of signals through the analog network, accounting for signal delays. [00105] In some implementations, the trained neural network is a recurrent neural network (RNN), and the analog network further includes one or more analog components other than the plurality of operational amplifiers, and the plurality of resistors. In such cases, the method further includes, in accordance with a determination that the level of signal output is equilibrated, turning off power, for the one or more analog components, for the predetermined period of time. [00106] In some implementations, the method further includes turning on power for the one or more analog neurons of the analog network after the predetermined period of time. [00107] In some implementations, determining if the level of signal output of the plurality of operational amplifiers is equilibrated is based on detecting if one or more operational amplifiers of the analog network is outputting more than a predetermined threshold signal level. [00108] In some implementations, the method further includes repeating the turning off for the predetermined period of time and turning on the active set of analog neurons for the predetermined period of time, while generating the inferences. [00109] In some implementations, the method further includes: (i) in accordance with a determination that the level of signal output is equilibrated, for each inference cycle: (a) during a first time interval, determining a first layer of analog neurons of the analog network influencing signal formation for propagation of signals; and (b) turning off power for a first one or more analog neurons of the analog network, prior to the first layer, for the predetermined period of time; and (ii) during a second time interval subsequent to the first time interval, turning off power for a second one or more analog neurons including the first layer of analog neurons and the first one or more analog neurons of the analog network, for the predetermined period. [00110] In some implementations, the one or more analog neurons consist of analog neurons of a first one or more layers of the analog network, and the active set of analog neurons consist of analog neurons of a second layer of the analog network, and the second layer of the analog network is distinct from layers of the first one or more layers. [00111] In another aspect, a method is provided for recognizing human activities. The method includes using one or more sensors to track activity of a user, including obtaining a plurality of electrical signals from the one or more sensors. The method also includes forming a feature vector by extracting a plurality of features from the plurality of electrical signals. The features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities. The method also includes applying an analog neurocomputing hardware device to the feature vector to generate an embedding vector that specifies a descriptor. The analog neurocomputing hardware device implements the trained neural network model. The method also includes applying a trained machine learning classifier to the embedding vector to classify the activity of the user as one of the predefined human activities. [00112] In some implementations, the trained neural network model is an autoencoder that includes an encoder and a decoder. [00113] In some implementations, the trained machine learning classifier is a KNN (K-Nearest Neighbors) classifier. In some implementations, a number of neighbors for the KNN classifier equals five. In some implementations, the trained machine learning classifier is trained separately for each of the predefined human activities using binary classification. [00114] In some implementations, the one or more sensors include one or more of IMUs, cameras, microphones, and biofeedback devices. [00115] In some implementations, the method further includes smoothing an output of the trained machine learning classifier to obtain a basic class of activity. [00116] In some implementations, the analog neurocomputing hardware device is fabricated by steps including: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model, wherein each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating an integrated circuit, according to the schematic model, using a lithographic process. [00117] In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. [00118] In some implementations, the trained machine learning classifier is implemented using one or more digital components and the trained machine learning classifier can be retrained for new users. [00119] In another aspect, a method is provided for recognizing human activities. The method includes obtaining a sequence of electrical signals from one or more sensors that track activity of a user. The method also includes forming a plurality of feature vectors by extracting features from the sequence of electrical signals. The features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities. The method also includes applying the analog neurocomputing hardware device to the plurality of feature vectors to generate a plurality of embedding vectors that each specify a corresponding descriptor. The method also includes using the plurality of embedding vectors for classifying the activity of the user as one of the predefined human activities. [00120] In some implementations, the method further includes: receiving, from the user, a set of descriptors that describes specific physical activities; and using the set of descriptors and the plurality of embedding vectors to classify the activity of the user as one of the specific physical activities. In some implementations, the method further includes generating statistics of personal daily routines of the user based on classifying the activity of the user as one of the specific physical activities. [00121] In some implementations, the method further includes: storing, for the user, the plurality of embedding vectors as describing a specific activity; and using the plurality of embedding vectors for classifying subsequent activities of the user as the specific activity. [00122] In some implementations, the method further includes: receiving, from a trainer distinct from the user, a set of descriptors that describes a specific activity; and providing feedback to the user if the activity matches the specific activity based on the plurality of embedding vectors and the set of descriptors. [00123] In another aspect, a human activity recognition device is provided. The device includes an integrated circuit for human activity recognition. The integrated circuit includes an analog network of analog components configured to implement a trained neural network model that is trained to generate a plurality of descriptors for a plurality of predefined human activities based on a plurality of features extracted from a plurality of electrical signals from one or more sensors. The device also includes one or more digital components configured to classify human activity as one of the plurality of predefined human activities according to the plurality of descriptors generated by the integrated circuit. [00124] In some implementations, the human activity recognition device further includes the one or more sensors configured to collect the plurality of electrical signals during the human activity. [00125] In some implementations, the trained neural network model is an autoencoder that includes an encoder and a decoder. [00126] In some implementations, the one or more digital components implement a trained machine learning classifier that is a KNN (K-Nearest Neighbors) classifier which can be retrained. [00127] In some implementations, a number of neighbors for the KNN classifier equals five. [00128] In some implementations, the trained machine learning classifier is trained separately for each of the plurality of predefined human activities using binary classification. [00129] In some implementations, the one or more digital components are further configured to smooth the output of the trained machine learning classifier to obtain a basic class of activity. [00130] In some implementations, the one or more sensors include one or more of IMUs, cameras, microphones, and biofeedback devices. [00131] In some implementations, the integrated circuit is fabricated by steps comprising: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model. Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating the integrated circuit, according to the schematic model, using a lithographic process. In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. [00132] In another aspect, a system is provided for predictive maintenance of a machine, according to some implementations. The system may include a hardware apparatus. The hardware apparatus may include a vibration sensor configured to sense vibrations of a vibration source and an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors. The analog circuit may be coupled to the vibration sensor, and configured to (i) receive an analog signal from the vibration sensor and (ii) compute an output based on the analog signal, by performing a portion of a trained neural network. [00133] In some implementations, the hardware apparatus further includes a transceiver coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN). [00134] In some implementations, the vibration sensor is disposed adjacent to a movable part (e.g., a rotating or reciprocating part) of a machine (sometimes called a vibration source), and the vibration sensor is configured to collect vibration signals from the rotating or reciprocating part. In some implementations, the rotating part includes a ball bearing of the machine. [00135] In some implementations, the output of the analog circuit represents embeddings used for at least one of: defining a source of vibration, predicting failures of a machine coupled to the vibration source, and generating suggestions for maintenance of the machine. [00136] In some implementations, the vibration sensor is disposed in or on a tire and is configured to collect vibration data for the tire, which is an example of a vibration source. [00137] In some implementations, the output represents embeddings used to predict at least one of: a road surface, a physical condition, a tire condition, a suspension condition, or a time-to-failure of the vibration source. [00138] In some implementations, the vibration sensor is configured to sample signals in a range of 0 to 20 kilohertz (kHz). [00139] In some implementations, the vibration sensor is configured to sample signals up to 41 kHz. [00140] In some implementations, the vibration sensor is configured to sample signals below a Nyquist sampling rate for fault detection and classification in a compressed domain. [00141] In some implementations, the vibration sensor is configured to sample signals for compressed sensing (CS) for condition classification of rolling element bearings in rotating machines. [00142] In some implementations, the trained neural network comprises a plurality of layers of neurons including a first set of layers and a second set of layers and the analog circuit is configured to implement the first set of layers. In some implementations, the second set of layers consists of a last layer of the trained neural network. [00143] In some implementations, the trained neural network includes a deep neural network for unsupervised learning based on a sparse autoencoder. [00144] In some implementations, the trained neural network comprises a ResNet Convolutional Neural Network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings. [00145] In some implementations, the vibration sensor is configured to sample and output a one-dimensional time domain signal of a rolling bearing fault signal. [00146] In some implementations, the trained neural network comprises a stacked noise reduction autoencoder. [00147] In some implementations, the analog circuit is configured to be powered by vibrations of the vibration source. [00148] In some implementations, the hardware apparatus further includes a power harvesting circuit configured to harvest power from vibrations of the vibration source and supply power to the analog circuit. [00149] In some implementations, the plurality of operational amplifiers is configured to implement neurons of the portion of the trained neural network, and the plurality of resistors is configured to implement axons or connections between neurons of the portion of the trained neural network. [00150] In some implementations, the analog circuit is configured to implement an optimized neural network corresponding to the trained neural network. [00151] In some implementations, values of the plurality of resistors are based on weights of connections of the trained neural network. [00152] In some implementations, the plurality of resistors is configured to connect the plurality of operational amplifiers. [00153] In some implementations, the analog circuit comprises resistors in a backend- of-the-line (BEOL). [00154] In some implementations, the trained neural network is an autoencoder comprising an encoder portion and a decoder portion. The encoder portion reconstructs an input vector at an output layer after nonlinear transformations performed by hidden layers. The analog circuit corresponds to the encoder portion of the autoencoder. The encoder portion comprises the hidden layers. The analog circuit is configured to compute a representation of the input vector in fewer dimensions than an input space of the input vector. [00155] In some implementations, the analog circuit is configured to generate compressed data that encodes vibration sensor data based on vibration features from the vibration sensor. [00156] In another aspect, a system includes a hardware apparatus, a transceiver and a digital circuit. The hardware apparatus includes a vibration sensor configured to sense vibrations of the vibration source of a machine. The hardware apparatus also includes an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors. The analog circuit is coupled to the vibration sensor and is configured to: receive an analog signal from the vibration sensor; and compute an output based on the analog signal, by performing a portion of a trained neural network. The transceiver is coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN). The analog signal is converted before transmitting. An analog-to-digital converter (ADC) may be included in the hardware apparatus, or may be included as part of the transceiver. The digital circuit is communicatively coupled to the transceiver of the hardware apparatus via the LPWAN. The digital circuit is configured to receive the output from the analog circuit (via the transceiver) and predict the state of the machine for maintenance, based on the output. [00157] In some implementations, the digital circuit includes one or more digital computing units selected from the group consisting of: CPUs, GPUs, RISCs, FPGAs, and ASICs. [00158] In some implementations, the digital circuit comprises a processor configured to perform data classification. In some implementations, the data classification is performed by a neural network that is distinct from the trained neural network. In some implementations, the data classification is performed using k-nearest neighbors (k-NN). [00159] In some implementations, the output of the analog circuit represents embeddings and the digital circuit is configured to use the embeddings to classify the analog signal. [00160] In some implementations, a method is provided for vibration sensing. The method includes sensing vibrations of a vibration source using a vibration sensor to obtain an analog signal. The method also includes computing an output based on the analog signal, by performing a portion of a trained neural network, using an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors. The method also includes transmitting the output over a low power wide area network (LPWAN) using a transceiver. [00161] In some implementations, a computer system has one or more processors, memory, and a display. The one or more programs include instructions for performing any of the methods described herein. [00162] In some implementations, a non-transitory computer readable storage medium stores one or more programs configured for execution by a computer system having one or more processors, memory, and a display. The one or more programs include instructions for performing any of the methods described herein. [00163] Thus, methods, systems, and devices are disclosed that are used for hardware realization of trained neural networks. BRIEF DESCRIPTION OF THE DRAWINGS [00164] For a better understanding of the aforementioned systems, methods, and graphical user interfaces, as well as additional systems, methods, and graphical user interfaces that provide data visualization analytics and data preparation, reference should be made to the Description of Implementations below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures. [00165] Figure 1A is a block diagram of a system for hardware realization of trained neural networks using analog components, according to some implementations. Figure 1B is a block diagram of an alternative representation of the system of Figure 1A for hardware realization of trained neural networks using analog components, according to some implementations. Figure 1C is a block diagram of another representation of the system of Figure 1A for hardware realization of trained neural networks using analog components, according to some implementations. [00166] Figure 2A is a system diagram of a computing device in accordance with some implementations. Figure 2B shows optional modules of the computing device, according to some implementations. [00167] Figure 3A shows an example process for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations. Figure 3B shows an example manual prototyping process used for generating a target chip model, according to some implementations. [00168] Figures 4A, 4B, and 4C show examples of neural networks that are transformed to mathematically equivalent analog networks, according to some implementations. [00169] Figure 5 shows an example of a math model for a neuron, according to some implementations. [00170] Figures 6A-6C illustrate an example process for analog hardware realization of a neural network for computing an XOR of input values, according to some implementations. [00171] Figure 7 shows an example perceptron, according to some implementations. [00172] Figure 8 shows an example Pyramid-Neural Network, according to some implementations. [00173] Figure 9 shows an example Pyramid Single Neural Network, according to some implementations. [00174] Figure 10 shows an example of a transformed neural network, according to some implementations. [00175] Figures 11A-11C show an application of a T-transformation algorithm for a single layer neural network, according to some implementations. [00176] Figure 12 shows an example Recurrent Neural Network (RNN), according to some implementations. [00177] Figure 13A is a block diagram of a LSTM neuron, according to some implementations. [00178] Figure 13B shows delay blocks, according to some implementations. [00179] Figure 13C is a neuron schema for a LSTM neuron, according to some implementations. [00180] Figure 14A is a block diagram of a GRU neuron, according to some implementations. [00181] Figure 14B is a neuron schema for a GRU neuron, according to some implementations. [00182] Figures 15A and 15B are neuron schema of variants of a single Conv1D filter, according to some implementations. [00183] Figure 16 shows an example architecture of a transformed neural network, according to some implementations. [00184] Figures 17A – 17C provide example charts illustrating dependency between output error and classification error or weight error, according to some implementations. [00185] Figure 18 provides an example scheme of a neuron model used for resistors quantization, according to some implementations. [00186] Figure 19A shows a schematic diagram of an operational amplifier made on CMOS, according to some implementations. Figure 19B shows a table of description for the example circuit shown in Figure 19A, according to some implementations. [00187] Figures 20A-20E show a schematic diagram of a LSTM block, according to some implementations. Figure 20F shows a table of description for the example circuit shown in Figure 20A-20D, according to some implementations. [00188] Figures 21A-21I show a schematic diagram of a multiplier block, according to some implementations. Figure 21J shows a table of description for the schematic shown in Figures 21A-21I, according to some implementations. [00189] Figure 22A shows a schematic diagram of a sigmoid neuron, according to some implementations. Figure 22B shows a table of description for the schematic diagram shown in Figure 22A, according to some implementations. [00190] Figure 23A shows a schematic diagram of a hyperbolic tangent function block, according to some implementations. Figure 23B shows a table of description for the schematic diagram shown in Figure 23A, according to some implementations. [00191] Figures 24A-24C show a schematic diagram of a single neuron CMOS operational amplifier, according to some implementations. Figure 24D shows a table of description for the schematic diagram shown in Figure 24A-24C, according to some implementations. [00192] Figures 25A-25D show a schematic diagram of a variant of a single neuron CMOS operational amplifiers according to some implementations. Figure 25E shows a table of description for the schematic diagram shown in Figure 25A-25D, according to some implementations. [00193] Figures 26A-26K show example weight distribution histograms, according to some implementations. [00194] Figures 27A-27J show a flowchart of a method for hardware realization of neural networks, according to some implementations. [00195] Figures 28A-28S show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations. [00196] Figures 29A-29F show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations. [00197] Figures 30A-30M show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations. [00198] Figures 31A-31Q show a flowchart of a method for fabricating an integrated circuit that includes an analog network of analog components, according to some implementations. [00199] Figures 32A-32E show a flowchart of a method for generating libraries for hardware realization of neural networks, according to some implementations. [00200] Figures 33A-33K show a flowchart of a method for optimizing energy efficiency of analog neuromorphic circuits (that model trained neural networks), according to some implementations. [00201] Figure 34 shows a table describing the MobileNet v1 architecture, according to some implementations. [00202] Figures 35 shows an example 1-D convolutional neural network used for voice clarification, according to some implementations. [00203] Figure 36 shows an example T-transformation of fully connected layers of the neural network shown in Figure 35, according to some implementations. [00204] Figure 37 shows an example T-transformation of 1-D convolution of the neural network shown in Figure 35, according to some implementations. [00205] Figure 38 shows an example T-transformation of a MaxPooling operator of the neural network shown in Figure 35, according to some implementations. [00206] Figure 39 shows an example of splitting a dataset to a training dataset and a validation dataset, according to some implementations [00207] Figure 40 shows two graph plots of activity labels, according to some implementations. [00208] Figures 41A and 41B show an example graph plot for splitting experiments to training or testing test data, according to some implementations. [00209] Figures 42A, 42B, and 42C show schematic diagrams of an example encoder, an example decoder, and an example classifier, respectively, according to some implementations. [00210] Figures 43A-43F show graphs of results, according to some implementations. [00211] Figures 44A-44D show graphs of results, according to some implementations. [00212] Figures 45A and 45B show a table that presents predictions for a base class activity classifier model, according to some implementations. [00213] Figures 46A-46F show graphs of results, according to some implementations. [00214] Figures 47A-47F show graphs of results, according to some implementations. [00215] Figures 49A-49D show graphs of results, according to some implementations. [00216] Figure 48 shows a visualization for t-distributed stochastic neighbor embedding (t-SNE) by exercise sets for embedding different approaches of the crossfit_situp exercise in experiment shown in Figures 47E and 47F, according to some implementations. [00217] Figure 50 shows an example human activity recognition device, according to some implementations. [00218] Figure 51 shows a flowchart of an example method for recognizing human activities, according to some implementations. [00219] Figure 52 shows a flowchart of another example method for recognizing human activities, according to some implementations. [00220] Figure 53 shows an example sparse autoencoder for bearing fault classification, according to some implementations. [00221] Figure 54 shows an example deep ResNet framework with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings, according to some implementations. [00222] Figure 55 shows a flowchart for an example method for deep autoencoder training for bearing fault classification, according to some implementations. [00223] Figure 56 shows an example system for predictive maintenance of a machine, according to some implementations. [00224] Reference will now be made to implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without requiring these specific details. DESCRIPTION OF IMPLEMENTATIONS [00225] Figure 1A is a block diagram of a system 100 for hardware realization of trained neural networks using analog components, according to some implementations. The system includes transforming (126) trained neural networks 102 to analog neural networks 104. In some implementations, analog integrated circuit constraints 184 constrain (146) the transformation (126) to generate the analog neural networks 104. Subsequently, the system derives (calculates or generates) weights 106 for the analog neural networks 104 by a process that is sometimes called weight quantization (128). In some implementations, the analog neural network includes a plurality of analog neuron, each analog neuron represented by an analog component, such as an operational amplifier, and each analog neuron connected to another analog neuron via a connection. In some implementations, the connections are represented using resistors that reduce the current flow between two analog neurons. In some implementations, the system transforms (148) the weights 106 to resistance values 112 for the connections. The system subsequently generates (130) one or more schematic models 108 for implementing the analog neural networks 104 based on the weights 106. In some implementations, the system optimizes resistance values 112 (or the weights 106) to form optimized analog neural networks 114 which is further used to generate (150) the schematic models 108. In some implementations, the system generates (132) lithographic masks 110 for the connections and/or generates (136) lithographic masks 120 for the analog neurons. In some implementations, the system fabricates (134 and/or 138) analog integrated circuits 118 that implement the analog neural networks 104. In some implementations, the system generates (152) libraries of lithographic masks 116 based on the lithographic masks for connections 110 and/or lithographic masks 120 for the analog neurons. In some implementations, the system uses (154) the libraries of lithographic masks 116 to fabricate the analog integrated circuits 118. In some implementations, when the trained neural networks 142 are retrained (142), the system regenerates (or recalculates) (144) the resistance values 112 (and/or the weights 106), the schematic model 108, and/or the lithographic masks for connections 110. In some implementations, the system reuses the lithographic masks 120 for the analog neurons 120. In other words, in some implementations, only the weights 106 (or the resistance values 112 corresponding to the changed weights), and/or the lithographic masks for the connections 110 are regenerated. Since only the connections, weights, the schematic model, and/or the corresponding lithographic masks for the connections are regenerated, as indicated by the dashed line 156, the process for (or the path to) fabricating analog integrated circuits for the retrained neural networks is substantially simplified, and the time to market for re-spinning hardware for neural networks is reduced, when compared to conventional techniques for hardware realization of neural networks. [00226] Figure 1B is a block diagram of an alternative representation of the system 100 for hardware realization of trained neural networks using analog components, according to some implementations. The system includes training (156) neural networks in software, determining weights of connections, generating (158) electronic circuit equivalent to the neural network, calculating (160) resistor values corresponding to weights of each connection, and subsequently generating (162) lithography mask with resistor values. [00227] Figure 1C is a block diagram of another representation of the system 100 for hardware realization of trained neural networks using analog components, according to some implementations. The system is distributed as a software development kit (SDK) 180, according to some implementations. A user develops and trains (164) a neural network and inputs the trained neural net 166 to the SDK 180. The SDK estimates (168) complexity of the trained neural net 166. If the complexity of the trained neural net can be reduced (e.g., some connections and/or neurons can be removed, some layers can be reduced, or the density of the neurons can be changed), the SDK 180 prunes (178) the trained neural net and retrains (182) the neural net to obtain an updated trained neural net 166. Once the complexity of the trained neural net is reduced, the SDK 180 transforms (170) the trained neural net 166 into a sparse network of analog components (e.g., a pyramid- or a trapezia-shaped network). The SDK 180 also generates a circuit model 172 of the analog network. In some implementations, the SDK estimates (176) a deviation in an output generated by the circuit model 172 relative to the trained neural network for a same input, using software simulations. If the estimated error exceeds a threshold error (e.g., a value set by the user), the SDK 180 prompts the user to reconfigure, redevelop, and/or retrain the neural network. In some implementations, although not shown, the SDK automatically reconfigures the trained neural net 166 so as to reduce the estimated error. This process is iterated multiple times until the error is reduced below the threshold error. In Figure 1C, the dashed line from the block 176 (“Estimation of error raised in circuitry”) to the block 164 (“Development and training of neural network”) indicates a feedback loop. For example, if the pruned network did not show desired accuracy, some implementations prune the network differently, until accuracy exceeds a predetermined threshold (e.g., 98% accuracy) for a given application. In some implementations, this process includes recalculating the weights, since pruning includes retraining of the whole network. [00228] In some implementations, components of the system 100 described above are implemented in one or more computing devices or server systems as computing modules. Figure 2A is a system diagram of a computing device 200 in accordance with some implementations. As used herein, the term “computing device” includes both personal devices 102 and servers. A computing device 200 typically includes one or more processing units/cores (CPUs) 202 for executing modules, programs, and/or instructions stored in the memory 214 and thereby performing processing operations; one or more network or other communications interfaces 204; memory 214; and one or more communication buses 212 for interconnecting these components. The communication buses 212 may include circuitry that interconnects and controls communications between system components. A computing device 200 may include a user interface 206 comprising a display device 208 and one or more input devices or mechanisms 210. In some implementations, the input device/mechanism 210 includes a keyboard; in some implementations, the input device/mechanism includes a “soft” keyboard, which is displayed as needed on the display device 208, enabling a user to “press keys” that appear on the display 208. In some implementations, the display 208 and input device / mechanism 210 comprise a touch screen display (also called a touch sensitive display). In some implementations, the memory 214 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices. In some implementations, the memory 214 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some implementations, the memory 214 includes one or more storage devices remotely located from the CPU(s) 202. The memory 214, or alternatively the non-volatile memory device(s) within the memory 214, comprises a computer readable storage medium. In some implementations, the memory 214, or the computer readable storage medium of the memory 214, stores the following programs, modules, and data structures, or a subset thereof: • an operating system 216, which includes procedures for handling various basic system services and for performing hardware dependent tasks; • a communications module 218, which is used for connecting the computing device 200 to other computers and devices via the one or more communication network interfaces 204 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on; • trained neural networks 220 that includes weights 222 and neural network topologies 224. Examples of input neural networks are described below in reference to Figures 4A-4C, Figure 12, Figure 13A, and 14A, according to some implementations; • a neural network transformation module 226 that includes transformed analog neural networks 228, mathematical formulations 230, the basic function blocks 232, analog models 234 (sometimes called neuron models), and/or analog integrated circuit (IC) design constraints 236. Example operations of the neural network transformation module 226 are described below in reference to at least Figures 5, 6A-6C, 7, 8, 9, 10, and 11A-11C, and the flowcharts shown in Figures 27A-27J, and Figures 28A-28S; and/or • a weight matrix computation (sometimes called a weight quantization) module 238 that includes weights 272 of transformed networks, and optionally includes resistance calculation module 240, resistance values 242. Example operations of the weight matrix computation module 238 and/or weight quantization are described in reference to at least Figures 17A-17C, Figure 18, and Figures 29A- 29F, according to some implementations. [00229] Some implementations include one or more optional modules 244 as shown in Figure 2B. Some implementations include an analog neural network optimization module 246. Examples of analog neural network optimization are described below in reference to Figures 30A-30M, according to some implementations. [00230] Some implementations include a lithographic mask generation module 248 that further includes lithographic masks 250 for resistances (corresponding to connections), and/or lithographic masks for analog components (e.g., operational amplifiers, multipliers, delay blocks, etc.) other than the resistances (or connections). In some implementations, lithographic masks are generated based on chip design layout following chip design using Cadence, Synopsys, or Mentor Graphics software packages. Some implementations use a design kit from a silicon wafer manufacturing plant (sometimes called a fab). Lithographic masks are intended to be used in that particular fab that provides the design kit (e.g., TSMC 65 nm design kit). The lithographic mask files that are generated are used to fabricate the chip at the fab. In some implementations, the Cadence, Mentor Graphics, or Synopsys software packages-based chip design is generated semi-automatically from the SPICE or Fast SPICE (Mentor Graphics) software packages. In some implementations, a user with chip design skill drives the conversion from the SPICE or Fast SPICE circuit into Cadence, Mentor Graphics or Synopsis chip design. Some implementations combine Cadence design blocks for single neuron unit, establishing proper interconnects between the blocks. [00231] Some implementations include a library generation module 254 that further includes libraries of lithographic masks 256. Examples of library generation are described below in reference to Figures 32A-32E, according to some implementations. [00232] Some implementations include Integrated Circuit (IC) fabrication module 258 that further includes Analog-to-Digital Conversion (ADC), Digital-to-Analog Conversion (DAC), or similar other interfaces 260, and/or fabricated ICs or models 262. Example integrated circuits and/or related modules are described below in reference to Figures 31A- 31Q, according to some implementations. [00233] Some implementations include an energy efficiency optimization module 264 that further includes an inferencing module 266, a signal monitoring module 268, and/or a power optimization module 270. Examples of energy efficiency optimizations are described below in reference to Figures 33A-33K, according to some implementations. [00234] Each of the above identified executable modules, applications, or sets of procedures may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various implementations. In some implementations, the memory 214 stores a subset of the modules and data structures identified above. Furthermore, in some implementations, the memory 214 stores additional modules or data structures not described above. [00235] Although Figure 2A shows a computing device 200, Figure 2A is intended more as a functional description of the various features that may be present rather than as a structural schematic of the implementations described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. Example Process for Generating Schematic Models of Analog Networks [00236] Figure 3A shows an example process 300 for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations. As shown in Figure 3A, a trained neural network 302 (e.g., MobileNet) is converted (322) to a target or equivalent analog network 304 (using a process that is sometimes called T-transformation). The target neural network (sometimes called a T-network) 304 is exported (324) to SPICE (as a SPICE model 306) using a single neuron model (SNM), which is exported (326) from SPICE to CADENCE and full on-chip designs using a CADENCE model 308. The CADENCE model 308 is cross-validated (328) against the initial neural network for one or more validation inputs. [00237] In the description above and below, a math neuron is a mathematical function which receives one or more weighted inputs and produces a scalar output. In some implementations, a math neuron can have memory (e.g., long short-term memory (LSTM), recurrent neuron). A trivial neuron is a math neuron that performs a function, representing an ‘ideal’ mathematical neuron, ^^^௨௧ = ^^(∑( ^^^ ^^ .ω^ + bias), where f(x) is an activation function. A SNM is a schematic model with analog components (e.g., operational amplifiers, resistors R1, …, Rn, and other components) representing a specific type of math neuron (for example, trivial neuron) in schematic form. SNM output voltage is represented by a corresponding formula that depends on K input voltages and SNM component values ^^^௨௧ =
Figure imgf000045_0001
… ^^^). According to some implementations, with properly selected component values, SNM formula is equivalent to math neuron formula, with a desired weights set. In some implementations, the weights set is fully determined by resistors used in a SNM. A target (analog) neural network 304 (sometimes called a T-network) is a set of math neurons which have defined SNM representation, and weighted connections between them, forming a neural network. A T-network follows several restrictions, such as an inbound limit (a maximum limit of inbound connections for any neuron within the T-network), an outbound limit (a maximum limit of outbound connections for any neuron within the T-network), and a signal range (e.g., all signals should be inside pre-defined signal range). T-transformation (322) is a process of converting some desired neural network, such as MobileNet, to a corresponding T-network. A SPICE model 306 is a SPICE Neural Network model of a T- network 304, where each math neuron is substituted with corresponding one or more SNMs. A Cadence NN model 310 is a Cadence model of the T-network 304, where each math neuron is substituted with a corresponding one or more SNMs. Also, as described herein, two networks L and M have mathematical equivalence, if for all neuron outputs of these networks ห ^^^ ^ − ^^^ ெห < ^^ ^^ ^^, where ^^ ^^ ^^ is relatively small (e.g., between 0.1-1% of operating voltage range). Also, two networks L and M have functional equivalence, if for a given validation input data set { ^^^, … , ^^^}, the classification results are mostly the same, i.e., ^^( ^^( ^^^) = ^^( ^^^)) = 1 − ^^ ^^ ^^, where ^^ ^^ ^^ is relatively small. [00238] Figure 3B shows an example manual prototyping process used for generating a target chip model 320 based on a SNM model on Cadence 314, according to some implementations. Note that although the following description uses Cadence, alternate tools from Mentor Graphic design or Synopsys (e.g., Synopsys design kit) may be used in place of Cadence tools, according to some implementations. The process includes selecting SNM limitations, including inbound and outbound limits and signal limitation, selecting analog components (e.g., resistors, including specific resistor array technology) for connections between neurons, and developing a Cadence SNM model 314. A prototype SNM model 316 (e.g., a PCB prototype) is developed (330) based on the SNM model on Cadence 314. The prototype SNM model 316 is compared with a SPICE model for equivalence. In some implementations, a neural network is selected for an on-chip prototype, when the neural network satisfies equivalence requirements. Because the neural network is small in size, the T-transformation can be hand-verified for equivalence. Subsequently, an on-chip SNM model 318 is generated (332) based on the SNM model prototype 316. The on-chip SNM model is optimized as possible, according to some implementations. In some implementations, an on- chip density for the SNM model is calculated prior to generating (334) a target chip model 320 based on the on-chip SNM model 318, after finalizing the SNM. During the prototyping process, a practitioner may iterate selecting neural network task or application and specific neural network (e.g., a neural network having in the order of 0.1 to 1.1 million neurons), performing T-transformation, building a Cadence neural network model, designing interfaces and/or the target chip model. Example Input Neural Networks [00239] Figures 4A, 4B, and 4C show examples of trained neural networks (e.g., the neural networks 220) that are input to the system 100 and transformed to mathematically equivalent analog networks, according to some implementations. Figure 4A shows an example neural network (sometimes called an artificial neural network) that are composed of artificial neurons that receive input, combine the input using an activation function, and produce one or more outputs. The input includes data, such as images, sensor data, and documents. Typically, each neural network performs a specific task, such as object recognition. The networks include connections between the neurons, each connection providing the output of a neuron as an input to another neuron. After training, each connection is assigned a corresponding weight. As shown in Figure 4A, the neurons are typically organized into multiple layers, with each layer of neurons connected only to the immediately preceding and following layer of neurons. An input layer of neurons 402 receives external input (e.g., the input X1, X2, …, Xn). The input layer 402 is followed by one or more hidden layers of neurons (e.g., the layers 404 and 406), that is followed by an output layer 408 that produces outputs 410. Various types of connection patterns connect neurons of consecutive layers, such as a fully-connected pattern that connects every neuron in one layer to all the neurons of the next layer, or a pooling pattern that connect output of a group of neurons in one layer to a single neuron in the next layer. In contrast to the neural network shown in Figure 4A that are sometimes called feedforward networks, the neural network shown in Figure 4B includes one or more connections from neurons in one layer to either other neurons in the same layer or neurons in a preceding layer. The example shown in Figure 4B is an example of a recurrent neural network, and includes two input neurons 412 (that accepts an input X1) and 414 (that accepts an input X2) in an input layer followed by two hidden layers. The first hidden layer includes neurons 416 and 418 that is fully connected with neurons in the input layer, and the neurons 420, 422, and 424 in the second hidden layer. The output of the neuron 420 in the second hidden layer is connected to the neuron 416 in the first hidden layer, providing a feedback loop. The hidden layer including the neurons 420, 422, and 424 are input to a neuron 426 in the output layer that produces an output y. [00240] Figure 4C shows an example of a convolutional neural network (CNN), according to some implementations. In contrast to the neural networks shown in Figure 4A and 4B, the example shown in Figure 4C includes different types of neural network layers, that includes a first stage of layers for feature learning, and a second stage of layers for classification tasks, such as object recognition. The feature learning stage includes a convolution and Rectified Linear Unit (ReLU) layer 430, followed by a pooling layer 432, that is followed by another convolution and ReLU layer 434, which is in turn followed by another pooling layer 436. The first layer 430 extracts features from an input 428 (e.g., an input image or portions thereof), and performs a convolution operation on its input, and one or more non-linear operations (e.g., ReLU, tanh, or sigmoid). A pooling layer, such as the layer 432, reduces the number of parameters when the inputs are large. The output of the pooling layer 436 is flattened by the layer 438 and input to a fully connected neural network with one or more layers (e.g., the layers 440 and 442). The output of the fully-connected neural network is input to a softmax layer 444 to classify the output of the layer 442 of the fully-connected network to produce one of many different output 446 (e.g., object class or type of the input image 428). [00241] Some implementations store the layout or the organization of the input neural networks including number of neurons in each layer, total number of neurons, operations or activation functions of each neuron, and/or connections between the neurons, in the memory 214, as the neural network topology 224. [00242] Figure 5 shows an example of a math model 500 for a neuron, according to some implementations. The math model includes incoming signals 502 input multiplied by synaptic weights 504 and summed by a unit summation 506. The result of the unit summation 506 is input to a nonlinear conversion unit 508 to produce an output signal 510, according to some implementations. [00243] Figures 6A-6C illustrate an example process for analog hardware realization of a neural network for computing an XOR (classification of XOR results) of input values, according to some implementations. Figure 6A shows a table 600 of possible input values X1 and X2 along x- and y-axis, respectively. The expected result values are indicated by hollow circle (represents a value of 1) and a filled or dark circle (represents a value of 0) – this is a typical XOR problem with 2 input signals and 2 classes. Only if either, not both, of the values X1 and X2 are 1, the expected result is 1, and 0, otherwise. Training set consists of 4 possible input signal combinations (binary values for the X1 and X2 inputs). Figure 6B shows a ReLU- based neural network 602 to solve the XOR classification of Figure 6A, according to some implementations. The neurons do not use any bias values, and use ReLU activation. Inputs 604 and 606 (that correspond to X1 and X2, respectively) are input to a first ReLU neuron 608- 2. The inputs 604 and 606 are also input to a second ReLU neuron 608-4. The results of the two ReLU neurons 608-2 and 608-4 are input to a third neuron 608-6 that performs linear summation of the input values, to produce an output value 510 (the Out value). The neural network 602 has the weights -1 and 1 (for the input values X1 and X2, respectively) for the ReLU neuron 608-2, the weights 1 and -1 (for the input values X1 and X2, respectively) for the ReLU neuron 608-4, and the weights 1 and 1 (for the output of the ReLU neurons 608-2 and 608-4, respectively). In some implementations, the weights of trained neural networks are stored in memory 214, as the weights 222. [00244] Figure 6C shows an example equivalent analog network for the network 602, according to some implementations. The analog equivalent inputs 614 and 616 of the X1 and X2 inputs 604 and 606 are input to analog neurons N1618 and N2620 of a first layer. The neurons N1 and N2 are densely connected with neurons N3 and N4 of a second layer. The neurons of a second layer (i.e. neuron N3622 and neuron N4624) are connected with an output neuron N5626 that produces the output Out (equivalent to the output 610 of the network 602). The neurons N1, N2, N3, N4 and N5 have ReLU (maximum value = 1) activation function. [00245] Some implementations use Keras learning that converges in approximately 1000 iterations, and results in weights for the connections. In some implementations, the weights are stored in memory 214, as part of the weights 222. In the following example, data format is ‘Neuron [1st link weight, 2nd link weight, bias]’. • N1 [-0.9824321, 0.976517, -0.00204677]; • N2 [1.0066702, -1.0101418, -0.00045485]; • N3 [1.0357606, 1.0072469, -0.00483723]; • N4 [-0.07376373, -0.7682612, 0.0]; and • N5 [1.0029935, -1.1994369, -0.00147767]. [00246] Next, to compute resistor values for connections between the neurons, some implementations compute resistor range. Some implementations set resistor nominal values (R+, R-) of 1 MΩ, possible resistor range of 100 KΩ to 1 MΩ and nominal series E24. Some implementations compute w1, w2, wbias resistor values for each connection as follows. For each weight value wi (e.g., the weights 222), some implementations evaluate all possible (Ri-, Ri+) resistor pairs options within the chosen nominal series and choose a resistor pair which
Figure imgf000049_0001
൬ ^^^ − ^ ோశ + ^ ோష ^ ^ ^. The following table provides example values for the weights w1, w2, and bias, for each connection, according to some implementations.
Figure imgf000049_0002
Example Advantages of Transformed Neural Networks [00247] Before describing examples of transformation, it is worth noting some of the advantages of the transformed neural networks over conventional architectures. As described herein, the input trained neural networks are transformed to pyramid- or trapezium-shaped analog networks. Some of the advantages of pyramid or trapezium over cross bars include lower latency, simultaneous analog signal propagation, possibility for manufacture using standard integrated circuit (IC) design elements, including resistors and operational amplifiers, high parallelism of computation, high accuracy (e.g., accuracy increases with the number of layers, relative to conventional methods), tolerance towards error(s) in each weight and/or at each connection (e.g., pyramids balance the errors), low RC (low Resistance Capacitance delay related to propagation of signal through network), and/or ability to manipulate biases and functions of each neuron in each layer of the transformed network. Also, pyramids are excellent computation block by itself, since it is a multi-level perceptron, which can model any neural network with one output. Networks with several outputs are implemented using different pyramids or trapezia geometry, according to some implementations. A pyramid can be thought of as a multi-layer perceptron with one output and several layers (e.g., N layers), where each neuron has n inputs and 1 output. Similarly, a trapezium is a multilayer perceptron, where each neuron has n inputs and m outputs. Each trapezium is a pyramid-like network, where each neuron has n inputs and m outputs, where n and m are limited by IC analog chip design limitations, according to some implementations. [00248] Some implementations perform lossless transformation of any trained neural network into subsystems of pyramids or trapezia. Thus, pyramids and trapezia can be used as universal building blocks for transforming any neural networks. An advantage of pyramid- or trapezia-based neural networks is the possibility to realize any neural network using standard IC analog elements (e.g., operational amplifiers, resistors, signal delay lines in case of recurrent neurons) using standard lithography techniques. It is also possible to restrict the weights of transformed networks to some interval. In other words, lossless transformation is performed with weights limited to some predefined range, according to some implementations. Another advantage of using pyramids or trapezia is the high degree of parallelism in signal processing or the simultaneous propagation of analog signals that increases the speed of calculations, providing lower latency. Moreover, many modern neural networks are sparsely connected networks and are much better (e.g., more compact, have low RC values, absence of leakage currents) when transformed into pyramids than into cross-bars, Pyramids and trapezia networks are relatively more compact than cross-bar based memristor networks. [00249] Furthermore, analog neuromorphic trapezia-like chips possess a number of properties, not typical for analog devices. For example, signal to noise ratio is not increasing with the number of cascades in analog chip, the external noise is suppressed, and influence of temperature is greatly reduced. Such properties make trapezia-like analog neuromorphic chips analogous to digital circuits. For example, individual neurons, based on operational amplifier, level the signal and are operated with the frequencies of 20,000-100,000 Hz, and are not influenced by noise or signals with frequency higher than the operational range, according to some implementations. Trapezia-like analog neuromorphic chip also perform filtration of output signal due to peculiarities in how operational amplifiers function. Such trapezia-like analog neuromorphic chip suppresses the synphase noise. Due to low-ohmic outputs of operational amplifiers, the noise is also significantly reduced. Due to the leveling of signal at each operational amplifier output and synchronous work of amplifiers, the drift of parameters, caused by temperature does not influence the signals at final outputs. Trapezia-like analogous neuromorphic circuit is tolerant towards the errors and noise in input signals and is tolerant towards deviation of resistor values, corresponding to weight values in neural network. Trapezia-like analog neuromorphic networks are also tolerant towards any kind of systemic error, like error in resistor value settings, if such error is same for all resistors, due to the very nature of analog neuromorphic trapezia-like circuits, based on operational amplifiers. Example Lossless Transformation (T-Transformation) of Trained Neural Networks [00250] In some implementations, the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228. [00251] Figure 7 shows an example perceptron 700, according to some implementations. The perceptron includes K = 8 inputs and 8 neurons 702-2, …, 702-16 in an input layer that receives the 8 inputs. There is an output layer with 4 neurons 704-2, …, 704- 8, in an output layer, that correspond to L = 4 outputs. The neurons in the input layer are fully connected to the neurons in the output layer, making 8 times 4 = 32 connections. Suppose the weights of the connections are represented by a weight matrix WP (element WPi, j corresponds to the weight of the connection between the i-th neuron in the input layer and the j-th neuron in the output layer). Suppose further each neuron performs an activation function F. [00252] Figure 8 shows an example Pyramid-Neural Network (P-NN) 800, a type of Target-Neural Network (T-NN, or TNN), that is equivalent to the perceptron shown in Figure 7, according to some implementations. To perform this transformation of the perceptron (Figure 7) to the PN-NN architecture (Figure 8), suppose, for the T-NN, that number of inputs is restricted to Ni = 4 and number of outputs is restricted to No = 2. The T-NN includes an input layer LTI of neurons 802-2, …, 802-34, that is a concatenation of two copies of the input layer of neurons 802-2, …, 802-16, for a total of 2 times 8 = 16 input neurons. The set of neurons 804, including neurons 802-20, …, 802-34, is a copy of the neurons 802-2, …, 802- 18, and the input is replicated. For example, the input to the neuron 802-2 is also input to the neuron 802-20, the input 20 the neuron 802-4 is also input to the neuron 802-22, and so on. Figure 8 also includes a hidden layer LTH1 of neurons 806-02, …, 806-16 (2 times 16 divided by 4 = 8 neurons) that are linear neurons. Each group of Ni neurons from the input layer LTI are fully connected to two neurons from the LTH1 layer. Figure 8 also includes an output layer LTO with 2 times 8 divided by 4 = 4 neurons 808-02, …, 808-08, each neuron performing the activation function F. Each neuron in the layer LTO is connected to distinct neurons from different groups in the layer LTH1. The network shown in Figure 8 includes 40 connections. Some implementations perform weight matrix calculation for the P-NN in Figure 8, as follows. Weights for the hidden layer LTH1 (WTH1) are calculated from the weight matrix WP, and weights corresponding to the output layer LTO (WTO) form a sparse matrix with elements equal to 1. [00253] Figure 9 shows a Pyramid Single Neural Network (PSNN) 900 corresponding to an output neuron of Figure 8, according to some implementations. The PSNN includes a layer (LPSI) of input neurons 902-02, …, 902-16 (corresponding to the 8 input neurons in the network 700 of Figure 7). A hidden layer LPSH1 includes 8 divided by 4 = 2 linear neurons 904-02 and 904-04, and each group of Ni neurons from LTI is connected to one neuron of the LPSH1 layer. An output layer LPSO consists of 1 neuron 906 with an activation function F, that is connected to both the neurons 904-02 and 904-04 of the hidden layer. For calculating weight matrix for the PSNN 900, some implementations compute a vector WPSH1 that is equal to the first row of WP, for the LPSH1 layer. For the LPSOlayer, some implementations compute a weight vector WPSO with 2 elements, each element equal to 1. The process is repeated for the first, second, third, and fourth output neurons. A P-NN, such as the network shown in Figure 8, is a union of the PSNNs (for the 4 output neurons). Input layer for every PSNN is a separate copy of P’s input layer. For this example, the P-NN 800 includes an input layer with 8 times 4 = 32 inputs, a hidden layer with 2 times 4 = 8 neurons, and an output layer with 4 neurons. Example Transformations with Target Neurons with N Inputs and 1 Output [00254] In some implementations, the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or analog design constraints 236, to obtain the transformed neural networks 228. Single Layer Perceptron with One Output [00255] Suppose a single layer perceptron SLP(K,1) includes K inputs and one output neuron with activation function F. Suppose further ^^ ∈ ^^^ is a vector of weights for SLP(K,1). The following algorithm Neuron2TNN1 constructs a T-neural network from T-neurons with N inputs and 1 output (referred to as TN(N,1)). Algorithm Neuron2TNN1 1. Construct an input layer for T-NN by including all inputs from SLP(K,1). 2. If K>N then: a. Divide K input neurons into ^^^ =
Figure imgf000053_0001
groups such that every group consists of no more than N inputs. b. Construct the first hidden layer LTH1 of the T-NN from ^^^neurons, each neuron performing an identity activation function. c. Connect input neurons from every group to corresponding neuron from the next layer. So every neuron from the LTH1 has no more than N input connections. d. Set the weights for the new connections according to the following equation:
Figure imgf000053_0002
^^ = 1, … , ^ ^^ ^^^ 3. Else (i.e., if K <= N) then): a. Construct the output layer with 1 neuron calculating activation function F b. Connect input neurons to the single output neuron. It has K≤N connections. c. Set the weights of the new connections by means of the following equation: ^^^ ^ = ^^^ , ^^ = 1, … , ^^ d. Terminate the algorithm 4. Set l=1 5. If ml > N: a. Divide ml neurons into ^^^ା^ =
Figure imgf000054_0001
groups, every group consists of no more than N neurons. b. Construct the hidden layer LTHl+1 of the T-NN from m୪ା^ neurons, every neuron has identity activation function. c. Connect input neurons from every group to the corresponded neuron from the next layer. d. Set the weights of the new connections according to the following equation:
Figure imgf000054_0002
e. Set l = l + 1 6. Else (if m >= N): a. Construct the output layer with 1 neuron calculating activation function F b. Connect all LTHl’s neurons to the single output neuron. c. Set the weights of the new connections according to the following equation: a. w୨ ୪ା^ = 1 d. Terminate the algorithm 7. Repeat steps 5 and 6. [00256] Here ⌈ ^^⌉- minimum integer number being no less than x. Number of layers in T-NN constructed by means of the algorithm Neuron2TNN1 is ℎ = ⌈ ^^ ^^ ^^ே ^^⌉. The total number of weights in T-NN is: ^^^ − 1 ^^ = ^^ ^^^ି^( ^^ − 1) [00257] Figure 10 shows an example of the constructed T-NN, according to some implementations. All layers except the first one perform identity transformation of their inputs. Weight matrices of the constructed T-NN have the following forms, according to some implementations. • Layer 1 (e.g., layer 1002): u1 u 2 … u N 0 0 … 0 … … 0 W1 = 0 0 … 0 uN+1 uN+2 … u2N 0 … 0 … … … … … … … … … … … u(h- 0 0 … … … … 0 1)N+1 … uK • Layers i = 2, 3, …, h (e.g., layers 1004, 1006,1008, and 1010): 1 1 … 1 0 0 … 0 … … 0 Wi = 0 0 … 0 1 1 … 1 0 … 0 … … … … … … … … … … … 0 0 … … … … 0 1 … 1 [00258] Output value of the T-NN is calculated according to the following formula:
Figure imgf000055_0001
[00259] Output for the first layer is calculated as an output vector according to the following formula:
Figure imgf000055_0002
[00260] Multiplying the obtained vector by the weight matrix of the second layer:
Figure imgf000055_0003
[00261] Every subsequent layer outputs a vector with components equal to linear combination of some sub-vector of x. [00262] Finally, the T-NN’s output is equal to:
Figure imgf000056_0001
[00263] This is the same value as the one calculated in SLP(K,1) for the same input vector x. So output values of SLP(K,1) and constructed T-NN are equal. Single Layer Perceptron with Several Outputs [00264] Suppose there is a single layer perceptron SLP(K, L) with K inputs and L output neurons, each neuron performing an activation function F. Suppose further ^^ ∈ ^^^×^ is a weight matrix for SLP(K, L). The following algorithm Layer2TNN1 constructs a T-neural network from neurons TN(N, 1). Algorithm Layer2TNN1 1. For every output neuron i=1,…,L a. Apply the algorithm Neuron2TNN1 to SLPi(K, 1) consisting of K inputs, 1 output neuron and weight vector Uij, j=1,2,…,K. A TNNi is constructed as a result. 2. Construct PTNN by composing all TNNi into one neural net: a. Concatenate input vectors of all TNNi, so the input of PTNN has L groups of K inputs, with each group being a copy of the SLP(K, L)’s input layer. [00265] Output of the PTNN is equal to the SLP(K, L)’s output for the same input vector because output of every pair SLPi(K, 1) and TNNi are equal. Multilayer Perceptron [00266] Suppose a multilayer perceptron (MLP) includes K inputs, S layers and Li calculation neurons in i-th layer, represented as MLP(K, S, L1,… LS). Suppose ^^^ ∈ ^^^^×^^షభ is a weight matrix for the i-th layer. [00267] The following is an example algorithm to construct a T-neural network from neurons TN(N, 1)., according to some implementations. Algorithm MLP2TNN1 1. For every layer i=1,…,S a. Apply the algorithm Layer2TNN1 to SLPi(Li-1, Li) consisting of Li-1 inputs, Li output neurons, and a weight matrix Ui, constructing PTNNi as a result. 2. Construct MTNN by stacking all PTNNi into one neural net; output of a TNNi-1 is set as input for TNNi. [00268] Output of the MTNN is equal to the MLP(K, S, L1,… LS)’s output for the same input vector because output of every pair SLPi(Li-1, Li) and PTNNi are equal. Example T-transformations with Target neurons with NI Inputs and NO Outputs [00269] In some implementations, the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228. Example Transformation of Single Layer Perceptron with Several Outputs [00270] Suppose a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each neuron performing an activation function F. Suppose further ^^ ∈ ^^^×^ is a weight matrix for SLP(K,L). The following algorithm constructs a T-neural network from neurons TN(NI, NO), according to some implementations. Algorithm Layer2TNNX 1. Construct a PTNN from SLP(K,L) by using the algorithm Layer2TNN1 (see description above). PTNN has an input layer consisting of L groups of K inputs. 2. Compose ^ ^ ^^^ subsets from L groups. Each subset contains no more than NO groups of input vector copies. 3. Replace groups in every subset with one copy of input vector. 4. Construct PTNNX by rebuild connections in every subset by making NO output connections from every input neuron. [00271] According to some implementations, output of the PTNNX is calculated by means of the same formulas as for PTNN (described above), so the outputs are equal. [00272] Figures 11A-11C show an application 1100 of the above algorithm for a single layer neural network (NN) with 2 output neurons and TN(NI, 2), according to some implementations. Figure 11A shows an example source or input NN, according to some implementations. K inputs are input to two neurons 1 and 2 belonging to a layer 1104. Figure 11B shows a PTNN constructed after the first step of the algorithm, according to some implementations. The PTNN consists of two parts implementing subnets corresponding to the output neuron 1 and neuron 2 of the NN shown in Figure 11A. In Figure 11B, the input 1102 is replicated and input to two sets of input neurons 1106-2 and 1106-4. Each set of input neurons is connected to a subsequent layer of neurons with two sets of neurons 1108-2 and 1108-4, each set of neurons including m1 neurons. The input layer is followed by identity transform blocks 1110-2 and 1110-4, each block containing one or more layers with identity weight matrix. The output of the identity transform block 1110-2 is connected to the output neuron 1112 (corresponding to the output neuron 1 in Figure 11A), and the output of the identity transform block 1110-4 is connected to the output neuron 1114 (corresponding to the output neuron 1 in Figure 11A). Figure 11C shows application of the final steps of the algorithm, including replacing two copies of the input vector (1106-2 and 1106-4) with one vector 1116 (step 3), and rebuilding connections in the first layer 1118 by making two output links from every input neuron: one link connects to subnet related to output 1 and another link connects to subnet for the output 2. Example Transformation of Multilayer Perceptron [00273] Suppose a multilayer perceptron (MLP) includes K inputs, S layers and Li calculation neurons in ith layer, represented as MLP(K, S, L1,… LS). Suppose ^^^ ∈ ^^^^×^^షభ is a weight matrix for i-th layer. The following example algorithm constructs a T-neural network from neurons TN(NI, NO), according to some implementations. Algorithm MLP2TNNX 1. For every layer i=1,…,S: a. Apply the algorithm Layer2TNNX to SLPi(Li-1, Li) consisting of Li-1 inputs, Li output neuron and weight matrix Ui. PTNNXi is constructed as a result. 2. Construct MTNNX by stacking all PTNNXi into one neural net: a. Output of a TNNXi-1 is set as input for TNNXi. [00274] According to some implementations. output of the MTNNX is equal to the MLP(K, S, L1,… LS)’s output for the same input vector, because output of every pair SLPi(Li- 1, Li) and PTNNXi are equal. Example Transformation of Recurrent Neural Network [00275] A Recurrent Neural Network (RNN) contains backward connection allowing saving information. Figure 12 shows an example RNN 1200, according to some implementations. The example shows a block 1204 performing an activation function A, that accepts an input Xt 1206 and performs an activation function A, and outputs a value ht 1202. The backward arrow from the block 1204 to itself indicates a backward connection, according to some implementations. An equivalent network is shown on the right up to the point in time when the activation block receives the input Xt 1206. At time 0, the network accepts input Xt 1208 and performs the activation function A 1204, and outputs a value ho 1210; at time 1, the network accepts input X1 1212 and the output of the network at time 0, and performs the activation function A 1204, and outputs a value h11214; at time 2, the network accepts input X21216 and the output of the network at time 1, and performs the activation function A 1204, and outputs a value h11218. This process continues until time t, at which time the network accepts the input Xt 1206 and the output of the network at time t-1, and performs the activation function A 1204, and outputs the value ht 1202, according to some implementations. [00276] Data processing in an RNN is performed by means of the following formula:
Figure imgf000059_0001
[00277] In the equation above, xt is a current input vector, and ht-1 is the RNN’s output for the previous input vector xt-1. This expression consists of the several operations: calculation of linear combination for two fully connected layers (^௫)
Figure imgf000059_0002
and ^^ ^^௧, element-wise addition, and non-linear function calculation (f). The first and third operations can be implemented by trapezium-based network (one fully connected layer is implemented by pyramid-based network, a special case of trapezium networks). The second operation is a common operation that can be implemented in networks of any structure. [00278] In some implementations, the RNN’s layer without recurrent connections is transformed by means of Layer2TNNX algorithm described above. After transformation is completed, recurrent links are added between related neurons. Some implementations use delay blocks described below in reference to Figure 13B. Example Transformation of LSTM Network [00279] A Long Short-Term Memory (LSTM) neural network is a special case of a RNN. A LSTM network’s operations are represented by the following equations: • ^^௧ = ^^( ^^^ [ ℎ௧ି^, ^^௧ ] + ^^^); • ^^௧ = ^^( ^^^ [ ℎ௧ି^, ^^௧ ] + ^^^); • ^^௧ = ^^ ^^ ^^ℎ( ^^^[ℎ௧ି^, ^^௧] + ^^^); • ^^௧ = ( ^^௧ × ^^௧ି^ + ^^௧ × ^^௧);
Figure imgf000060_0001
• ℎ௧ = ^^௧ × tanh ( ^^௧). [00280] In the equations above, ^^^,
Figure imgf000060_0002
^^^, and ^^ை are trainable weight matrices, ^^^, ^^^, ^^^, and ^^ை are trainable biases, xt is a current input vector, ht-1 is an internal state of the LSTM calculated for the previous input vector xt-1, and ot is output for the current input vector. In the equations, the subscript t denotes a time instance t, and the subscript t -1 denotes a time instance t - 1. [00281] Figure 13A is a block diagram of a LSTM neuron 1300, according to some implementations. A sigmoid (σ) block 1318 processes the inputs ℎ௧ି^ 1330 and ^^௧1332, and produces the output ^^௧1336. A second sigmoid (σ) block 1320 processes the inputs ℎ௧ି^ 1330 and ^^௧ 1332, and produces the output ^^௧ 1338. A hyperbolic tangent (tanh) block 1322 processes the inputs ℎ௧ି^ 1330 and ^^௧1332, and produces the output ^^௧1340. A third sigmoid (σ) block 1328 processes the inputs ℎ௧ି^ 1330 and ^^௧1332, and produces the output ^^௧1342. A multiplier block 1304 processes ^^௧1336 and the output of a summing block 1306 (from a prior time instance) ^^௧ି^ 1302 to produce an output that is in turn summed by the summing block 1306 along with the output of a second multiplier block 1314 that multiplies the outputs ^^௧1338 and ^^௧1340 to produce the output ^^௧1310. The output ^^௧1310 is input to another tanh block 1312 that produces an output that is multiplied a third multiplier block 1316 with the output ^^௧1342 to produce the output ℎ௧1334. [00282] There are several types of operations utilized in these expressions: (i) calculation of linear combination for several fully connected layers, (ii) elementwise addition, (iii) Hadamard product, and (iv) non-linear function calculation (e.g., sigmoid (σ) and hyperbolic tangent (tanh)). Some implementations implement the (i) and (iv) operations by a trapezium-based network (one fully connected layer is implemented by a pyramid-based network, a special case of trapezium networks). Some implementations use networks of various structures for the (ii) and (iii) operations which are common operations. [00283] The layer in an LSTM layer without recurrent connections is transformed by using the Layer2TNNX algorithm described above, according to some implementations. After transformation is completed, recurrent links are added between related neurons, according to some implementations. [00284] Figure 13B shows delay blocks, according to some implementations. As described above, some of the expressions in the equations for the LSTM operations depend on saving, restoring, and/or recalling an output from a previous time instance. For example, the multiplier block 1304 processes the output of the summing block 1306 (from a prior time instance) ^^௧ି^ 1302. Figure 13B shows two examples of delay blocks, according to some implementations. The example 1350 includes a delay block 1354 on the left accepts input xt 1352 at time t, and outputs the input after a delay of dt indicated by the output xt-dt 1356. The example 1360 on the right shows cascaded (or multiple) delay blocks 1364 and 1366 outputs the input xt 1362 after 2 units of time delays, indicated by the output xt-2dt 1368, according to some implementations. [00285] Figure 13C is a neuron schema for a LSTM neuron, according to some implementations. The schema includes weighted summator nodes (sometimes called adder blocks) 1372, 1374, 1376, 1378, and 1396, multiplier blocks 1384, 1392, and 1394, and delay blocks 1380 and 1382. The input ^^௧1332 is connected to the adder blocks 1372, 1374, 1376, and 1378. The output
Figure imgf000061_0001
1330 for a prior input
Figure imgf000061_0002
is also input to the adder blocks 1372, 1374, 1376, and 1378. The adder block 1372 produces an output that is input to a sigmoid block 1394-2 that produces the output ^^௧1336. Similarly, the adder block 1374 produces an output that is input to the sigmoid block 1386 that produces the output ^^௧1338. Similarly, the adder block 1376 produces an output that is input to a hyperbolic tangent block 1388 that produces the output ^^௧1340. Similarly, the adder block 1378 produces an output that is input to the sigmoid block 1390 that produces the output ^^௧1342. The multiplier block 1392 uses the outputs ^^௧1338, ^^௧1336, and output of the adder block 1396 from a prior time instance
Figure imgf000062_0001
1302 to produce a first output. The multiplier block 1394 uses the outputs ^^௧1338 and ^^௧1340 to produce a second output. The adder block 1396 sums the first output and second output to produce the output ^^௧1310. The output ^^௧1310 is input to a hyperbolic tangent block 1398 that produces an output that is input, along with the output of the sigmoid block 1390, ^^௧1342, to the multiplier block 1384 to produce the output ℎ௧1334. The delay block 1382 is used to recall (e.g., save and restore) the output of the adder block 1396 from a prior time instance. Similarly, the delay block 1380 is used to recall or save and restore the output of the multiplier block 1384 for a prior input
Figure imgf000062_0002
(e.g., from a prior time instance). Examples of delay blocks are described above in reference to Figure 13B, according to some implementations. Example Transformation of GRU Networks [00286] A Gated Recurrent Unit) (GRU) neural network is a special case of RNN. A RNN’s operations are represented by the following expressions: • ^^௧ = ^^( ^^௭ ^^௧ + ^^௭ℎ௧ି^); • ^^௧ = ^^( ^^^ ^^௧ + ^^^ℎ௧ି^); • ^^௧ = ^^ ^^ ^^ℎ( ^^ ^^௧ + ^^௧ × ^^ℎ௧ି^); • ℎ௧ = ^^௧ × ℎ௧ି^ + (1 − ^^௧) × ^^௧). [00287] In the equations above, xt is a current input vector, and ht-1 is an output calculated for the previous input vector xt-1. [00288] Figure 14A is a block diagram of a GRU neuron, according to some implementations. A sigmoid (σ) block 1418 processes the inputs ℎ௧ି^ 1402 and ^^௧1422, and produces the output ^^௧1426. A second sigmoid (σ) block 1420 processes the inputs ℎ௧ି^ 1402 and ^^௧1422, and produces the output ^^௧1428. A multiplier block 1412 multiplies the output ^^௧1426 and the input
Figure imgf000062_0003
1402 to produce and output that is input (along with the input ^^௧ 1422) to a hyperbolic tangent (tanh) block 1424 to produce the output ^^௧ 1430. A second multiplier block 1414 multiplies the output ^^௧1430 and the output ^^௧1428 to produce a first output. The block 1410 computes 1 – the output ^^௧1428 to produce an output that is input to a third multiplier block 1404 that multiplies the output and the input ℎ௧ି^ 1402 to produce a product that is input to an adder block 1406 along with the first output (from the multiplier block 1414) to produce the output ℎ௧ 1408. The input ℎ௧ି^ 1402 is the output of the GRU neuron from a prior time interval output ^^ − 1. [00289] Figure 14B is a neuron schema for a GRU neuron 1440, according to some implementations. The schema includes weighted summator nodes (sometimes called adder blocks) 1404, 1406, 1410, 1406, and 1434, multiplier blocks 1404, 1412, and 1414, and delay block 1432. The input ^^௧1422 is connected to the adder blocks 1404, 1410, and 1406. The output 1402 for a prior input
Figure imgf000063_0001
is also input to the adder blocks 1404 and 1406, and the multiplier blocks 1404 and 1412. The adder block 1404 produces an output that is input to a sigmoid block 1418 that produces the output ^^௧1428. Similarly, the adder block 1406 produces an output that is input to the sigmoid block 1420 that produces the output ^^௧1426 that is input to the multiplier block 1412. The output of the multiplier block 1412 is input to the adder block 1410 whose output is input to a hyperbolic tangent block 1424 that produces an output 1430. The output 1430 as well as the output of the sigmoid block 1418 are input to the multiplier block 1414. The output of the sigmoid block 1418 is input to the multiplier block 1404 that multiplies that output with the input from the delay block 1432 to produce a first output. The multiplier block produces a second output. The adder block 1434 sums the first output and the second output to produce the output ℎ௧1408. The delay block 1432 is used to recall (e.g., save and restore) the output of the adder block 1434 from a prior time instance. Examples of delay blocks are described above in reference to Figure 13B, according to some implementations. [00290] Operation types used in GRU are the same as the operation types for LSTM networks (described above), so GRU is transformed to trapezium-based networks following the principles described above for LSTM (e.g., using the Layer2TNNX algorithm), according to some implementations. Example Transformation of Convolutional Neural Network [00291] In general, Convolutional Neural Networks (CNN) include several basic operations, such as convolution (a set of linear combinations of image’s (or internal map’s) fragments with a kernel), activation function, and pooling (e.g., max, mean, etc.). Every calculation neuron in a CNN follows the general processing scheme of a neuron in an MLP: linear combination of some inputs with subsequent calculation of activation function. So a CNN is transformed using the MLP2TNNX algorithm described above for multilayer perceptrons, according to some implementations. [00292] Conv1D is a convolution performed over time coordinate. Figures 15A and 15B are neuron schema of variants of a single Conv1D filter, according to some implementations. In Figure 15A, a weighted summator node 1502 (sometimes called adder block, marked ‘+’) has 5 inputs, so it corresponds to 1Dconvolution with a kernel of 5. The inputs are ^^௧1504 from time ^^, ^^௧ି^ 1514 from time ^^ − 1 (obtained by inputting the input to a delay block 1506), ^^௧ିଶ 1516 from time ^^ − 2 (obtained by inputting the output of the delay block 1506 to another delay block 1508), ^^௧ିଷ 1518 from time ^^ − 3 (obtained by inputting the output of the delay block 1508 to another delay block 1510), and ^^௧ିସ 1520 from time ^^ − 4 (obtained by inputting the output of the delay block 1510 to another delay block 1512. For large kernels, it is sometimes beneficial to utilize different frequency delay blocks, so that some of the blocks produce bigger delays. Some implementations substitute several small delay blocks for one large delay block, as shown in Figure 15B. In addition to the delay blocks in Figure 15A, the example uses a delay_3 block 1524 that produces ^^௧ିଷ 1518 from time ^^ − 3, and another delay block 1526 that produces the ^^௧ିହ 1522 from time ^^ − 5. The delay_31524 block is an example of multiple delay blocks, according to some implementations. This operation does not decrease total number of blocks, but it may decrease total number of consequent operations performed over the input signal and reduce accumulation of errors, according to some implementations. [00293] In some implementations, convolutional layers are represented by trapezia- like neurons and fully connected layer is represented by cross-bar of resistors. Some implementations use cross-bars, and calculate resistance matrix for the cross-bars. Example Approximation Algorithm for Single Layer Perceptron with Multiple Outputs [00294] In some implementations, the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, and/or the analog neural network optimization module 246, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228. [00295] Suppose a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each output neuron performing an activation function F. Suppose further that ^^ ∈ ^^^×^ is a weight matrix for SLP(K, L). The following is an example for constructing a T- neural network from neurons TN(NI, NO) using an approximation algorithm Layer2TNNX_Approx, according to some implementations. The algorithm applies Layer2TNN1 algorithm (described above) at the first stage in order to decrease a number of neurons and connections, and subsequently applies Layer2TNNX to process the input of the decreased size. The outputs of the resulted neural net are calculated using shared weights of the layers constructed by the Layer2TNN1 algorithm. The number of these layers is determined by the value p, a parameter of the algorithm. If p is equal to 0 then Layer2TNNX algorithm is applied only and the transformation is equivalent. If p > 0, then p layers have shared weights and the transformation is approximate. Algorithm Layer2TNNX_Approx 1. Set the parameter p with a value from the set {0,1,…,
Figure imgf000065_0001
− 2. If p>0 apply the algorithm Layer2TNN1 with neuron TN(NI, 1) to the net SLP(K, L) and construct first p layers of the resulted subnet (PNN). The net PNN has ^^^ = neurons in the output layer.
Figure imgf000065_0002
3. Apply the algorithm Layer2TNNX with a neuron TN(NI, NO) and construct a neural subnet TNN with N୮ inputs and L outputs. 4. Set the weights of the PNN net. The weights of every neuron i of the first layer of the PNN are set according to the rule ^^ (^) ^^^ = ^^. Here, C is any constant not equal to zero, ^^ = ( ^^ − 1) ^^ூ + 1, for all weights j of this neuron except k
Figure imgf000065_0003
All other weights of the PNN net are set to 1. ^^ (^) ^^^ represents a weight for the first layer (as denoted by the superscript (1)) for the connection between the neuron i and the neuron ^^^ in the first layer. 5. Set the weights of the TNN subnet. The weights of every neuron i of the first layer of the TNN (considering the whole net this is (p+1)th layer) are set according to the equation ^^
Figure imgf000065_0004
= ^^ೖ^
Figure imgf000065_0005
. All other weights of the TNN are set to 6. Set activation functions for all neurons of the last layer of the TNN subnet as F. Activation functions of all other neurons are identity. [00296] Figure 16 shows an example architecture 1600 of the resulting neural net, according to some implementations. The example includes a PNN 1602 connected to a TNN 1606. The PNN 1602 includes a layer for ^^ inputs and produce ^^^ outputs, that is connected as input 1612 to the TNN 1606. The TNN 1606 generates L outputs 1610, according to some implementations. Approximation Algorithm for Multilayer Perceptron with Several Outputs [00297] Suppose a multilayer perceptron (MLP) includes K inputs, S layers and Li calculation neurons in i-th layer, represented as MLP( K, S, L1,…LS ). Suppose further ^^^ ∈ ^^^^×^^షభ is a weight matrix for the i-th layer. The following example algorithm constructs a T- neural network from neurons TN(NI, NO), according to some implementations. Algorithm MLP2TNNX_Approx 1. For every layer i=1,…,S: a. Apply the algorithm Layer2TNNX_Approx (described above) to SLPi(Li-1, Li) consisting of Li-1 inputs, Li output neuron, and weight matrix Ui. If i=1, then L0=K. Suppose this step constructs PTNNXi as a result. 2. Construct a MTNNX (a multilayer perceptron) by stacking all PTNNXi into one neural net, where output of a TNNXi-1 is set as input for TNNXi. Example Methods of Compression of Transformed Neural Networks [00298] In some implementations, the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, and/or the analog neural network optimization module 246, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228. [00299] This section describes example methods of compression of transformed neural networks, according to some implementations. Some implementations compress analog pyramid-like neural networks in order to minimize the number of operational amplifiers and resistors, necessary to realize the analog network on chip. In some implementations, the method of compression of analog neural networks is pruning, similar to pruning in software neural networks. There is nevertheless some peculiarities in compression of pyramid-like analog networks, which are realizable as IC analog chip in hardware. Since the number of elements, such as operational amplifiers and resistors, define the weights in analog based neural networks, it is crucial to minimize the number of operational amplifiers and resistors to be placed on chip. This will also help minimize the power consumption of the chip. Modern neural networks, such as convolutional neural networks, can be compressed 5-200 times without significant loss of the accuracy of the networks. Often, whole blocks in modern neural networks can be pruned without significant loss of accuracy. The transformation of dense neural networks into sparsely connected pyramid or trapezia or cross-bar like neural networks presents opportunities to prune the sparsely connected pyramid or trapezia-like analog networks, which are then represented by operational amplifiers and resistors in analog IC chips. In some implementations, such techniques are applied in addition to conventional neural network compression techniques. In some implementations, the compression techniques are applied based on the specific architecture of the input neural network and/or the transformed neural networks (e.g., pyramids versus trapezia versus cross-bars). [00300] For example, since the networks are realized by means of analog elements, such as operational amplifiers, some implementations determine the current which flows through the operational amplifier when the standard training dataset is presented, and thereby determine if a knot (an operational amplifier) is needed for the whole chip or not. Some implementations analyze the SPICE model of the chip and determine the knots and connections, where no current is flowing and no power is consumed. Some implementations determine the current flow through the analog IC network and thus determine the knots and connections, which are then pruned. Besides, some implementations also remove the connections if the weight of connection is too high, and/or substitute resistor to direct connector if the weight of connection is too low. Some implementations prune the knot if all connections leading to this knot have weights that are lower than a predetermined threshold (e.g., close to 0), deleting the connections where an operational amplifier always provides zero at output, and/or changing an operational amplifier to a linear junction if the amplifier gives linear function without amplification. [00301] Some implementations apply compression techniques specific to pyramid, trapezia, or cross-bar types of neural networks. Some implementations generate pyramids or trapezia with larger amount of inputs (than without the compression), thus minimizing the number of layers in pyramid or trapezia. Some implementations generate a more compact trapezia network by maximizing the number of outputs of each neuron. Example Generation of Optimal Resistor Set [00302] In some implementations, the example computations described herein are performed by the weight matrix computation or weight quantization module 238 (e.g., using the resistance calculation module 240) that compute the weights 272 for connections of the transformed neural networks, and/or corresponding resistance values 242 for the weights 272. [00303] This section describes an example of generating an optimal resistor set for a trained neural network, according to some implementations. An example method is provided for converting connection weights to resistor nominals for implementing the neural network (sometimes called a NN model) on a microchip with possibly less resistor nominals and possibly higher allowed resistor variance. [00304] Suppose a test set ‘Test’ includes around 10,000 values of input vector (x and y coordinates) with both coordinates varying in the range [0;1], with a step of 0.01. Suppose network NN output for given input X is given by Out = NN(X). Suppose further that input value class is found as follows: Class_nn(X) = NN(X) > 0.61 ? 1 : 0. [00305] The following compares a mathematical network model M with a schematic network model S. The schematic network model includes possible resistor variance of rv and processes the ‘Test’ set, each time producing a different vector of output values S(Test) = Out_s. Output error is defined by the following equation:
Figure imgf000068_0001
[00306] Classification error is defined by the following equation: (X୧) ≠ Class_m(X୧) ^
Figure imgf000068_0002
^^ [00307] Some implementations set the desired classification error as no more than 1%. Example Error Analysis [00308] Figure 17A shows an example chart 1700 illustrating dependency between output error and classification error on the M network, according to some implementations. In Figure 17A, the x-axis corresponds to classification margin 1704, and the y-axis corresponds to total error 1702 (see description above). The graph shows total error (difference between output of model M and real data) for different classification margins of output signal. For this example, according to the chart, the optimal classification margin 1706 is 0.610. [00309] Suppose another network O produces output values with a constant shift versus relevant M output values, there would be classification error between O and M. To keep the classification error below 1%, this shift should be in the range of [-0.045, 0.040]. Thus, possible output error for S is 45 mV. [00310] Possible weight error is determined by analyzing dependency between weight/bias relative error over the whole network and output error. The charts 1710 and 1720 shown in Figures 17B and 17C, respectively, are obtained by averaging 20 randomly modified networks over the ‘Test’ set, according to some implementations. In these charts, x-axis represents the absolute weight error 1712 and y-axis represents the absolute output error 1714. As can be seen from the charts, output error limit of 45 mV (y = 0.045) allows for 0.01 relative or 0.01 absolute error value (value of x) for each weight. Maximum weight modulus (maximum of absolute value of weights among all weights) for the neural network is 1.94. Example Process for Choosing Resistor Set [00311] A resistor set together with a {R+, R-} pair chosen from this set has a value function over the required weight range [-wlim; wlim] with some degree of resistor error r_err. In some implementations, value function of a resistor set is calculated as follows: • Possible weight options array is calculated together with weight average error dependent on resistor error; • The weight options in the array is limited to the required weight range [-wlim; wlim]; • Values that are worse than neighboring values in terms of weight error are removed; • An array of distances between neighboring values is calculated; and • The value function is a composition of square mean or maximum of the distances array. [00312] Some implementations iteratively search for an optimal resistor set by consecutively adjusting each resistor value in the resistor set on a learning rate value. In some implementations, the learning rate changes over time. In some implementations, an initial resistor set is chosen as uniform (e.g., [1;1;…;1]), with minimum and maximum resistor values chosen to be within two orders of magnitude range (e.g., [1;100] or [0.1;10]). Some implementation choose R+ = R-. In some implementations, the iterative process converges to a local minimum. In one case, the process resulted in the following set: [0.17, 1.036, 0.238, 0.21, 0.362, 1.473, 0.858, 0.69, 5.138, 1.215, 2.083, 0.275]. This is a locally optimal resistor set of 12 resistors for the weight range [-2; 2] with rmin = 0.1 (minimum resistance), rmax = 10 (maximum resistance), and r_err = 0.001 (an estimated error in the resistance). Some implementations do not use the whole available range [rmin; rmax] for finding a good local optimum. Only part of the available range (e.g., in this case [0.17; 5.13]) is used. The resistor set values are relative, not absolute. Is this case, relative value range of 30 is enough for the resistor set. [00313] In one instance, the following resistor set of length 20 is obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02]. In this example, the value 1.763 is also the R- = R+ value. This set is subsequently used to produce weights for NN, producing corresponding model S. The model S’s mean square output error was 11 mV given the relative resistor error is close to zero, so the set of 20 resistors is more than required. Maximum error over a set of input data was calculated to be 33 mV. In one instance, S, DAC, and ADC converters with 256 levels were analyzed as a separate model, and the result showed 14 mV mean square output error and 49 mV max output error. An output error of 45 mV on NN corresponds to a relative recognition error of 1%. The 45 mV output error value also corresponds to 0.01 relative or 0.01 absolute weight error, which is acceptable. Maximum weight modulus in NN is 1.94. In this way, the optimal (or near optimal) resistor set is determined using the iterative process, based on desired weight range [-wlim; wlim], resistors error (relative), and possible resistors range. [00314] Typically, a very broad resistor set is not very beneficial (e.g., between 1-1/5 orders of magnitude is enough) unless different precision is required within different layers or weight spectrum parts. For example, suppose weights are in the range of [0, 1], but most of the weights are in the range of [0, 0.001], then better precision is needed within that range. In the example described above, given the relative resistor error is close to zero, the set of 20 resistors is more than sufficient for quantizing the NN network, with given precision. In one instance, on a set of resistors [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] (note values are relative), an average S output error of 11 mV was obtained. Example Process for Quantization of Resistor Values [00315] In some implementations, the example computations described herein are performed by the weight matrix computation or weight quantization module 238 (e.g., using the resistance calculation module 240) that compute the weights 272 for connections of the transformed neural networks, and/or corresponding resistance values 242 for the weights 272. [00316] This section describes an example process for quantizing resistor values corresponding to weights of a trained neural network, according to some implementations. The example process substantially simplifies the process of manufacturing chips using analog hardware components for realizing neural networks. As described above, some implementations use resistors to represent neural network weights and/or biases for operational amplifiers that represent analog neurons. The example process described here specifically reduces the complexity in lithographically fabricating sets of resistors for the chip. With the procedure of quantizing the resistor values, only select values of resistances are needed for chip manufacture. In this way, the example process simplifies the overall process of chip manufacture and enables automatic resistor lithographic mask manufacturing on demand. [00317] Figure 18 provides an example scheme of a neuron model 1800 used for resistors quantization, according to some implementations. In some implementations, the circuit is based on an operational amplifier 1824 (e.g., AD824 series precision amplifier) that receives input signals from negative weight fixing resistors (R1- 1804, R2- 1806, Rb- bias 1816, Rn- 1818, and R- 1812), and positive weight fixing resistors (R1+ 1808, R2+ 1810, Rb+ bias 1820, Rn+ 1822), and R+ 1814). The positive weight voltages are fed into direct input of the operational amplifier 1824 and negative weights voltages are fed into inverse input of the operational amplifier 1824. The operational amplifier 1824 is used to allow weighted summation operation of weighted outputs from each resistor, where negative weights are subtracted from positive weights. The operational amplifier 1824 also amplifies signal to the extent necessary for the circuit operation. In some implementations, the operational amplifier 1824 also accomplishes RELU transformation of output signal at it’s output cascade. [00318] The following equations determine the weights, based on resistor values: • Voltage at the output of neuron is determined by the following equation:
Figure imgf000072_0001
• The weights of each connection are determined by following equation: ^^ା ି ^^ = − ^^ ^ ^^ା ^ ^^^ [00319] The following example optimization procedure quantizes the values of each resistance and minimize the error of neural network output, according to some implementations: 1. Obtain a set of connection weights and biases {w1, …, wn, b}.. 2. Obtain possible minimum and maximum resistor values {Rmin, Rmax}. These parameters are determined based on the technology used for manufacturing. Some implementations use TaN or Tellurium high resistivity materials. In some implementations, the minimum value of resistor is determined by minimum square that can be formed lithographically. The maximum value is determined by length, allowable for resistors (e.g., resistors made from TaN or Tellurium) to fit to the desired area, which is in turn determined by the area of an operational amplifier square on lithographic mask. In some implementations, the area of arrays of resistors is smaller than the area of one operational amplifier, since the arrays of resistors are stacked (e.g., one in BEOL, another in FEOL). 3. Assume that each resistor has r_err relative tolerance value 4. The goal is to select a set of resistor values {R1, …, Rn} of given length N within the defined [Rmin; Rmax], based on {w1, …, wn, b} values. An example search algorithm is provided below to find sub-optimal {R1, …, Rn} set based on particular optimality criteria. 5. Another algorithm chooses {Rn, Rp, Rni, Rpi} for a network given that {R1..Rn} is determined. Example {R1, …, Rn} Search Algorithm [00320] Some implementations use an iterative approach for resistor set search. Some implementations select an initial (random or uniform) set {R1, …, Rn} within the defined range. Some implementations select one of the elements of the resistor set as a R- = R+ value. Some implementations alter each resistor within the set by a current learning rate value until such alterations produce ‘better’ set (according to a value function). This process is repeated for all resistors within the set and with several different learning rate values, until no further improvement is possible. [00321] Some implementations define the value function of a resistor set as follows: • Possible weight options are calculated according to the formula (described above):
Figure imgf000073_0001
• Expected error value for each weight option is estimated based on potential resistor relative error r_err determined by IC manufacturing technology. • Weight options list is limited or restricted to [-wlim; wlim] range • Some values, which have expected error beyond a high threshold (e.g., 10 times r_err), are removed • Value function is calculated as a square mean of distance between two neighboring weight options. So, value function is minimal when weight options are distributed uniformly within [-wlim; wlim] range [00322] Suppose the required weight range [-wlim; wlim] for a model is set to [-5; 5], and the other parameters include N = 20, r_err = 0.1%, rmin = 100 KΩ, rmax = 5 MΩ. Here, rmin and rmax are minimum and maximum values for resistances, respectively. [00323] In one instance, the following resistor set of length 20 was obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] MΩ. R- = R+ = 1.763 MΩ. Example {Rn, Rp, Rni, Rpi} Search Algorithm [00324] Some implementations determine Rn and Rp using an iterative algorithm such as the algorithm described above. Some implementations set Rp = Rn (the tasks to determine Rn and Rp are symmetrical – the two quantities typically converge to a similar value). Then for each weight
Figure imgf000074_0001
some implementations select a pair of resistances {Rni, Rpi} that the estimated weight error value:
Figure imgf000074_0002
[00325] Some implementations subsequently use the {Rni; Rpi; Rn; Rp} values set to implement neural network schematics. In one instance, the schematics produced mean square output error (sometimes called S mean square output error, described above) of 11 mV and max error of 33 mV over a set of 10,000 uniformly distributed input data samples, according to some implementations. In one instance, S model was analyzed along with digital-to-analog converters (DAC), analog-to-digital converters (ADC), with 256 levels as a separate model. The model produced 14 mV mean square output error and 49 mV max output error on the same data set, according to some implementations. DAC and ADC have levels because they convert analog value to bit value and vice-versa. 8 bits of digital value is equal to 256 levels. Precision cannot be better than 1/256 for 8-bit ADC. [00326] Some implementations calculate the resistance values for analog IC chips, when the weights of connections are known, based on Kirchhoff’s circuit laws and basic principles of operational amplifiers (described below in reference to Figure 19A), using Mathcad or any other similar software. In some implementations, operational amplifiers are used both for amplification of signal and for transformation according to the activation functions (e.g., ReLU, sigmoid, Tangent hyperbolic, or linear mathematical equations), [00327] Some implementations manufacture resistors in a lithography layer where resistors are formed as cylindrical holes in the SiO2 matrix and the resistance value is set by the diameter of hole. Some implementations use amorphous TaN, TiN of CrN or Tellurium as the highly resistive material to make high density resistor arrays. Some ratios of Ta to N Ti to N and Cr to N provide high resistance for making ultra-dense high resistivity elements arrays. For example, for TaN, Ta5N6, Ta3N5, the higher the N ratio to Ta, the higher is the resistivity. Some implementations use Ti2N, TiN, CrN, or Cr5N, and determine the ratios accordingly. TaN deposition is a standard procedure used in chip manufacturing and is available at all major Foundries. Example Operational Amplifier [00328] Figure 19A shows a schematic diagram of an operational amplifier made on CMOS (CMOS OpAmp) 1900, according to some implementations. In Figure 19A, In+ (positive input or pos) 1404, and In- (negative input or neg) 1406, and Vdd- (positive supply voltage relative to GND) 1402 are contact inputs. Contact Vss- (negative supply voltage or GND) is indicated by the label 1408. The circuit output is Out 1410 (contact output). Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel) to W (the width of the gate channel), examples of which are shown in the Table shown in Figure 19B (described below). The current mirror is made on NMOS transistors M111944, M121946, and resistor R11921 (with an example resistance value of 12 kΩ), and provides the offset current of the differential pair (M11926 and M31930). The differential amplifier stage (differential pair) is made on the NMOS transistors M11926 and M3 1930. Transistors M1, M3 are amplifying, and PMOS transistors M21928 and M41932 play the role of active current load. From the M3 transistor, the signal is input to the gate of the output PMOS transistor M71936. From the transistor M1, the signal is input to the PMOS transistor M5 (inverter) 1934 and the active load on the NMOS transistor M61934. The current flowing through the transistor M5 1934 is the setting for the NMOS transistor M8 1938. Transistors M71936 is included in the scheme with a common source for a positive half-wave signal. The M8 transistors 1938 are enabled by a common source circuit for a negative half- wave signal. To increase the overall load capacity of the operational amplifier, the M71936 and M81938 outputs include an inverter on the M91940 and M101942 transistors. Capacitors C11912 and C21914 are blocking. [00329] Figure 19B shows a table 1948 of description for the example circuit shown in Figure 19A, according to some implementations. The values for the parameters are provided as examples, and various other configurations are possible. The transistors M1, M3, M6, M8, M10, M11, and M12 are N-Channel MOSFET transistors with explicit substrate connection. The other transistors M2, M4, M5, M7, and M9 are P-Channel MOSFET transistors with explicit substrate connection. The Table shows example shutter ratio of length (L, column 1) and width (W, column 2) are provided for each of the transistors (column 3). [00330] In some implementations, operational amplifiers such as the example described above are used as the basic element of integrated circuits for hardware realization of neural networks. In some implementations, the operational amplifiers are of the size of 40 square microns and fabricated according to 45 nm node standard. [00331] In some implementations, activation functions, such as ReLU, Hyperbolic Tangent, and Sigmoid functions are represented by operational amplifiers with modified output cascade. For example, RELU, Sigmoid, or Tangent function is realized as an output cascade of an operational amplifier (sometimes called OpAmp) using corresponding well-known analog schematics, according to some implementations. [00332] In the examples described above and below, in some implementations, the operational amplifiers are substituted by inverters, current mirrors, two-quadrant or four quadrant multipliers, and/or other analog functional blocks, that allow weighted summation operation. Example Scheme of a LSTM Block [00333] Figures 20A-20E show a schematic diagram of a LSTM neuron 20000, according to some implementations. The inputs of the neuron are Vin120002 and Vin220004 that are values in the range [-0.1,0.1]. The LSTM neuron also input the value of the result of calculating the neuron at time H(t-1) (previous value; see description above for LST neuron) 20006 and the state vector of the neuron at time C(t-1) (previous value) 20008. Outputs of the neuron LSTM (shown in Figure 20B) include the result of calculating the neuron at the present time H(t) 20118 and the state vector of the neuron at the present time C(t) 20120. The scheme includes: • a “neuron O” assembled on the operational amplifiers U120094 and U220100, shown in Figure 20A. Resistors R_Wo1 20018, R_Wo2 20016, R_Wo3 20012, R_Wo4 20010, R_Uop120014, R_Uom120020, Rr 20068 and Rf220066 set the weights of connections of the single “neuron O”. The “neuron O” uses a sigmoid (module X1 20078, Figure 20B) as a nonlinear function; • a "neuron C“ assembled on the operational amplifiers U320098 (shown in Figure 20C) and U420100 (shown in Figure 20A). Resistors R_Wc120030, R_Wc220028, R_Wc3 20024, R_Wc420022, R_Ucp120026, R_Ucm120032, Rr 20122, and Rf220120, set the weights of connections of the “neuron С”. The “neuron C” uses a hyperbolic tangent (module X222080, Figure 2B) as a nonlinear function; • a “neuron I” assembled on the operational amplifiers U520102 and U620104, shown in Figure 20C. Resistors R_Wi120042, R_Wi2 20040, R_Wi320036, and R_Wi4 20034, R_Uip120038, R_Uim120044, Rr 20124, and Rf220126 set the weights of connections of the “neuron I”. The “neuron I” uses a sigmoid (module X320082) as a nonlinear function; and • a "neuron f“ assembled on the operational amplifiers U720106 and U820108, as shown in Figure 20D. Resistors R_Wf120054, R_Wf220052, R_Wf320048, R_Wf420046, R_Ufp1 20050, R_Ufm1 20056, Rr 20128 and Rf2 20130 set the weights of connections of the “neuron f”. The “neuron f” uses a sigmoid (module X420084) as a nonlinear function. [00334] The outputs of modules X220080 (Figure 20B) and X320082 (Figure 20C) are input to the X5 multiplier module 20086 (Figure 20B). The outputs of modules X420084 (Figure 20D) and buffer to U920010 are input to the multiplier module X620088. The outputs of the modules X520086 and X620088 are input to the adder (U1020112). A divider 10 is assembled on the resistors R1 20070, R2 20072, and R3 20074. A nonlinear function of hyperbolic tangent (module X720090, Figure 20B) is obtained with the release of the divisor signal. The output C(t) 20120 (a current state vector of the LSTM neuron) is obtained with the buffer-inverter on the U1120114 output signal. The outputs of modules X120078 and X7 20090 is input to a multiplier (module X820092) whose output is input to a buffer divider by 10 on the U1220116. The result of calculating the LSTM neuron at the present time H(t) 20118 is obtained from the output signal of U1220116. [00335] Figure 20E shows example values for the different configurable parameters (e.g., voltages) for the circuit shown in Figures 20A-20D, according to some implementations. Vdd 20058 is set to +1.5V, Vss 20064 is set to –1.5V, Vdd120060 is set to +1.8V, Vss120062 is set to -1.0V, and GND 20118 is set to GND, according to some implementations. [00336] Figure 20F shows a table 20132 of description for the example circuit shown in Figure 20A-20D, according to some implementations. The values for the parameters are provided as examples, and various other configurations are possible. The transistors U1 – U12 are CMOS OpAmps (described above in reference to Figures 19A and 19B). X1, X3, and X4 are modules that perform the Sigmoid function. X2 and X7 are modules that perform the Hyperbolic Tangent function. X5 and X8 are modules that perform the multiplication function. Example resistor ratings include: Rw = 10 kΩ, and Rr = 1.25 kΩ. The other resistor values are expressed relative to Rw. For example, Rf2 = 12 times Rw, R_Wo4 = 5 times Rw, R_Wo3 = 8 times Rw, R_Uop1 = 2.6 times Rw, R_Wo2 = 12 times Rw, R_W1 = w times Rw, and R_Uom1 = 2.3 times Rw, R_wc4 = 4 times Rw, R_Wc3 = 5.45 times Rw, R_Ucp1 = 3 times Rw, R_Wc2= 12 times Rw, R_Wc1= 2.72 times Rw, R_Ucm1= 3.7 times Rw, R_Wi4 = 4.8 times Rw, W_Wi3 = 6 times Rw, W_Uip1 = 2 times Rw, R_Wi2 = 12 times Rw, R_Wi1 = 3 times Rw, R_Uim1 = 2.3 times Rw, R_Wf4 = 2.2 times Rw, R_Wf3 = 5 times Rw, R_Wfp = 4 times Rw, R_Wf2= 2 times Rw, R_Wf1= 5.7 times Rw, and Rfm1 = 4.2 times Rw. Example Scheme of a Multiplier Block [00337] Figures 21A-21I show a schematic diagram of a multiplier block 21000, according to some implementations. The neuron 21000 is based on the principle of a four- quadrant multiplier, assembled using operational amplifiers U121040 and U221042 (shown in Figure 21B), U321044 (shown in Figure 21H), and U421046 and U521048 (shown in Figure 21I), and CMOS transistors M121052 through M6821182. The inputs of the multiplier include V_one 21020 21006 and V_two 21008 (shown in Figure 21B), and contact Vdd (positive supply voltage, e.g., +1.5 V relative to GND) 21004 and contact Vss (negative supply voltage, e.g., -1.5 V relative to GND) 21002. In this scheme, additional supply voltages are used: contact Input Vdd1 (positive supply voltage, e.g., +1.8 V relative to GND), contact Vss1 (negative supply voltage, e.g., -1.0 V relative to GND). The result of the circuit calculations are output at mult_out (output pin) 21170 (shown in Figure 21I). [00338] Referring to Figure 21B, input signal (V_one) from V_one 21006 is connected to the inverter with a single gain made on U121040, the output of which forms a signal negA 21006, which is equal in amplitude, but the opposite sign with the signal V_one. Similarly, the signal (V_two) from the input V_two 21008 is connected to the inverter with a single gain made on U221042, the output of which forms a signal negB 21012 which is equal in amplitude, but the opposite sign with the signal V_two. Pairwise combinations of signals from possible combinations (V_one, V_two, negA, negB) are output to the corresponding mixers on CMOS transistors. [00339] Referring back to Figure 21A, V_two 21008 and negA 21010 are input to a multiplexer assembled on NMOS transistors M1921086, M2021088, M2121090, M2221092, and PMOS transistors M2321094 and M2421096. The output of this multiplexer is input to the NMOS transistor M621060 (Figure 21D). [00340] Similar transformations that occur with the signals include: • negB 21012 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M1121070, M122072, M132074, M1421076, and PMOS transistors M15 2078 and M1621080. The output of this multiplexer is input to the M521058 NMOS transistor (shown in Figure 21D); • V_one 21020 and negB 21012 are input to a multiplexer assembled on PMOS transistors M18 21084, M48 21144, M49 21146, and M50 21148, and NMOS transistors M1721082, M4721142. The output of this multiplexer is input to the M9 PMOS transistor 21066 (shown in Figure 21D); • negA 21010 and V_two 21008 are input to a multiplexer assembled on PMOS transistors M52 21152, M54 21156, M55 21158, and M56 21160, and NMOS transistors M5121150, and M5321154. The output of this multiplexer is input to the M2 NMOS transistor 21054 (shown in Figure 21C); • negB 21012 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M11 21070, M12 21072, M13 21074, and M14 21076, and PMOS transistors M1521078, and M1621080. The output of this multiplexer is input to the M10 NMOS transistor 21068 (shown in Figure 21D); • negB 21012 and negA 21010 are input to a multiplexer assembled on NMOS transistors M35 21118, M3621120, M3721122, and M38 21124, and PMOS transistors M39 21126, and M4021128. The output of this multiplexer is input to the M27 PMOS transistor 21102 (shown in Figure 21H); • V_two 21008 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M41 21130, M42 21132, M43 21134, and M44 21136, and PMOS transistors M4521138, and M4621140. The output of this multiplexer is input to the M30 NMOS transistor 21108 (shown in Figure 21H); • V_one 21020 and V_two 21008 are input to a multiplexer assembled on PMOS transistors M58 21162, M60 21166, M61 21168, and M62 21170, and NMOS transistors M5721160, and M5921164. The output of this multiplexer is input to the M34 PMOS transistor 21116 (shown in Figure 21H); and • negA 21010 and negB 21012 are input to a multiplexer assembled on PMOS transistors M6421174, M6621178, M6721180, and M6821182, and NMOS transistors M63 21172, and M6521176. The output of this multiplexer is input to the PMOS transistor M3321114 (shown in Figure 21H). [00341] The current mirror (transistors M1 21052, M2 21053, M3 21054, and M4 21056) powers the portion of the four quadrant multiplier circuit shown on the left, made with transistors M521058, M621060, M721062, M821064, M921066, and M1021068. Current mirrors (on transistors M2521098, M2621100, M2721102, and M2821104) power supply of the right portion of the four-quadrant multiplier, made with transistors M2921106, M3021108, M31 21110, M3221112, M3321114, and M3421116. The multiplication result is taken from the resistor Ro 21022 enabled in parallel to the transistor M321054 and the resistor Ro 21188 enabled in parallel to the transistor M2821104, supplied to the adder on U321044. The output of U321044 is supplied to an adder with a gain of 7,1, assembled on U521048, the second input of which is compensated by the reference voltage set by resistors R121024 and R221026 and the buffer U421046, as shown in Figure 21I. The multiplication result is output via the Mult_Out output 21170 from the output of U521048. [00342] Figure 21J shows a table 21198 of description for the schematic shown in Figures 21A-21I, according to some implementations. U1 – U5 are CMOS OpAmps. The N- Channel MOSFET transistors with explicit substrate connection include transistors M1, M2, M25, and M26 (with shutter ratio of length (L) = 2.4u, and shutter ratio of width (W) = 1.26u), transistors M5, M6, M29, and M30 (with L = 0.36u, and W = 7.2u), transistors M7, M8, M31, and M32 (with L = 0.36u, and W = 199.98u), transistors M11-M14, M19-M22, M35-M38, and M41-M44 (with L = 0.36u and W= 0.4u), and transistors M17, M47, M51, M53, M57, M59, M43, and M64 (with L = 0.36u and W = 0.72u). The P-Channel MOSFET transistors with explicit substrate connection include transistors M3, M4, M27, and M28 (with shutter ratio of length (L) = 2.4u, and shutter ratio of width (W) = 1.26u), transistors M9, M10, M33, and M34 (with L = 0.36u, and W = 7.2u), transistors M18, M48, M49, M50, M52, M54, M55, M56, M58, M60, M61, M62, M64, M66, M67, and M68 (with L = 0.36u, and W = 0.8u), and transistors M15, M16, M23, M24, M39, M40, M45, and M46 (with L = 0.36u and W = 0.72u). Example resistor ratings include Ro = 1 kΩ, Rin = 1 kΩ, Rf = 1 kΩ, Rc4 = 2 kΩ, and Rc5 = 2 kΩ, according to some implementations. Example Scheme of a Sigmoid Block [00343] Figure 22A shows a schematic diagram of a sigmoid block 2200, according to some implementations. The sigmoid function (e.g., modules X120078, X320082, and X4 20084, described above in reference to Figures 20A-20F) is implemented using operational amplifiers U12250, U22252, U32254, U42256, U52258, U62260, U7, 2262, and U82264, and NMOS transistors M12266, M22268, and M32270. Contact sigm_in 2206 is module input, contact Input Vdd12222 is positive supply voltage +1.8 V relative to GND 2208, and contact Vss12204 is negative supply voltage -1.0 V relative to GND. In this scheme, U42256 has a reference voltage source of -0.2332 V, and the voltage is set by the divider R102230 and R112232. The U52258 has a reference voltage source of 0.4 V, and the voltage is set by the divider R122234 and R132236. The U62260 has a reference voltage source of 0.32687 V, the voltage is set by the divider R142238 and R152240. The U72262 has a reference voltage source of -0.5 V, the voltage is set by the divider R162242 and R172244. The U82264 has a reference voltage source of -0.33 V, the voltage is set by the divider R182246 and R192248. [00344] The sigmoid function is formed by adding the corresponding reference voltages on a differential module assembled on the transistors M1 2266 and M2 2268. A current mirror for a differential stage is assembled with active regulation operational amplifier U32254, and the NMOS transistor M32270. The signal from the differential stage is removed with the NMOS transistor M2 and resistor R52220 is input to the adder U22252. The output signal sigm_out 2210 is removed from the U2 adder 2252 output. [00345] Figure 22B shows a table 2278 of description for the schematic diagram shown in Figure 22A, according to some implementations. U1-U8 are CMOS OpAmps. M1, M2, and M3 are N-Channel MOSFET transistors with a shutter ratio of length (L) = 0.18u, and shutter ration of width (W) = 0.9u, according to some implementations. Example Scheme of a Hyperbolic Tangent Block [00346] Figure 23A shows a schematic diagram of a hyperbolic tangent function block 2300, according to some implementations. The hyperbolic tangent function (e.g., the modules X2 20080, and X720090 described above in reference to Figures 20A-20F) is implemented using operational amplifiers (U12312, U22314, U32316, U42318, U52320, U62322, U7 2328, and U82330) and NMOS transistors (M12332, M22334, and M32336). In this scheme, contact tanh_in 2306 is module input, contact Input Vdd12304 is positive supply voltage +1.8 V relative to GND 2308, and contact Vss12302 is negative supply voltage -1.0 V relative to GND. Further, in this scheme, U42318 has a reference voltage source of -0.1 V, the voltage set by the divider R102356 and R112358. The U52320 has a reference voltage source of 1.2 V, the voltage set by the divider R122360 and R132362. The U62322 has a reference voltage source of 0.32687 V, the voltage set by the divider R142364 and R152366. The U72328 has a reference voltage source of -0.5 V, the voltage set by the divider R162368 and R172370. The U82330 has a reference voltage source of -0.33 V, the voltage set by the divider R182372 and R19 2374. The hyperbolic tangent function is formed by adding the corresponding reference voltages on a differential module made on transistors M12332 and M22334. A current mirror for a differential stage is obtained with active regulation operational amplifier U3 2316, and NMOS transistor M32336. With NMOS transistor M22334 and resistor R5 2346, the signal is removed from the differential stage and input to the adder U22314. The output signal tanh_out 2310 is removed from the U2 adder 2314 output. [00347] Figure 23B shows a table 2382 of description for the schematic diagram shown in Figure 23A, according to some implementations. U1-U8 are CMOS OpAmps, and M1, M2, and M3 are N-Channel MOSFET transistors, with a shutter ratio of length (L) = 0.18u, and the shutter ratio of width (W) = 0.9u. Example Scheme of a Single Neuron OP1 CMOS OpAmp [00348] Figures 24A-24C show a schematic diagram of a single neuron OP1 CMOS OpAmp 2400, according to some implementations. The example is a variant of a single neuron on an operational amplifier, made on CMOS according to an OP1 scheme described herein. In this scheme, contacts V12410 and V22408 are inputs of a single neuron, contact bias 2406 is voltage +0.4 V relative to GND, contact Input Vdd 2402 is positive supply voltage +5.0 V relative to GND, contact Vss 2404 is GND, and contact Out 2474 is output of a single neuron. Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel), and W (the width of the gate channel). This Op Amp has two current mirrors. The current mirror on NMOS transistors M32420, M62426, and M132440 provides the offset current of the differential pair on NMOS transistors M22418 and M52424. The current mirror in the PMOS transistors M72428, M82430, and M152444 provides the offset current of the differential pair on the PMOS transistors M92432 and M102434. In the first differential amplifier stage, NMOS transistors M22418 and M52424 are amplifying, and PMOS transistors M12416 and M42422 play the role of active current load. From the M5 2424 transistor, the signal is output to the PMOS gate of the transistor M132440. From the M2 2418 transistor, the signal is output to the right input of the second differential amplifier stage on PMOS transistors M92432 and M102434. NMOS transistors M112436 and M12 2438 play the role of active current load for the M92432 and M102434 transistors. The M17 2448 transistor is switched on according to the scheme with a common source for a positive half-wave of the signal. The M182450 transistor is switched on according to the scheme with a common source for the negative half-wave of the signal. To increase the overall load capacity of the Op Amp, an inverter on the M172448 and M182450 transistors is enabled at the output of the M132440 and M142442 transistors. [00349] Figure 24D shows a table 2476 of description for the schematic diagram shown in Figure 24A-24C, according to some implementations. The weights of the connections of a single neuron (with two inputs and one output) are set by the resistor ratio: w1 = (Rp / R1+) – (Rn / R1 -); w2 = (Rp / R2+) – (Rn / R2 -); w bias = (Rp / Rbias+) – (Rn / Rbias -). Normalizing resistors (Rnorm - and Rnorm+) are necessary to obtain exact equality: (Rn /R1 -) + (Rn / R2 -) + (Rn / Rbias -) + (Rn / Rnorm -) = (Rp /R1 +) + (Rp / R2 +) + (Rp /Rbias+) + (Rp / Rnorm +). N-Channel MOSFET transistors with explicit substrate connection include transistors M2 and M5 with L = 0.36u and W = 3.6u, transistors M3, M6, M11, M12, M14, and M16 with L = 0.36u and W = 1.8u, and transistor M18 with L = 0.36u and W = 18u. P- Channel MOSFET transistors with explicit substrate connection include transistors M1,M4, M7, M8, M13, and M15 with L = 0.36u and W = 3.96u, transistors M9 and M10 with L = 0.36u and W = 11.88u, and transistor M17 with L = 0.36u and W = 39.6u. Example Scheme of a Single Neuron OP3 CMOS OpAmp [00350] Figures 25A-25D show a schematic diagram of a variant of a single neuron 25000 on operational amplifiers, made on CMOS according to an OP3 scheme, according to some implementations. The single neuron consists of three simple operational amplifiers (OpAmps), according to some implementations. The unit Neuron adder is performed on two Opamps with bipolar power supply and the RELU activation function is performed on an OpAmp with unipolar power supply and with a gain of = 10. Transistors M125028 – M16 25058 are used for summation of negative connections of the neuron. Transistors M1725060 – M3225090 are used for adding the positive connections of the neuron. The RELU activation function is performed on the transistors M3325092 – M4625118. In the scheme, contacts V1 25008 and V225010 are inputs of the single neuron, contact bias 25002 is voltage +0.4 V relative to GND, contact Input Vdd 25004 is positive supply voltage +2.5 V relative to GND, contact Vss 25006 is negative supply voltage -2.5 V, and contact Out 25134 is output of the single neuron. Parameters of CMOS transistors used in a single neuron are determined by the ratio of geometric dimensions: L (the length of the gate channel) and W (the width of the gate channel). Consider the operation of the simplest OpAmp included in a single neuron. Each op amp has two current mirrors. The current mirror on NMOS transistors M325032 (M1925064, M35 25096), M625038 (M2225070, M3825102) and M1625058 (M3225090, M4825122) provides the offset current of the differential pair on NMOS transistors M225030 (M1825062, M34 25094) and M525036 (M2125068, M3525096). The current mirror in PMOS transistors M7 25040 (M2325072, M3925104), M825042 (M2425074, M4025106) and M1525056 (M312588) provides the offset current of the differential pair on PMOS transistors M925044 (M2525076, M4125108) and M1025046 (M2625078, M4225110). In the first differential amplifier stage, NMOS transistors M225030 (M1825062, M3425094) and M525036 (M21 25068, M37 25100) are amplifying, and PMOS transistors M1 25028 (M17 25060, M33 25092) and M425034 (M2025066, M3625098) play the role of active current load. From the transistor M525036 (M2125068, M3725100), the signal is input to the PMOS gate of the transistor M1325052 (M2925084, M4525116). From the transistor M225030 (M1825062, M34 25094), the signal is input to the right input of the second differential amplifier stage on PMOS transistors M925044 (M2525076, M4125108) and M1025046 (M2625078, M42 25110). NMOS transistors M1125048 (M2725080, M4325112) and M1225048 (M2825080, M44 25114) play the role of active current load for transistors M925044 (M2525076, M41 25108) and M1025046 (M2625078, M4225110). Transistor M1325052 (M2925082, M45 25116) is included in the scheme with a common source for a positive half-wave signal. The transistor M14 25054 (M3025084, M4625118) is switched on according to the scheme with a common source for the negative half-wave of the signal. [00351] The weights of the connections of a single neuron (with two inputs and one output) are set by the resistor ratio: w1 = (R feedback / R1+) – (R feedback / R1 -); w2 = (R feedback / R2+) – (R feedback / R2 -); wbias = (R feedback / Rbias+) – (R feedback / Rbias-); w1 = (R p*K amp / R1+) – (R n*K amp / R1 -); w2 = (R p*K amp / R2+) – (R n*K amp / R2 -); wbias = (R p*K amp / Rbias+) – (R n*K amp / Rbias-), where K amp = R1ReLU / R2ReLU. R feedback = 100k – used only for calculating w1, w2, wbias. According to some implementations, example values include: R feedback = 100k, Rn = Rp = Rcom = 10k, K amp ReLU = 1 + 90k / 10k = 10, w1 = (10k * 10 / 22.1k) – (10k * 10 / 21.5k) = -0.126276, w2 = (10k * 10 / 75k) – (10k * 10 / 71.5k) = - 0.065268, wbias = (10k * 10 / 71.5k) –(10k * 10 / 78.7k) = 0.127953. [00352] The input of the negative link adder of the neuron (M1 – M17) is received from the positive link adder of the neuron (M17 – M32) through the Rcom resistor. [00353] Figure 25E shows a table 25136 of description for the schematic diagram shown in Figure 25A-25D, according to some implementations. N-Channel MOSFET transistors with explicit substrate connection include transistors M2, M5, M18, M21, M34, and M37, with L = 0.36u and W = 3.6u, transistors M3, M6, M11, M12, M14, M16, M19, M22, M27, M28, M32, M38, M35, M38, M43, M44, M46, and M48, with L = 0.36u and W = 1.8u. P-Channel MOSFET transistors with explicit substrate connection include transistors M1, M4, M7, M8, M13, M15, M17, M20, M23, M24, M29, M31, M33, M36, M39, M40, M45, and M47 with L = 0.36u and W = 3.96u, and transistor M9, M10, M25, M26, M41, and M42, with L = 0.36u and W = 11.88u. Example Methods for Analog Hardware Realization of Trained Neural Networks [00354] Figures 27A-27J show a flowchart of a method 2700 for hardware realization (2702) of neural networks, according to some implementations. The method is performed (2704) at the computing device 200 (e.g., using the neural network transformation module 226) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. The method includes obtaining (2706) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220). In some implementations, the trained neural network is trained (2708) using software simulations to generate the weights. [00355] The method also includes transforming (2710) the neural network topology to an equivalent analog network of analog components. Referring next to Figure 27C, in some implementations, the neural network topology includes (2724) one or more layers of neurons. Each layer of neurons computing respective outputs based on a respective mathematical function. In such cases, transforming the neural network topology to the equivalent analog network of analog components includes, performing (2726) a sequence of steps for each layer of the one or more layers of neurons. The sequence of steps include identifying (2728) one or more function blocks, based on the respective mathematical function, for the respective layer. Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function. In some implementations, identifying the one or more function blocks includes selecting (2730) the one or more function blocks based on a type of the respective layer. For example, a layer can consist of neurons, and the layer’s output is a linear superposition of its inputs. Selecting the one or more function blocks is based on this identification of a layer type, if a layer’s output is a linear superposition, or similar pattern identification. Some implementations determine if number of output > 1, then use either a trapezium or a pyramid transformation. [00356] Referring next to Figure 27D, in some implementations, the one or more function blocks include one or more basic function blocks (e.g., the basic function blocks 232) selected (2734) from the group consisting of: (i) a weighted summation block (2736) with a block output ^^^௨௧ = ^^ ^^ ^^ ^^൫ ^^ ^^^ . ^^^ ^^ + ^^ ^^ ^^ ^^൯. ^^ ^^ ^^ ^^ is Rectified Linear Unit (ReLU) activation function or a similar activation function (e.g., ReLU with a threshold), ^^^ represents an i-th input,
Figure imgf000086_0001
represents a weight corresponding to the i-th input, and ^^ ^^ ^^ ^^ represents a bias value, and ^^ is a summation operator; (ii) a signal multiplier block (2738) with a block output ^^^௨௧ = ^^ ^^ ^^ ^^ ^^. ^^^ . ^^^. ^^^ represents an i-th input and ^^^ represents a j-th input, and ^^ ^^ ^^ ^^ ^^ is a predetermined coefficient; (iii) a sigmoid activation block (2740) with a block output ^^^௨௧ = ^
Figure imgf000086_0002
represents an input, and A and B are predetermined coefficient values (e.g., A = - 0.1; B = 11.3) of the sigmoid activation block; (iv) a hyperbolic tangent activation block (2742) with a block output ^^^௨௧ = ^^ ∗ tanh ( ^^ ∗ ^^^^). ^^^^ represents an input, and A and B are predetermined coefficient values (e.g., A = 0.1, B = -10.1); and a signal delay block (2744) with a block output ^^( ^^) = ^^( ^^ − ^^ ^^). ^^ represents a current time-period, ^^( ^^ − 1) represents an output of the signal delay block for a preceding time period ^^ − 1, and ^^ ^^ is a delay value. [00357] Referring now back to Figure 27C, the sequence of steps also includes generating (2732) a respective multilayer network of analog neurons based on arranging the one or more function blocks. Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multilayer network is connected to one or more analog neurons of a second layer of the multilayer network. [00358] Referring now back to Figure 27A, for some networks, such as GRU and LSTM, transforming (2710) the neural network topology to an equivalent analog network of analog components requires more complex processing, according to some implementations. Referring next to Figure 27E, suppose the neural network topology includes (2746) one or more layers of neurons. Suppose further that each layer of neurons computes respective outputs based on a respective mathematical function. In such cases, transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing (2748) a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function. In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting (2750) the one or more weights such that combining the one or more intermediate functions results in the mathematical function.; and (ii) performing (2752) a sequence of steps for each sub-layer of the first layer of the neural network topology. The sequence of steps includes selecting (2754) one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and generating (2756) a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network. [00359] Referring next to Figure 27H, suppose the neural network topology includes (2768) one or more GRU or LSTM neurons. In that case, transforming the neural network topology includes generating (2770) one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons. In some implementations, an external cycle timer activates the one or more signal delay blocks with a constant time period (e.g., 1, 5, or 10 time steps). Some implementations use multiple delay blocks over one signal for producing additive time shift. In some implementations, the activation frequency of the one or more signal delay blocks is/are synchronized to network input signal frequency. In some implementations, the one or more signal delay blocks are activated (2772) at a frequency that matches a predetermined input signal frequency for the neural network topology. In some implementations, this predetermined input signal frequency may be dependent on the application, such as Human Activity Recognition (HAR) or PPG. For example, the predetermined input signal frequency is 30-60 Hz for video processing, around 100 Hz for HAR and PPG, 16 KHz for sound processing, and around 1-3 Hz for battery management. Some implementations activate different signal delay blocks activate at different frequencies. [00360] Referring next to Figure 27I, suppose the neural network topology includes (2774) one or more layers of neurons that perform unlimited activation functions. In some implementations, in such cases, transforming the neural network topology includes applying (2776) one or more transformations selected from the group consisting of: replacing (2778) the unlimited activation functions with limited activation (e.g., replacing ReLU with a threshold ReLU); and adjusting (2780) connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized. [00361] Referring now back to Figure 27A, the method also includes computing (2712) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. [00362] The method also includes generating (2714) a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. Referring next to Figure 27B, in some implementations, generating the schematic model includes generating (2716) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. In some implementations, the method includes regenerating just the resistance matrix for the resistors for a retrained network. In some implementations, the method further includes obtaining (2718) new weights for the trained neural network, computing (2720) a new weight matrix for the equivalent analog network based on the new weights, and generating (2722) a new resistance matrix for the new weight matrix. [00363] Referring next to Figure 27J, in some implementations, the method further includes generating (2782) one or more lithographic masks (e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix. In some implementations, the method includes regenerating just the masks for resistors (e.g., the masks 250) for retrained networks. In some implementations, the method further includes: (i) obtaining (2784) new weights for the trained neural network; (ii) computing (2786) a new weight matrix for the equivalent analog network based on the new weights; (iii) generating (2788) a new resistance matrix for the new weight matrix; and (iv) generating (2790) a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix. [00364] Referring now back to Figure 27G, the analog components include (2762) a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. Some implementations include other analog components, such as four-quadrant multipliers, sigmoid and hyperbolic tangent function circuits, delay lines, summers, and/or dividers. In some implementations, selecting (2764) component values of the analog components includes performing (2766) a gradient descent method and/or other weight quantization methods to identify possible resistance values for the plurality of resistors. [00365] Referring now back to Figure 27F, in some implementations, the method further includes implementing certain activation functions (e.g., Softmax) in output layer in digital. In some implementations, the method further includes generating (2758) equivalent digital network of digital components for one or more output layers of the neural network topology, and connecting (2760) output of one or more layers of the equivalent analog network to the equivalent digital network of digital components. Example Methods for Constrained Analog Hardware Realization of Neural Networks [00366] Figures 28A-28S show a flowchart of a method 28000 for hardware realization (28002) of neural networks according to hardware design constraints, according to some implementations The method is performed (28004) at the computing device 200 (e.g., using the neural network transformation module 226) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. The method includes obtaining (28006) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220). [00367] The method also includes calculating (28008) one or more connection constraints based on analog integrated circuit (IC) design constraints (e.g., the constraints 236). For example, IC design constraints can set the current limit (e.g., 1A), and neuron schematics and operational amplifier (OpAmp) design can set the OpAmp output current in the range [0- 10mA], so this limits output neuron connections to 100. This means that the neuron has 100 outputs which allow the current to flow to the next layer through 100 connections, but current at the output of the operational amplifier is limited to 10 mA, so some implementations use a maximum of 100 outputs (0.1 mA times 100 = 10 mA). Without this constraint, some implementations use current repeaters to increase number of outputs to more than 100, for example. [00368] The method also includes transforming (28010) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints. [00369] In some implementations, transforming the neural network topology includes deriving (28012) a possible input connection degree
Figure imgf000090_0001
and output connection degree ^^^, according to the one or more connection constraints. [00370] Referring next to Figure 28B, in some implementations, the neural network topology includes (28018) at least one densely connected layer with ^^ inputs (neurons in previous layer) and ^^ outputs (neurons in current layer) and a weight matrix ^^, and transforming (28020) the at least one densely connected layer includes constructing (28022) the equivalent sparsely connected network with ^^ inputs, ^^ outputs, and ^ ^^ ^^ ^^ே^ ^^^ + ^ ^^ ^^ ^^ே^ ^^^ − 1 layers, such that input connection degree does not exceed ^^^, and output connection degree does not exceed ^^^. [00371] Referring next to Figure 28C, in some implementations, the neural network topology includes (28024) at least one densely connected layer with ^^ inputs (neurons in previous layer) and ^^ outputs (neurons in current layer) and a weight matrix ^^, and transforming (28026) the at least one densely connected layer includes: constructing (28028) the equivalent sparsely connected network with ^^ inputs, ^^ outputs, and ^^ ≥ ^^ ^^ ^^൫⌈ ^^ ^^ ^^ே^ ^^⌉, ⌈ ^^ ^^ ^^ே^ ^^⌉൯ layers. Each layer m is represented by a corresponding weight matrix ^^^, where absent connections are represented with zeros, such that input connection degree does not exceed ^^^, and output connection degree does not exceed ^^^. The equation ^^ = ∏^ୀ^..ெ ^^^ is satisfied with a predetermined precision. The predetermined precision is a reasonable precision value that statistically guarantees that altered networks output differs from referent network output by no more than allowed error value, and this error value is task- dependent (typically between 0.1% and 1%). [00372] Referring next to Figure 28D, in some implementations, the neural network topology includes (28030) a single sparsely connected layer with ^^ inputs and ^^ outputs, a maximum input connection degree of ^^^, a maximum output connection degree of ^^^, and a weight matrix of ^^, where absent connections are represented with zeros. In such cases, transforming (28032) the single sparsely connected layer includes constructing (28034) the equivalent sparsely connected network with ^^ inputs, ^^ outputs, ^^ ≥ ^^ ^^ ^^൫⌈ ^^ ^^ ^^ே^ ^^^⌉, ⌈ ^^ ^^ ^^ே^ ^^^⌉൯ layers. Each layer ^^ is represented by a corresponding weight matrix ^^^, where absent connections are represented with zeros, such that input connection degree does not exceed ^^^, and output connection degree does not exceed ^^^, and the equation ^^ = ∏^ୀ^..ெ ^^^ is satisfied with a predetermined precision. [00373] Referring next to Figure 28E, in some implementations, the neural network topology includes (28036) a convolutional layer (e.g., a Depthwise convolutional layer, or a Separable convolutional layer) with ^^ inputs (neurons in previous layer) and ^^ outputs (neurons in current layer). In such cases, transforming (28038) the neural network topology to the equivalent sparsely connected network of analog components includes decomposing (28040) the convolutional layer into a single sparsely connected layer with ^^ inputs, ^^ outputs, a maximum input connection degree of ^^^, and a maximum output connection degree of ^^^,
Figure imgf000091_0001
[00374] Referring back to Figure 28A, the method also includes computing (28014) a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network. [00375] Referring now to Figure 28F, in some implementations, the neural network topology includes (28042) a recurrent neural layer, and transforming (28044) the neural network topology to the equivalent sparsely connected network of analog components includes transforming (28046) the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections. [00376] Referring next to Figure 28G, in some implementations, the neural network topology includes a recurrent neural layer (e.g., a long short-term memory (LSTM) layer or a gated recurrent unit (GRU) layer), and transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with ^^ inputs (neurons in previous layer) and ^^ outputs (neurons in current layer) and a weight matrix ^^, where absent connections are represented with zeros. [00377] Referring next to Figure 28H, in some implementations, the method includes performing a transformation of a single layer perceptron with one calculation neurons. In some implementations, the neural network topology includes (28054) K inputs, a weight vector ^^ ∈ ^^^, and a single layer perceptron with a calculation neuron with an activation function F. In such cases, transforming (28056) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving (28058) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating (28060) a number of layers m for the equivalent sparsely connected network using the equation ^^ = ⌈ ^^ ^^ ^^ே ^^⌉; and (iii) constructing (28062) the equivalent sparsely connected network with the K inputs, m layers and the connection degree N. The equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers. Each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron. Furthermore, in such cases, computing (28064) the weight matrix for the equivalent sparsely connected network includes calculating (28066) a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation ^^ =
Figure imgf000092_0001
[00378] Referring next to Figure 28I, in some implementations, the method includes performing a transformation of a single layer perceptron with L calculation neurons. In some implementations, the neural network topology includes (28068) K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons. In such cases, transforming (28070) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving (28072) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating (28074) number of layers m for the equivalent sparsely connected network using the equation ^^ = ⌈ ^^ ^^ ^^ே ^^⌉; (iii) decomposing (28076) the single layer perceptron into L single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network (28078) of the L single layer perceptron networks, constructing (28080) a respective equivalent pyramid- like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (v) constructing (28082) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs. Furthermore, in such cases, computing (28084) the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network (28086) of the L single layer perceptron networks, (i) setting (28088) a weight vector ^^ = ^^^, ith row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, and (ii) calculating (28090) a weight vector Wi for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation ^^ =
Figure imgf000093_0001
[00379] Referring next to Figure 28J, in some implementations, the method includes performing a transformation algorithm for multi-layer perceptron. In some implementations, the neural network topology includes (28092) K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons Li and corresponding weight matrices Vi that includes a row of weights for each calculation neuron of the Li calculation neurons. In such cases, transforming (28094) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving (28096) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing (28098) the multi-layer perceptron into ^^ =
Figure imgf000093_0002
single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the ^^ calculation neurons. Decomposing the multi-layer perceptron includes duplicating one or more input of the K inputs that are shared by the ^^ calculation neurons; (iii) for each single layer perceptron network (28100) of the Q single layer perceptron networks, (a) calculating (28102) a number of layers m for a respective equivalent pyramid-like sub-network using the equation ^^ = ^ ^^ ^^ ^^ே ^^^,^^. Ki, j is number of inputs for the respective calculation neuron in the multi-layer perceptron, and (b) constructing (28104) the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with Ki, j inputs, the m layers and the connection degree N. The equivalent pyramid- like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing (28106) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*Ki, j inputs. In such cases, computing (28108) the weight matrix for the equivalent sparsely connected network includes: for each single layer perceptron network (28110) of the Q single layer perceptron networks, (i) setting (28112) a weight vector ^^ =
Figure imgf000094_0001
the ith row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the multi-layer perceptron; and (ii) calculating (28114) a weight vector Wi for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes Ki, j equations with S variables, and S is computed using the equation
Figure imgf000094_0002
[00380] Referring next to Figure 28K, in some implementations, the neural network topology includes (28116) a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons Li and corresponding weight matrices Vi that includes a row of weights for each calculation neuron of the Li calculation neurons. In such cases, transforming (28118) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving (28120) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing (28122) the CNN into ^^ = ^^^ୀ^,ௌ( ^^^) single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the ^^ calculation neurons. Decomposing the CNN includes duplicating one or more input of the K inputs that are shared by the ^^ calculation neurons; (iii) for each single layer perceptron network of the Q single layer perceptron networks: (a) calculating number of layers m for a respective equivalent pyramid-like sub-network using the equation ^^ = ^ ^^ ^^ ^^ே ^^^,^^. j is the corresponding layer of the respective calculation neuron in the CNN, and Ki, j is number of inputs for the respective calculation neuron in the CNN; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with Ki, j inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m-1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing (28130) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub- network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*Ki, j inputs. In such cases, computing (28132) the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network (28134) of the Q single layer perceptron networks: (i) setting a weight vector ^^ = ^^^ ^ , the ith row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the CNN; and (ii) calculating weight vector Wi for connections of the respective equivalent pyramid-like sub- network by solving a system of equations based on the weight vector U. The system of equations includes Ki, j equations with S variables, and S is computed using the equation ^^ =
Figure imgf000095_0001
[00381] Referring next to Figure 28L, in some implementations, the method includes transforming two layers to trapezium-based network. In some implementations, the neural network topology includes (28140) K inputs, a layer Lp with K neurons, a layer Ln with L neurons, and a weight matrix ^^ ∈ ^^^×^, where R is the set of real numbers, each neuron of the layer Lp is connected to each neuron of the layer Ln, and each neuron of the layer Ln performs an activation function F, such that output of the layer Ln is computed using the equation ^^^ = ^^( ^^. ^^) for an input ^^. In such cases, transforming (28142) the neural network topology to the equivalent sparsely connected network of analog components includes performing a trapezium transformation that includes: (i) deriving (28144) a possible input connection degree ^^ூ > 1 and a possible output connection degree ^^ை > 1, according to the one or more connection constraints; and (ii) in accordance with a determination that ^^ ⋅ ^^ < ^^ ⋅ ^^ூ + ^^ ⋅ ^^ை, constructing (28146) a three-layered analog network that includes a layer LAp with K analog neurons performing identity activation function, a layer LAh with ^^ = analog neurons performing identity activation function, and a layer LAo
Figure imgf000095_0002
with L analog neurons performing the activation function F, such that each analog neuron in the layer LAp has NO outputs, each analog neuron in the layer LAh has not more than NI inputs and NO outputs, and each analog neuron in the layer LAo has NI inputs. In some such cases, computing (28148) the weight matrix for the equivalent sparsely connected network includes generating (2850) a sparse weight matrices ^^^ and ^^^ by solving a matrix equation ^^^. ^^^ = ^^ that includes ^^ ⋅ ^^ equations in ^^ ⋅ ^^ை + ^^ ⋅ ^^ூ variables, so that the total output of the layer LAo is calculated using the equation ^^^ = ^^( ^^^. ^^^. ^^). The sparse weight matrix ^^^ ∈ ^^^×ெ represents connections between the layers LAp and LAh, and the sparse weight matrix ^^^ ∈ ^^ெ×^ represents connections between the layers LAh and LAo,. [00382] Referring next to Figure 28M, in some implementations, performing the trapezium transformation further includes: in accordance with a determination that ^^ ⋅ ^^ ≥ ^^ ⋅ ^^ூ + ^^ ⋅ ^^ை: (i) splitting (28154) the layer Lp to obtain a sub-layer Lp1 with K’ neurons and a sub-layer Lp2 with (K - K’) neurons such that ^^′ ⋅ ^^ ≥ ^^ ⋅ ^^ூ + ^^′ ⋅ ^^ை; (ii) for the sub-layer Lp1 with K’ neurons, performing (28156) the constructing, and generating steps; and (iii) for the sub-layer Lp2 with K - K’ neurons, recursively performing (28158) the splitting, constructing, and generating steps. [00383] Referring next to Figure 28N, the method includes transforming multilayer perceptron to trapezium-based network. In some implementations, the neural network topology includes (28160) a multilayer perceptron network, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing (28162) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. [00384] Referring next to Figure 28O, the method includes transforming recurrent neural network to trapezium-based network. In some implementations, the neural network topology includes (28164) a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non- linear function calculation. In such cases, the method further includes performing (28166) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation. Element-wise addition is a common operation that can be implemented in networks of any structure, examples of which are provided above. Non-linear function calculation is a neuron-wise operation that is independent of the No and Ni restrictions, and are usually calculated with ‘sigmoid’ or ‘tanh’ block on each neuron separately. [00385] Referring next to Figure 28P, the neural network topology includes (28168) a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations (sigmoid and hyperbolic tangent operations). In such cases, the method further includes performing (28170) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations. Element-wise addition and Hadamard products are common operations that can be implemented in networks of any structure described above. [00386] Referring next to Figure 28Q, the neural network topology includes (28172) a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers (e.g., sequence of convolutional and pooling layers; each pooling layer is assumed to be a convolutional later with stride larger than 1) and (ii) one or more fully-connected layers (the sequence ends in the fully-connected layers). In such cases, the method further includes (i) transforming (28174) the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing (28176) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. [00387] Referring next to Figure 28R, the neural network topology includes (28178) K inputs, L output neurons, and a weight matrix ^^ ∈ ^^^×^, where R is the set of real numbers, each output neuron performs an activation function F. In such cases, transforming (28180) the neural network topology to the equivalent sparsely connected network of analog components includes performing an approximation transformation that includes: (i) deriving (28182) a possible input connection degree ^^ூ > 1 and a possible output connection degree ^^ை > 1, according to the one or more connection constraints; (ii) selecting (28184) a parameter p from the set {0, 1, … ,
Figure imgf000097_0001
− 1}; (iii) in accordance with a determination that p > 0, constructing (28186) a pyramid neural network that forms first p layers of the equivalent sparsely connected network, such that the pyramid neural network has ^^ ^ ^ = ^ ^^/ ^^ூ ^ neurons in its output layer. Each neuron in the pyramid neural network performs identity function; and (iv) constructing (28188) a trapezium neural network with ^^^ inputs and ^^ outputs. Each neuron in the last layer of the trapezium neural network performs the activation function F and all other neurons perform identity function. Also, in such cases, computing (28190) the weight matrix for the equivalent sparsely connected network includes: (i) generating (28192) weights for the pyramid neural network including (i) setting weights of every neuron i of the first layer of the pyramid neural network according to following rule: (a)
Figure imgf000097_0002
= ^^. ^^ is a non-zero constant and ^^^ = ( ^^ − 1 ) ^^ூ + for all weights j of the neuron
Figure imgf000097_0003
except k୧; and (ii) setting all other weights of the pyramid neural network to 1; and (ii) generating(28194) weights for the trapezium neural network including (i) setting weights of each neuron i of the first layer of the trapezium neural network (considering the whole net, this is (p+1)th layer) according to the equation ^^
Figure imgf000098_0001
^^ೖ^ ^^^ = ^ ; and (ii) setting other weights of the trapezium neural network to 1. [00388] Referring next to Figure 28S, in some implementations, the neural network topology includes (28196) a multilayer perceptron with the ^^ inputs, ^^ layers, and ^^^ୀ^,ௌ calculation neurons in i-th layer, and a weight matrix ^^^ୀ^,ௌ ∈ ^^^^×^^షభ for the i-th layer, where ^^^ = ^^. In such cases, transforming (28198) the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer ^^ (28200) of the ^^ layers of the multilayer perceptron, constructing (28202) a respective pyramid- trapezium network PTNNXj by performing the approximation transformation to a respective single layer perceptron consisting of ^^^ି^inputs, ^^^ output neurons, and a weight matrix ^^^; and (ii) constructing (28204) the equivalent sparsely connected network by stacking each pyramid trapezium network (e.g., output of a pyramid trapezium network PTNNXj-1 is set as an input for PTNNXj). [00389] Referring back to Figure 28A, In some implementations, the method further includes generating (28016) a schematic model for implementing the equivalent sparsely connected network utilizing the weight matrix. Example Methods of Calculating Resistance Values for Analog Hardware Realization of Trained Neural Networks [00390] Figures 29A-29F show a flowchart of a method 2900 for hardware realization (2902) of neural networks according to hardware design constraints, according to some implementations. The method is performed (2904) at the computing device 200 (e.g., using the weight quantization module 238) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. [00391] The method includes obtaining (2906) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220). In some implementations, weight quantization is performed during training. In some implementations, the trained neural network is trained (2908) so that each layer of the neural network topology has quantized weights (e.g., a particular value from a list of discrete values; e.g., each layer has only 3 weight values of +1, 0, -1). [00392] The method also includes transforming (2910) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. [00393] The method also includes computing (2912) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. [00394] The method also includes generating (2914) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. [00395] Referring next to Figure 29B, in some implementations, generating the resistance matrix for the weight matrix includes a simplified gradient-descent based iterative method to find a resistor set. In some implementations, generating the resistance matrix for the weight matrix includes: (i) obtaining (2916) a predetermined range of possible resistance values { ^^^^^, ^^^^௫} and selecting an initial base resistance value ^^^^^^ within the predetermined range. For example, the range and the base resistance are selected according to values of elements of the weight matrix; the values are determined by the manufacturing process; ranges - resistors that can be actually manufactured; large resistors are not preferred; quantization of what can be actually manufactured. In some implementations, the predetermined range of possible resistance values includes (2918) resistances according to nominal series E24 in the range 100 KΩ to 1 MΩ; (ii) selecting (2920) a limited length set of resistance values, within the predetermined range, that provide most uniform distribution of possible weights within the range [− ^^^^^^, ^^^^^^] for all combinations
Figure imgf000099_0001
of { ^^^, ^^^} within the limited length set of resistance values. In some implementations, weight values are outside this range, but the square average distance between weights within this range is minimum; (iii) selecting (2922) a resistance value ^^ା = ^^ି, from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias ^^^^௫ of each neuron or for each layer of the equivalent analog network, such that ^^ା = ^^ି is the closest resistor set value to ^^^^^^ ∗ ^^^^௫. In some implementations, ^^ା and ^^ି are chosen (2924) independently for each layer of the equivalent analog network. In some implementations, ^^ା and ^^ି are chosen (2926) independently for each analog neuron of the equivalent analog network; and (iv) for each element of the weight matrix, selecting (2928) a respective first resistance value ^^^ and a respective second resistance value ^^ଶ that minimizes an error according to equation all possible values of ^^^ and ^^ within the
Figure imgf000100_0001
ଶ predetermined range of possible resistance values. w is the respective element of the weight matrix, and ^^^^^ is a predetermined relative tolerance value for the possible resistance values. [00396] Referring next to Figure 29C, some implementations perform weight reduction. In some implementations, a first one or more weights of the weight matrix and a first one or more inputs represent (2930) one or more connections to a first operational amplifier of the equivalent analog network. The method further includes: prior to generating (2932) the resistance matrix, (i) modifying (2934) the first one or more weights by a first value (e.g., dividing the first one or more weights by the first value to reduce weight range, or multiplying the first one or more weights by the first value to increase weight range); and (ii) configuring (2936) the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function. Some implementations perform the weight reduction so as to change multiplication factor of one or more operational amplifiers. In some implementations, the resistor values set produce weights of some range, and in some parts of this range the error will be higher than in others. Suppose there are only 2 nominals (e.g., 1Ω and 4Ω), these resistors can produce weights [-3; -0.75; 0; 0.75; 3]. Suppose the first layer of a neural network has weights of {0, 9} and the second layer has weights of {0, 1}, some implementations divide the first layer’s weights by 3 and multiply the second layer’s weights by 3 to reduce overall error. Some implementations consider restricting weight values during training, by adjusting loss function (e.g., using l1 or l2 regularizer), so that resulting network does not have weights too large for the resistor set. [00397] Referring next to Figure 29D, the method further includes restricting weights to intervals. For example, the method further includes obtaining (2938) a predetermined range of weights, and updating (2940) the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input. [00398] Referring next to Figure 29E, the method further includes reducing weight sensitivity of network. For example, the method further includes retraining (2942) the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network. In other words, some implementations include additional training for an already trained neural network in order to give it less sensitivity to small randomly distributed weight errors. Quantization and resistor manufacturing produce small weight errors. Some implementations transform networks so that the resultant network is less sensitive to each particular weight value. In some implementations, this is performed by adding a small relative random value to each signal in at least some of the layers during training (e.g., similar to a dropout layer). [00399] Referring next to Figure 29F, some implementations include reducing weight distribution range. Some implementations include retraining (2944) the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. Some implementations perform this step via retraining. Example penalty function include a sum over all layers (e.g., A * max(abs(w)) / mean(abs(w)), where max and mean are calculated over a layer. Another example include order of magnitude higher and above. In some implementations, this function impacts weight quantization and network weight sensitivity. For e.g., small relative changes of weights due to quantization might cause high output error. Example techniques include introducing some penalty functions during training that penalize network when it has such weight outcasts. Example Methods of Optimizations for Analog Hardware Realization of Trained Neural Networks [00400] Figures 30A-30M show a flowchart of a method 3000 for hardware realization (3002) of neural networks according to hardware design constraints, according to some implementations. The method is performed (3004) at the computing device 200 (e.g., using the analog neural network optimization module 246) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. [00401] The method includes obtaining (3006) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220). [00402] The method also includes transforming (3008) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. [00403] Referring next to Figure 30L, in some implementations, the method further includes pruning the trained neural network. In some implementations, the method further includes pruning (3052) the trained neural network to update the neural network topology and the weights of the trained neural network, prior to transforming the neural network topology, using pruning techniques for neural networks, so that the equivalent analog network includes less than a predetermined number of analog components. In some implementations, the pruning is performed (3054) iteratively taking into account accuracy or a level of match in output between the trained neural network and the equivalent analog network. [00404] Referring next to Figure 30M, in some implementations, the method further includes, prior to transforming the neural network topology to the equivalent analog network, performing (3056) network knowledge extraction. Knowledge extraction is unlike stochastic/learning like pruning, but more deterministic than pruning. In some implementations, knowledge extraction is performed independent of the pruning step. In some implementations, prior to transforming the neural network topology to the equivalent analog network, connection weights are adjusted according to predetermined optimality criteria (such as preferring zero weights, or weights in a particular range, over other weights) through methods of knowledge extraction, by derivation of causal relationships between inputs and outputs of hidden neurons. Conceptually, in a single neuron or a set of neurons, on particular data set, there might be causal relationships between inputs and outputs which allows readjustment of weights in such a manner, that (1) new set of weights produces the same network output, and (2) new set of weights is easier to implement with resistors (e.g., more uniformly distributed values, more zero values or no connection). For example, if some neuron output is always 1 on some dataset, some implementations remove this neuron’s output connections (and the neuron as a whole), and instead adjust bias weight of the neurons following the neuron. In this way, knowledge extraction step is different to pruning, because pruning requires re-learning after removing a neuron, and learning is stochastic, while knowledge extraction is deterministic. [00405] Referring back to Figure 30A, the method also includes computing (3010) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. [00406] Referring next to Figure 30J, in some implementations, the method further includes removing or transforming neurons based on bias values. In some implementations, the method further includes, for each analog neuron of the equivalent analog network: (i) computing (3044) a respective bias value for the respective analog neuron based on the weights of the trained neural network, while computing the weight matrix; (ii) in accordance with a determination that the respective bias value is above a predetermined maximum bias threshold, removing (3046) the respective analog neuron from the equivalent analog network; and (iii) in accordance with a determination that the respective bias value is below a predetermined minimum bias threshold, replacing (3048) the respective analog neuron with a linear junction in the equivalent analog network. [00407] Referring next to Figure 30K, in some implementations, the method further includes minimizing number of neurons or compacting the network. In some implementations, the method further includes reducing (3050) number of neurons of the equivalent analog network, prior to generating the weight matrix, by increasing number of connections (inputs and outputs) from one or more analog neurons of the equivalent analog network. [00408] Referring back to Figure 30A, the method also includes generating (3012) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix. [00409] The method also includes pruning (3014) the equivalent analog network to reduce number of the plurality of operational amplifiers or the plurality of resistors, based on the resistance matrix, to obtain an optimized analog network of analog components. [00410] Referring next to Figure 30B, in some implementations, the method includes substituting insignificant resistances with conductors. In some implementations, pruning the equivalent analog network includes substituting (3016), with conductors, resistors corresponding to one or more elements of the resistance matrix that have resistance values below a predetermined minimum threshold resistance value. [00411] Referring next to Figure 30C, in some implementations, the method further includes removing connections with very high resistances. In some implementations, pruning the equivalent analog network includes removing (3018) one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value. [00412] Referring next to Figure 30D, in some implementations, pruning the equivalent analog network includes removing (3020) one or more connections of the equivalent analog network corresponding to one or more elements of the weight matrix that are approximately zero. In some implementations, pruning the equivalent analog network further includes removing (3022) one or more analog neurons of the equivalent analog network without any input connections. [00413] Referring next to Figure 30E, in some implementations, the method includes removing unimportant neurons. In some implementations, pruning the equivalent analog network includes (i) ranking (3024) analog neurons of the equivalent analog network based on detecting use of the analog neurons when making calculations for one or more data sets. For example, training data set used to train the trained neural network; typical data sets; data sets developed for pruning procedure. Some implementations perform ranking of neurons for pruning based on frequency of use of given neuron or block of neurons when subjected to training data set. For example, (a) if there is no signal at given neuron never, when using test data set - meaning this neuron or block of neurons was never in use and are pruned; (b) if the frequency of use of the neuron is very low, then the neuron is pruned without significant loss of accuracy; and (c) the neuron is always in use, then the neuron cannot be pruned); (ii) selecting (3026) one or more analog neurons of the equivalent analog network based on the ranking; and (iii) removing (3028) the one or more analog neurons from the equivalent analog network. [00414] Referring next to Figure 30F, in some implementations, detecting use of the analog neurons includes: (i) building (3030) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring (3032) propagation of analog signals (currents) by using the model (remove the blocks where the signal is not propagating when using special training sets) to generate calculations for the one or more data sets. [00415] Referring next to Figure 30G, in some implementations, detecting use of the analog neurons includes: (i) building (3034) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring (3036) output signals (currents or voltages) of the model (e.g., signals at outputs of some blocks or amplifiers in SPICe model or in real circuit, and deleting the areas where output signal for training set is always zero volts) by using the model to generate calculations for the one or more data sets. [00416] Referring next to Figure 30H, in some implementations, detecting use of the analog neurons includes: (i) building (3038) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring (3040) power consumed by the analog neurons (e.g., power consumed by certain neurons or blocks of neurons, represented by operational amplifiers either in a SPICE model or in real circuit and deleting the neurons or blocks of neurons which did not consume any power) by using the model to generate calculations for the one or more data sets. [00417] Referring next to Figure 30I, in some implementations, the method further includes, subsequent to pruning the equivalent analog network, and prior to generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network, recomputing (3042) the weight matrix for the equivalent analog network and updating the resistance matrix based on the recomputed weight matrix. Example Analog Neuromorphic Integrated Circuits and Fabrication Methods Example Methods for Fabricating Analog Integrated Circuits for Neural Networks [00418] Figures 31A-31Q show a flowchart of a method 3100 for fabricating an integrated circuit 3102 that includes an analog network of analog components, according to some implementations. The method is performed at the computing device 200 (e.g., using the IC fabrication module 258) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. The method includes obtaining (3104) a neural network topology and weights of a trained neural network. [00419] The method also includes transforming (3106) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (for recurrent neural networks, also use signal delay lines, multipliers, Tanh analog block, Sigmoid Analog Block). Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. [00420] The method also includes computing (3108) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. [00421] The method also includes generating (3110) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix. [00422] The method also includes generating (3112) one or more lithographic masks (e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix, and fabricating (3114) the circuit (e.g., the ICs 262) based on the one or more lithographic masks using a lithographic process. [00423] Referring next to Figure 31B, in some implementations, the integrated circuit further includes one or more digital to analog converters (3116) (e.g., the DAC converters 260) configured to generate analog input for the equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD/CMOS image sensors). [00424] Referring next to Figure 31C, in some implementations, the integrated circuit further includes an analog signal sampling module (3118) configured to process 1-dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit (number of inferences for the IC is determined by product Spec - we know sampling rate from Neural Network operation and exact task the chip is intended to solve). [00425] Referring next to Figure 31D, in some implementations, the integrated circuit further includes a voltage converter module (3120) to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers. [00426] Referring next to Figure 31E, in some implementations, the integrated circuit further includes a tact signal processing module (3122) configured to process one or more frames obtained from a CCD camera. [00427] Referring next to Figure 31F, in some implementations, the trained neural network is a long short-term memory (LSTM) network, AND the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing. [00428] Referring next to Figure 31G, in some implementations, the integrated circuit further includes one or more analog to digital converters (3126) (e.g., the ADC converters 260) configured to generate digital signal based on output of the equivalent analog network of analog components. [00429] Referring next to Figure 31H, in some implementations, the integrated circuit includes one or more signal processing modules (3128) configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications. [00430] Referring next to Figure 31I, the trained neural network is trained (3130), using training datasets containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected (in other words, the operation of trained chip is used to determine each of known to neural network gases in the gas mixture individually, despite the presence of other gases in the mixture). In some implementations, the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes (3132) 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers. In some implementations, the equivalent analog network includes (3134): (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections. [00431] Referring next to Figure 31J, the trained neural network is trained (3136), using training datasets containing thermal aging time series data for different MOSFETs (e.g., NASA MOSFET dataset that contains thermal aging time series for 42 different MOSFETs; data is sampled every 400 ms and typically several hours of data for each device), for predicting remaining useful life (RUL) of a MOSFET device. In some implementations, the neural network topology includes (3138) 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In some implementations, the equivalent analog network includes (3140): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons (e.g., 3137 analog neurons), and (v) between 123,000 and 124,000 connections (e.g., 123,200 connections). [00432] Referring next to Figure 31K, the trained neural network is trained (3142), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery ), for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS). In some implementations, the neural network topology includes (3144) an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values. The equivalent analog network includes (3146): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons (e.g., 1271 analog neurons), and (v) between 51,000 and 52,000 connections (e.g., 51,776 connections). [00433] Referring next to Figure 31L, the trained neural network is trained (3148), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery ), for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS). In some implementations, the neural network topology includes (3150) an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In some implementations, the equivalent analog network includes (3152): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons (e.g., 201 analog neurons), and (v) between 2,200 and 2,400 connections (e.g., 2,300 connections). [00434] Referring next to Figure 31M, the trained neural network is trained (3154), using training datasets containing speech commands (e.g., Google Speech Commands Dataset), for identifying voice commands (e.g., 10 short spoken keywords, including “yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”). In some implementations, the neural network topology is (3156) a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron. In some implementations, the equivalent analog network includes (3158): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections. [00435] Referring next to Figure 31N, the trained neural network is trained (3160), using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor (e.g., PPG data from the PPG-Dalia dataset (CHECK LICENSE). Data is collected for 15 individuals performing various physical activities during 1-4 hours each. Wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electrodermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from ECG sensor with sampling around 2 Hz. Original data was split into sequences of 1000 timesteps (around 15 seconds), with a shift of 500 timesteps, thus getting 16541 samples total. Dataset was split into 13233 training samples and 3308 test samples), for determining pulse rate during physical exercises (e.g., jogging, fitness exercises, climbing stairs) based on PPG sensor data and 3-axis accelerometer data. The neural network topology includes (3162) two Conv1D layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively. In some implementations, the equivalent analog network includes (3164): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons (e.g., 713 analog neurons), and (vi) between 12,000 and 12,500 connections (e.g., 12,072 connections). [00436] Referring next to Figure 31O, the trained neural network is trained (3166) to classify different objects (e.g., humans, cars, cyclists, scooters) based on pulsed Doppler radar signal (remove clutter and provide noise to Doppler radar signal), and the neural network topology includes (3168) multi-scale LSTM neural network. [00437] Referring next to Figure 31P, the trained neural network is trained (3170) to perform human activity type recognition (e.g., walking, running, sitting, climbing stairs, exercising, activity tracking), based on inertial sensor data (e.g., 3-axes accelerometers, magnetometers, or gyroscope data, from fitness tracking devices, smart watches or mobile phones; 3-axis accelerometer data as input, sampled at up to 96Hz frequency. Network was trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs” etc.). In some implementations, the neural network topology includes (3172) three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes. In some implementations, the equivalent analog network includes (3174): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons (e.g., 1296 analog neurons), and (vi) between 20,000 and 21,000 connections (e.g., 20,022 connections). [00438] Referring next to Figure 31Q, the trained neural network is further trained (3176) to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation (so as to detect pre-stroke or pre- heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy, etc). [00439] Some implementations include components that are not integrated into the chip (i.e., these are external elements, connected to the chip) selected from the group consisting of: voice recognition, video signal processing, image sensing, temperature sensing, pressure sensing, radar processing, LIDAR processing, battery management, MOSFET circuits current and voltage, accelerometers, gyroscopes, magnetic sensors, heart rate sensors, gas sensors, volume sensors, liquid level sensors, GPS satellite signal, human body conductance sensor, gas flow sensor, concentration sensor, pH meter, and IR vision sensors. [00440] Examples of analog neuromorphic integrated circuits manufactured according to the processes described above are provided in the following section, according to some implementations. Example Analog Neuromorphic IC for Selective Gas Detection [00441] In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC is based on a Deep Convolutional Neural Network trained for selective sensing of different gases in the gas mixture containing some amounts of gases to be detected. The Deep Convolutional Neural Network is trained using training datasets, containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) in response to different gas mixtures. The integrated circuit (or the chip manufactured according to the techniques described herein) can be used to determine one or more known gases in the gas mixture, despite the presence of other gases in the mixture. [00442] In some implementations, the trained neural network is a Multi-label 1D- DCNN network used for Mixture Gases Classification. In some implementations, the network is designed for detecting 3 binary gas components based on measurements by 16 gas sensors. In some implementations, the 1D-DCNN includes sensor-wise 1D convolutional block (16 such blocks), 3 common 1D convolutional blocks, and 3 Dense layers. In some implementations, the 1D-DCNN network performance for this task is 96.3%. [00443] In some implementations, the original network is T-transformed with following parameters: maximum input and output connections per neuron = 100; delay blocks could produce delay by any number of time steps; and signal limit of 5. [00444] In some implementations, the resulting T-network has the following properties: 15 layers, approximately 100,000 analog neurons, approximately 4,900,000 connections. Example Analog Neuromorphic IC for MOSFET failure prediction [00445] MOSFET on-resistance degradation due to thermal stress is a well-known serious problem in power electronics. In real-world applications, frequently, MOSFET device temperature changes over a short period of time. This temperature sweeps produce thermal degradation of a device, as a result of which the device might exhibit exponential. This effect is typically studied by power cycling that produces temperature gradients, which cause MOSFET degradation. [00446] In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC is based on a network discussed in the article titled “Real-time Deep Learning at the Edge for Scalable Reliability Modeling of SI- MOSFET Power Electronics Converters” for predicting remaining useful life (RUL) of a MOSFET device. The neural network can be used to determine Remaining Useful Life (RUL) of a device, with an accuracy over 80%. [00447] In some implementations, the network is trained on NASA MOSFET Dataset which contains thermal aging timeseries for 42 different MOSFETs. Data is sampled every 400 ms and typically includes several hours of data for each device. The network contains 4 LSTM layers of 64 neurons each, followed by 2 Dense layers of 64 and 1 neurons. [00448] In some implementations, the network is T-transformed with following parameters: maximum input and output connections per neuron is 100; signal limit of 5, and the resulting T-network had following properties: 18 layers, approximately 3,000 neurons (e.g., 137 neurons), and approximately 120,000 connections (e.g., 123200 connections). Example Analog Neuromorphic IC for Lithium Ion Battery Health and SoC Monitoring [00449] In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC can be used for predictive analytics of Lithium Ion batteries to use in Battery Management Systems (BMS). BMS device typically presents such functions as overcharge and over-discharge protection, monitoring State of Health (SOH) and State of Charge (SOC), and load balancing for several cells. SOH and SOC monitoring normally requires digital data processor, which adds to the cost of the device and consumes power. In some implementations, the Integration Circuit is used to obtain precise SOC and SOH data without implementing digital data processor on the device. In some implementations, the Integrated Circuit determines SOC with over 99% accuracy and determines SOH with over 98% accuracy. [00450] In some implementations, network operation is based on analysis of the discharge curve of the battery, as well as temperature, and/or data is presented as a time series. Some implementations use data from NASA Battery Usage dataset. The dataset presents data of continuous usage of 6 commercially available Li-Ion batteries. In some implementations, the network includes an input layer, 2 LSTM layers of 64 neurons each, and an output dense layer of 2 neurons (SOC and SOH values). [00451] In some implementations, the network is T-transformed with following parameters: maximum input and output connections per neuron = 100, and a signal limit of 5. In some implementations, the resulting T-network include the following properties: 9 layers, approximately 1,200 neurons (e.g., 1,271 neurons), and approximately 50,000 connections (e.g., 51,776 connections). In some implementations, the network operation is based on analysis of the discharge curve of the battery, as well as temperature. The network is trained using Network IndRnn disclosed in the paper titled “State-of-Health Estimation of Li-ion Batteries inElectric Vehicle Using IndRNN under VariableLoad Condition” designed for processing data from NASA Battery Usage dataset. The dataset presents data of continuous usage of 6 commercially available Li-Ion batteries. The IndRnn network contains an input layer with 18 neurons, a simple recurrent layer of 100 neurons and a dense layer of 1 neuron. [00452] In some implementations, the IndRnn network is T-transformed with following parameters: maximum input and output connections per neuron = 100 and signal limit of 5. In some implementations, the resulting T-network had following properties: 4 layers, approximately 200 neurons (e.g., 201 neurons), and approximately 2,000 connections (e.g., 2,300 connections). Some implementations output only SOH with an estimation error of 1.3%. In some implementations, the SOC is obtained similar to how the SOH is obtained. Example Analog Neuromorphic IC for Keyword spotting [00453] In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC can be used for keyword spotting. [00454] The input network is a neural network with 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio mel-spectrogram of size 49 times 10. In some implementations, the network includes 5 convolutional layers, 4 depthwise convolutional layers, an average pooling layer, and a final dense layer. [00455] In some implementations, the networks are pre-trained to recognize 10 short spoken keywords (yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go") from Google Speech Commands Dataset, with a recognition accuracy of 94.4%. [00456] In some implementations, the Integration Circuit is manufactured based on Depthwise Separable Convolutional Neural Network (DS-CNN) for the voice command identification. In some implementations, the original DS-CNN network is T-transformed with following parameters: maximum input and output connections per neuron = 100, signal limit of 5. In some implementations, the resulting T-network had following properties: 13 layers, approximately 72,000 neurons, and approximately 2.6 million connections. Example DS-CNN Keyword Spotting Network [00457] In one instance, a keyword spotting network is transformed to a T-network, according to some implementations. The network is a neural network of 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio spectrogram of size 49x10. Network consists of 5 convolutional layers, 4 depthwise convolutional layers, average pooling layer and final dense layer. Network is pre-trained to recognize 10 short spoken keywords (yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go") from Google Speech Commands Dataset https://ai.googleblog.com/2017/08/launching-speech-commands-dataset.html. There are 2 additional classes which correspond to ‘silence’ and ‘unknown’. Network output is a softmax of length 12. [00458] The trained neural network (input to the transformation) had a recognition accuracy of 94.4%, according to some implementations. In the neural network topology, each convolutional layer is followed with BatchNorm layer and ReLU layer, and ReLU activations are unbounded, and included around 2.5 million multiply-add operations. [00459] After transformation, the transformed analog network was tested with a test set of 1000 samples (100 of each spoken command). All test samples are also used as test samples in the original dataset. Original DS-CNN network gave close to 5.7% recognition error for this test set. Network was converted to a T-network of trivial neurons. Batch normalization (sometimes referred to as BatchNorm) layers in ‘test’ mode produce simple linear signal transformation, so can be interpreted as weight multiplier + some additional bias. Convolutional, AveragePooling and Dense layers are T-transformed quite straightforwardly. Softmax activation function was not implemented in T-network and was applied to T-network output separately. [00460] Resulting T-network had 12 layers including an Input layer, approximately 72,000 neurons and approximately 2.5 million connections. [00461] Figures 26A-26K show example histograms 2600 for absolute weights for the layers 1 through 11, respectively, according to some implementations. The weight distribution histogram (for absolute weights) was calculated for each layer. The dashed lines in the charts correspond to a mean absolute weight value for the respective layer. After conversion (i.e., T transformation), the average output absolute error (calculated over test set) of converted network vs original is calculated to be 4.1e-9. [00462] Various examples for setting network limitations for the transformed network are described herein, according to some implementations. For signal limit, as ReLU activations used in the network are unbounded, and some implementations use a signal limit on each layer. This could potentially affect mathematical equivalence. For this, some implementations use a signal limit of 5 on all layers which corresponds to power voltage of 5 in relation to input signal range. [00463] For quantizing the weights, some implementations use a nominal set of 30 resistors [0.001, 0.003, 0.01, 0.03, 0.1, 0.324, 0.353, 0.436, 0.508, 0.542, 0.544, 0.596, 0.73, 0.767, 0.914, 0.985, 0.989, 1.043, 1.101, 1.149, 1.157, 1.253, 1.329, 1.432, 1.501, 1.597, 1.896, 2.233, 2.582, 2.844]. [00464] Some implementations select R- and R+ values (see description above) separately for each layer. For each layer, some implementations select a value which delivers most weight accuracy. In some implementations, subsequently all the weights (including bias) in the T-network are quantized (e.g., set to the closest value which can be achieved with the input or chosen resistors). [00465] Some implementations convert the output layer as follows. Output layer is a dense layer that does not have ReLU activation. The layer has softmax activation which is not implemented in T-conversion and is left for digital part, according to some implementations. Some implementations perform no additional conversion. Example Analog Neuromorphic IC for Obtaining Heartrate [00466] PPG is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue. A PPG is often obtained by using a pulse oximeter which illuminates the skin and measures changes in light absorption. PPG is often processed to determine heart rate in devices, such as fitness trackers. Deriving heart rate (HR) from PPG signal is an essential task in edge devices computing. PPG data obtained from device located on wrist usually allows to obtain reliable heartrate only when the device is stable. If a person is involved in physical exercise, obtaining heartrate from PPG data produces poor results unless combined with inertial sensor data. [00467] In some implementations, an Integrated Circuit, based on combination of Convolutional Neural Network and LSTM layers, can be used to precisely determine the pulse rate, basing on the data from photoplethysmography (PPG) sensor and 3-axis accelerometer. The integrated circuit can be used to suppress motion artifacts of PPG data and to determine the pulse rate during physical exercise, such as jogging, fitness exercises, and climbing stairs, with an accuracy exceeding 90% [00468] In some implementations, the input network is trained with PPG data from the PPG-Dalia dataset. Data is collected for 15 individuals performing various physical activities for a predetermined duration (e.g., 1-4 hours each). The training data included wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electro-dermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from an ECG sensor with sampling around 2 Hz. The original data was split into sequences of 1000 time steps (around 15 seconds), with a shift of 500 time steps, thus producing 16541 samples total. The dataset was split into 13233 training samples and 3308 test samples. [00469] In some implementations, the input network included 2 Conv1D layers with 16 filters each, performing time series convolution, 2 LSTM layers of 16 neurons each, and 2 dense layers of 16 and 1 neurons. In some implementations, the network produces MSE error of less than 6 beats per minute over the test set. [00470] In some implementations, the network is T-transformed with following parameters: delay blocks could produce delay by any number of time steps, maximum input and output connections per neuron = 100, and signal limit of 5. In some implementations, the resulting T-network had following properties: 15 layers, approximately 700 neurons (e.g., 713 neurons), and approximately 12,000 connections (e.g., 12072 connections). Example Processing PPG data with T-converted LSTM Network [00471] As described above, for recurrent neurons, some implementations use signal delay block which is added to each recurrent connection of GRU and LSTM neurons. In some implementations, the delay block has an external cycle timer (e.g., a digital timer) which activates the delay block with a constant period of time dt. This activation produces an output of x(t-dt) where x(t) is input signal of delay block. Such activation frequency can, for instance, correspond to network input signal frequency (e.g., output frequency of analog sensors processed by a T-converted network). Typically, all delay blocks are activated simultaneously with the same activation signal. Some blocks can be activated simultaneously on one frequency, and other blocks can be activated on another frequency. In some implementations, these frequencies have common multiplier, and signals are synchronized. In some implementations, multiple delay blocks are used over one signal producing additive time shift. Examples of delay blocks are described above in reference to Figure 13B shows two examples of delay blocks, according to some implementations. [00472] The network for processing PPG data uses one or more LSTM neurons, according to some implementations. Examples of LSTM neuron implementations are described above in reference to Figure 13A, according to some implementations. [00473] The network also uses Conv1D, a convolution performed over time coordinate. Examples of Conv1D implementations are described above in reference to Figures 15A and 15B, according to some implementations. [00474] Details of PPG data are described herein, according to some implementations. PPG is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue. A PPG is often obtained by using a pulse oximeter which illuminates the skin and measures changes in light absorption. PPG is often processed to determine heart rate in devices such as fitness trackers. Deriving heart rate (HR) from PPG signal is an essential task in edge devices computing. [00475] Some implementations use PPG data from the Capnobase PPG dataset. The data contains raw PPG signal for 42 individuals of 8 min duration each, sampling 300 samples per second, and a reference heartrate data obtained from ECG sensor with sampling around 1 sample per second. For training and evaluation, some implementations split the original data into sequences of 6000 time steps, with a shift of 1000 time steps, thus getting a total set of 5838 samples total. [00476] In some implementations, the input trained neural network NN-based allows for 1-3% accuracy in obtaining heartrate (HR) from PPG data. [00477] This section describes a relatively simple neural network in order to demonstrate how T-conversion and analog processing can deal with this task. This description is provided as an example, according to some implementations. [00478] In some implementations, dataset is split into 4,670 training samples and 1,168 test samples. The network included: 1 Conv1D layer with 16 filters and kernel of 20, 2 LSTM layers with 24 neurons each, 2 dense layers (with 24 and 1 neurons each). In some implementations, after training this network for 200 epochs, test accuracy was found to be 2.1%. [00479] In some implementations, the input network was T-transformed with following parameters: delay block with periods of 1, 5 and 10 time steps, and the following properties: 17 layers, 15,448 connections, and 329 neurons (OP3 neurons and multiplier blocks, not counting delay blocks). Example Analog Neuromorphic Integrated Circuit for Object Recognition Based on Pulsed Doppler Radar Signal [00480] In some implementations, an Integration Circuit is manufactured, based on a multi-scale LSTM neural network, that can be used to classify the objects, based on pulse Doppler Radar signal. The IC can be used to classify different objects, like humans, cars, cyclists, scooters, based on Doppler radar signal, removes clutter, and provides the noise to Doppler radar signal. In some implementations, the accuracy of classification of object with multi-scale LSTM network exceeded 90%. Example Analog Neuromorphic IC for Human Activity Type Recognition Based on Inertial Sensor Data [00481] In some implementations, a neuromorphic Integrated Circuit is manufactured, and can be used for human activity type recognition based on multi-channel convolutional neural networks, which have input signals from 3-axes accelerometers and possibly magnetometers and/or gyroscopes of fitness tracking devices, smart watches or mobile phones. The multi-channel convolutional neural network can be used to distinguish between different types of human activities, such as walking, running, sitting, climbing stairs, exercising and can be used for activity tracking. The IC can be used for detection of abnormal patterns of human activity, based on accelerometer data, convolutionally merged with heart rate data. Such IC can detect pre-stroke or pre heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy and others, according to some implementations. [00482] In some implementations, the IC is based on a channel-wise 1D convolutional network discussed in the article “Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors.” In some implementations, this network accepts 3-axis accelerometer data as input, sampled at up to 96Hz frequency. In some implementations, the network is trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs,” etc. In some implementations, the network included 3 channel- wise Conv networks with Conv layer of 12 filters and kernel of 64, followed by MaxPooling(4) layer each, and 2 common Dense layers of 1024 and N neurons respectively, where N is a number of classes. In some implementations, the activity classification was performed with a low error rate (e.g., 3.12% error). [00483] In some implementations, the network is T-transformed with following parameters: delay blocks could produce delay by any number of time steps, maximum input and output connections per neuron = 100, an output layer of 10 neurons, and a signal limit of 5. In some implementations, the resulting T-network had following properties: 10 layers, approximately 1,200 neurons (e.g., 1296 neurons), and approximately 20,000 connections (e.g., 20022 connections). Example Transformation of Modular Net Structure for Generating Libraries [00484] A modular structure of converted neural networks is described herein, according to some implementations. Each module of a modular type neural network is obtained after transformation of (a whole or a part of) one or more trained neural network. In some implementations, the one or more trained neural networks is subdivided into parts, and then subsequently transformed into an equivalent analog network. Modular structure is typical for some of the currently used neural networks, and modular division of neural networks corresponds to a trend in neural network development. Each module can have an arbitrary number of inputs or connections of input neurons to output neurons of a connected module, and an arbitrary number of outputs connected to input layers of a subsequent module. In some implementations, a library of preliminary (or a seed list of) transformed modules is developed, including lithographic masks for manufacture of each module. A final chip design is obtained as a combination of (or by connecting) preliminary developed modules. Some implementations perform commutation between the modules. In some implementations, the neurons and connections within the module are translated into chip design using ready-made module design templates. This significantly simplifies the manufacture of the chip, accomplished by just connecting corresponding modules. [00485] Some implementations generate libraries of ready-made T-converted neural networks and/or T-converted modules. For example, a layer of CNN network is a modular building block, LSTM chain is another building block, etc. Larger neural networks NNs also have modular structure (e.g., LSTM module and CNN module). In some implementations, libraries of neural networks are more than by-products of the example processes, and can be sold independently. For example, a third-party can manufacture a neural network starting with the analog circuits, schematics, or designs in the library (e.g., using CADENCE circuits, files and/or lithography masks). Some implementations generate T-converted neural networks (e.g., networks transformable to CADENCE or similar software) for typical neural networks, and the converted neural networks (or the associated information) are sold to a third-party. In some instances, a third-party chooses not to disclose structure and/or purpose of the initial neural network, but uses the conversion software (e.g., SDK described above) to converts the initial network into trapezia-like networks and passes the transformed networks to a manufacturer to the fabricate the transformed network, with a matrix of weights obtained using one of the processes described above, according to some implementations. As another example, where the library of ready-made networks are generated according to the processes described herein, corresponding lithographic masks are generated and a customer can train one of the available network architectures for his task, perform lossless transformation (sometimes called T transformation) and provide the weights to a manufacturer for fabricating a chip for the trained neural networks. [00486] In some implementations, the modular structure concept is also used in the manufacture of multi-chip systems or the multi-level 3D chips, where each layer of the 3D chip represents one module. The connections of outputs of modules to the inputs of connected modules in case of 3D chips will be made by standard interconnects that provide ohmic contacts of different layers in multi-layer 3D chip systems. In some implementations, the analog outputs of certain modules is connected to analog inputs of connected modules through interlayer interconnects. In some implementations, the modular structure is used to make multi-chip processor systems as well. A distinctive feature of such multi-chip assemblies is the analog signal data lines between different chips. The analog commutation schemes, typical for compressing several analog signals into one data line and corresponding de-commutation of analog signals at receiver chip, is accomplished using standard schemes of analog signal commutation and de-commutation, developed in analog circuitry. [00487] One main advantage of a chip manufactured according to the techniques described above, is that analog signal propagation can be broadened to multi-layer chips or multi-chip assemblies, where all signal interconnects and data lines transfer analog signals, without a need for analog-to-digital or digital-to-analog conversion. In this way, the analog signal transfer and processing can be extended to 3D multi-layer chips or multi-chip assemblies. Example Methods for Generating Libraries for Hardware Realization of Neural Networks [00488] Figures 32A-32E show a flowchart of a method 3200 for generating (3202) libraries for hardware realization of neural networks, according to some implementations. The method is performed (3204) at the computing device 200 (e.g., using the library generation module 254) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. [00489] The method includes obtaining (3206) a plurality of neural network topologies (e.g., the topologies 224), each neural network topology corresponding to a respective neural network (e.g., a neural network 220). [00490] The method also includes transforming (3208) each neural network topology (e.g., using the neural network transformation module 226) to a respective equivalent analog network of analog components. [00491] Referring next to Figure 32D, in some implementations, transforming (3230) a respective network topology to a respective equivalent analog network includes: (i) decomposing (3232) the respective network topology to a plurality of subnetwork topologies. In some implementations, decomposing the respective network topology includes identifying (3234) one or more layers (e.g., LSTM layer, fully connected layer) of the respective network topology as the plurality of subnetwork topologies; (ii) transforming (3236) each subnetwork topology to a respective equivalent analog subnetwork of analog components; and (iii) composing (3238) each equivalent analog subnetwork to obtain the respective equivalent analog network. [00492] Referring back to Figure 32A, the method also includes generating (3210) a plurality of lithographic masks (e.g., the masks 256) for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components. [00493] Referring next to Figure 32E, in some implementations, each circuit is obtained by: (i) generating (3240) schematics for a respective equivalent analog network of analog components; and (ii) generating (3242) a respective circuit layout design based on the schematics (using special software, e.g., CADENCE). In some implementations, the method further includes combining (3244) one or more circuit layout designs prior to generating the plurality of lithographic masks for fabricating the plurality of circuits. [00494] Referring next to Figure 32B, in some implementations, the method further includes: (i) obtaining (3212) a new neural network topology and weights of a trained neural network; (ii) selecting (3214) one or more lithographic masks from the plurality of lithographic masks based on comparing the new neural network topology to the plurality of neural network topologies. In some implementations, the new neural network topology includes a plurality of subnetwork topologies, and selecting the one or more lithographic masks is further based on comparing (3216) each subnetwork topology with each network topology of the plurality of network topologies; (iii) computing (3218) a weight matrix for a new equivalent analog network based on the weights; (iv) generating (3220) a resistance matrix for the weight matrix; and (v) generating (3222) a new lithographic mask for fabricating a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithographic masks. [00495] Referring next to Figure 32C, one or more subnetwork topologies of the plurality of subnetwork topologies fails to compare (3224) with any network topology of the plurality of network topologies, and the method further includes: (i) transforming (3226) each subnetwork topology of the one or more subnetwork topologies to a respective equivalent analog subnetwork of analog components; and generating (3228) one or more lithographic masks for fabricating one or more circuits, each circuit of the one or more circuits implementing a respective equivalent analog subnetwork of analog components. Example Methods for Optimizing Energy Efficiency of Neuromorphic Analog Integrated Circuits [00496] Figures 33A-33J show a flowchart of a method 3300 for optimizing (3302) energy efficiency of analog neuromorphic circuits (that model trained neural networks), according to some implementations. The method is performed (3204) at the computing device 200 (e.g., using the energy efficiency optimization module 264) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202. [00497] The method includes obtaining (3306) an integrated circuit (e.g., the ICs 262) implementing an analog network (e.g., the transformed analog neural network 228) of analog components including a plurality of operational amplifiers and a plurality of resistors. The analog network represents a trained neural network (e.g., the neural networks 220), each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. [00498] The method also includes generating (3308) inferences (e.g., using the inferencing module 266) using the integrated circuit for a plurality of test inputs, including simultaneously transferring signals from one layer to a subsequent layer of the analog network. In some implementations, the analog network has layered structure, with the signals simultaneously coming from previous layer to the next one. During inference process, the signals propagate through the circuit layer by layer; simulation at device level; time delays every minute. [00499] The method also includes, while generating inferences using the integrated circuit, determining (3310) if a level of signal output of the plurality of operational amplifiers is equilibrated (e.g., using the signal monitoring module 268). Operational amplifiers go through a transient period (e.g., a period that lasts less than 1 millisecond from transient to plateau signal) after receiving inputs, after which the level of signal is equilibrated and does not change. In accordance with a determination that the level of signal output is equilibrated, the method also includes: (i) determining (3312) an active set of analog neurons of the analog network influencing signal formation for propagation of signals. The active set of neurons need not be part of a layer/layers. In other words, the determination step works regardless of whether the analog network includes layers of neurons; and (ii) turning off power (3314) (e.g., using the power optimization module 270) for one or more analog neurons of the analog network, distinct from the active set of analog neurons, for a predetermined period of time. For example, some implementations switch off power (e.g., using the power optimization module 270) of operational amplifiers which are in layers behind an active layer (to where signal propagated at the moment), and which do not influence the signal formation on the active layer. This can be calculated based on RC delays of signal propagation through the IC. So all the layers behind the operational one (or the active layer) are switched off to save power. So the propagation of signals through the chip is like surfing - the wave of signal formation propagate through chip, and all layers which are not influencing signal formation are switched off. In some implementations, for layer-by-layer networks, signal propagates layer to layer, and the method further includes decreasing power consumption before a layer corresponding to the active set of neurons because there is no need for amplification before the layer. [00500] Referring next to Figure 33B, in some implementations, in some implementations, determining the active set of analog neurons is based on calculating (3316) delays of signal propagation through the analog network. Referring next to Figure 33C, in some implementations, determining the active set of analog neurons is based on detecting (3318) the propagation of signals through the analog network. [00501] Referring next to Figure 33D, in some implementations, the trained neural network is a feed-forward neural network, and the active set of analog neurons belong to an active layer of the analog network, and turning off power includes turning off power (3320) for one or more layers prior to the active layer of the analog network. [00502] Referring next to Figure 33E, in some implementations, the predetermined period of time is calculated (3322) based on simulating propagation of signals through the analog network, accounting for signal delays (using special software, e.g., CADENCE). [00503] Referring next to Figure 33F, in some implementations, the trained neural network is (3324) a recurrent neural network (RNN), and the analog network further includes one or more analog components other than the plurality of operational amplifiers, and the plurality of resistors. In such cases, the method further includes, in accordance with a determination that the level of signal output is equilibrated, turning off power (3326) (e.g., using the power optimization module 270), for the one or more analog components, for the predetermined period of time. [00504] Referring next to Figure 33G, in some implementations, the method further includes turning on power (3328) ) (e.g., using the power optimization module 270) for the one or more analog neurons of the analog network after the predetermined period of time. [00505] Referring next to Figure 33H, in some implementations, determining if the level of signal output of the plurality of operational amplifiers is equilibrated is based on detecting (3330) if one or more operational amplifiers of the analog network is outputting more than a predetermined threshold signal level (e.g., power, current, or voltage). [00506] Referring next to Figure 33I, in some implementations, the method further includes repeating (3332) ) (e.g., by the power optimization module 270) the turning off for the predetermined period of time and turning on the active set of analog neurons for the predetermined period of time, while generating the inferences. [00507] Referring next to Figure 33J, in some implementations, the method further includes, in accordance with a determination that the level of signal output is equilibrated, for each inference cycle (3334): (i) during a first time interval, determining (3336) a first layer of analog neurons of the analog network influencing signal formation for propagation of signals; and (ii) turning off power (3338) ) (e.g., using the power optimization module 270) for a first one or more analog neurons of the analog network, prior to the first layer, for the predetermined period of time; and during a second time interval subsequent to the first time interval, turning off power (3340) ) (e.g., using the power optimization module 270) for a second one or more analog neurons including the first layer of analog neurons and the first one or more analog neurons of the analog network, for the predetermined period. [00508] Referring next to Figure 33K, in some implementations, the one or more analog neurons consist (3342) of analog neurons of a first one or more layers of the analog network, and the active set of analog neurons consist of analog neurons of a second layer of the analog network, and the second layer of the analog network is distinct from layers of the first one or more layers. [00509] Some implementations include means for delaying and/or controlling signal propagation from layer to layer of the resulting hardware-implemented neural network. Example Transformation of MobileNet v.1 [00510] An example transformation of MobileNet v.1 into an equivalent analog network is described herein, according to some implementations. In some implementations, single analog neurons are generated, then converted into SPICE schematics with a transformation of weights from MobileNet into resistor values. MobileNet v1 architecture is depicted in the Table shown in Figure 34. In the Table, the first column 3402 corresponds to type of layer and stride, the second column 3404 corresponds to filter shape for the corresponding layer, and the third column 3406 corresponds to input size for the corresponding layer. In MobileNet v.1, each convolutional layer is followed by a batch normalization layer and a ReLU 6 activation function (y = max(0, min(6, x)). The network consists of 27 convolutional layers, 1 dense layer, and has around 600 million multiply-accumulate operations for a 224x224x3 input image. Output values are the result of softmax activation function which means the values are distributed in the range [0, 1] and the sum is 1. Some implementations accept as input MobileNet 32x32 and alpha = 1 for the transformation. In some implementations, the network is pre-trained for CIFAR-10 task (50,000 32x32x3 images divided into 10 non-intersecting classes). Batch normalization layers operate in ‘test’ mode to produce simple linear signal transformation, so the layers are interpreted as weight multiplier + some additional bias. Convolutional, AveragePooling and Dense layers are transformed using the techniques described above, according to some implementations. In some implementations, Softmax activation function is not implemented in transformed network but applied to output of the transformed network (or the equivalent analog network) separately. [00511] In some implementations, the resulting transformed network included 30 layers including an input layer, approximately 104,000 analog neurons, and approximately 11 million connections. After transformation, the average output absolute error (calculated over 100 random samples) of transformed network versus MobileNet v.1 was 4.9e-8. [00512] As every convolutional and other layers of MobileNet have an activation function ReLU6, the output signal on each layer of the transformed network is also limited by the value 6. As part of the transformation, the weights are brought into accordance with a resistor nominal set. Under each nominal set, different weight values are possible. Some implementations use resistor nominal sets e24, e48 and e96, within the range of [0.1 - 1] Mega Ohm. Given that the weight ranges for each layer vary, and for most layers weight values do not exceed 1-2, in order to achieve more weight accuracy, some implementations decrease R- and R+ values. In some implementations, the R- and R+ values are chosen separately for each layer from the set [0.05, 0.1, 0.2, 0.5, 1] Mega Ohm. In some implementations, for each layer, a value which delivers most weight accuracy is chosen. Then all the weights (including bias) in the transformed network are ‘quantized’, i.e., set to the closest value which can be achieved with used resistors. In some implementations, this reduced transformed network accuracy versus original MobileNet according to the Table shown below. The Table shows mean square error of transformed network, when using different resistor sets, according to some implementations.
Figure imgf000126_0001
Example Analog Hardware Realization of Trained Neural Networks for Voice Clarity [00513] Some implementations provide a method for fabricating a neuromorphic Integrated Circuit for voice clarification, using techniques described above. Various types of trained neural networks can be used for this purpose. For example, a neural network can be trained to identify only one voice, suppressing and removing everything else. In particular, the neural network can identify the voice that is the closest to the microphone. As another example, a neural network can be trained to identify several voices, suppressing and removing everything else. Voices can be identified and preserved regardless of their distance from the microphone(s). Alternatively, voices can be prioritized by their distances from the microphone(s) and given different weights in the output signal, based on their respective distances form the microphone. As another alternative, voices can be identified and preserved regardless of their relative strength (e.g., volume). As yet another alternative, voices can be prioritized by their relative strength and be given different weights in the output signal, based on their respective relative strength. A neural network can process the signal that is originating from the microphone(s). Such a signal may include analog and/or digital signals. A neural network can process an analog and/or a digital signal that is transmitted over a transmission media and received by the neural network. Such a signal can be transmitted across wireless or digital/internet networks for the purposes of phone communication. Such a signal can also be input after pre- and post-processing of the original voice(s), either before the signal is ready to be transmitted, or after the signal has been transmitted and delivered to the recipient. As another example, a neural network can process a signal that is a mix of several voice signals, with associated noises. In particular, such a mix can be delivered to the recipient from several different sources. Such a signal can be pre- and post-processed by different methods for different components. As yet another example, a neural network can process the signal that is a mix of several external voice signals, with associated noises, combined with the own voice(s) on the recipient side. In particular, such a mix can be delivered to the recipient from several different sources, including the recipient’s own voice overlapped with recipient’s own noises. Such a signal can be pre- and/or post-processed by different methods for different components. The clarification of voice(s) can be performed for the combined signal. As another example, a neural network can process a signal that includes voice(s) from the recipient side. In particular, such a signal can be processed before it is transmitted to the other party. Such a signal can be processed by the neural network before it is pre- and/or post-processed by different methods prior to transmission. Example Methods for Extracting Voice from Inbound or Outbound Analog Noisy Signal [00514] Described herein are example techniques for the extraction of voice from a noisy signal, both inbound and outbound, where noise can be either stationary or non- stationary, using a neuromorphic analog Integrated Circuit. Such a circuit implements a noise suppression neural network at the hardware level. The circuit design of the analog neuromorphic Integrated Circuit is realized by converting (using techniques described above) a noise suppression (or voice extraction) neural network. [00515] As described in the Background and Summary sections, the task of extracting the voice from noisy signal is of great importance for communication in smartphones, smartwatches, notebooks, or other voice transmitting devices. There are conventional realizations of noise cancellation or active noise suppression using dual microphone scheme, where the signal from one microphone is used to cancel noise at a main microphone. But these solutions do not cancel all noises, especially non-stationary ones, since not all noise is canceled in such combination of two microphones. There are also filters, which can filter out stationary noise from inbound or outbound analog signal. There are also software realizations of neural networks, which extract voice from a noisy signal by converting some part of the signal using Fourier transformation, thereby reducing components that are not similar to voice. These products are realized as software applications, which can be installed on smartphones or notebook computers, and can effectively suppress noise coming from a microphone. However, such applications require high computational power and consequently lead to higher power consumption. Also, such applications require powerful processors, which cannot be installed in earbuds or other miniature devices. [00516] Described herein are techniques for voice extraction using a specially designed Integrated Circuit, realized from a trained neural network. The Integrated Circuit is realized as a hardware solution and is represented by a set of operational amplifiers and resistors, connected in such a way that the resulting neuromorphic hardware chip operates similarly to the initial neural network (e.g., the neural network realized in software), with the absolute error not exceeding a maximum threshold percentage (e.g., 1% absolute) from the error corresponding to the software neural network. The schematics of the Integration Circuit are obtained using techniques described above, thus ensuring full equivalency of analog neuromorphic hardware realization of the neural network and its initial software neural network model. The analog Integrated Circuit may be used for voice extraction from noisy analog inbound or outbound signals, with low latency and low power consumption. [00517] In some implementations, the hardware realization of a voice extraction neural network can be used to process both inbound and outbound noisy signals. In some implementations, the Integrated Circuit has direct analog input and is placed adjacent to a microphone or a speaker of a smartphone, smartwatch, earbuds, notebook computer, or similar device. The Integrated Circuit provides telecommunication voice transfer, extracting voice from noisy analog signals. Such a solution suppresses both stationary and non-stationary noise from inbound or outbound analog signals (e.g., signals from a microphone or signals directed to a speaker or earbuds) and is characterized by excellent noise suppression, unlike conventional methods. [00518] The resulting hardware realization of a voice extraction algorithm is characterized by low power operation, small latency, and small die area, which makes analog hardware realization an advantageous solution for noise reduction in smartphones, earbuds, notebook computers, tablets, or other voice transmitting devices, in comparison with software neural network voice extraction algorithms. The small die area makes it possible to include the Integrated Circuit application in true wireless (TWS) earbuds or other miniature devices. Such analog Integrated Circuits may also be used for two-way voice extraction (noise reduction) in Notebook PCs or Smartphones, where a neuromorphic analog integration circuit is installed both at the analog output of the microphone and at the analog input of the speaker or earbuds. Example Neuromorphic Analog Integrated Circuit for Voice Clarity [00519] Some implementations obtain a convolutional neural network with 1D convolutions (e.g., as described in “Single Channel Speech Enhancement Using A Convolutional Neural Network,” by T. Kounovsky and J. Malek, 2017), an example of which is shown in Figure 35. The architecture 3500 shown in Figure 35 performs Fourier transformation of incoming analog signal, to obtain input features 3502 that form a network input 3504. Subsequently, the architecture uses convolution 3506, maxpooling 3508, convolution 3510 and fully connected layers (layers 3512 and 3514), to obtain output 3516. An inverse Fourier transformation is applied on the output 3516 to obtain an analog output signal. Some implementations convert this example network 3500 into a network of analog components using techniques described above and herein. Some implementations apply the techniques described above for fabricating a neuromorphic analog integrated circuit based on the network of analog components. Simulations have shown the resulting integrated circuit occupied a 30 square millimeter die area, consumed approximately 150 micro-Watts of power, and had a signal latency of 3 milliseconds. The resulting integrated circuit can be used for inbound and outbound voice extraction for smartphones, earbuds, notebook computers, smartwatches, or other telecommunication devices. In experiments, when the quality is measured according to PESQ criteria, an improvement of voice signal up to 30% was accomplished. The small power consumption allows the use of such integrated circuits in battery-powered devices. The small die area allows installing the device into TWS earbuds or other miniature devices. [00520] The network architecture shown in Figure 35 includes convolution, max pooling, and fully-connected layers. Transformation of convolutional layers is described above, according to some implementations. The following sections describe example transformation techniques for various components of the network shown in Figure 35, according to some implementations. Example Transformation of Fully Connected (Dense) Layers [00521] Figure 36 shows an example transformation 3600 of a fully-connected or dense layer, according to some implementations. A dense layer of a neural network is represented using the formula 3620, ^^ ^^ ^^ ^^ ^^ ^^ = ^^( ^^ ^^ ^^ ^^ ^^ ⋅ ^^ + ^^), where the Input 3602 (corresponding to Input 13604, Input 23606, …, Input i 3608) and Output 3612 (corresponding to SNM 13614, SNM 23616, …, SNM j 3618) are vectors. W is a weight matrix 3610, and each input is connected to every output. Each weight W[i][j] corresponds to an edge or connection between the Input i and the Output j. Bias b is a bias vector and f(x) is an element- wise activation function, typically ReLU. This mathematical formula is transformed into a 2- layer fully connected mesh of SNMs using techniques described above, where the first layer represents the Input and the second layer represents the Output, according to some implementations. Example Transformation of Conv1D Layers [00522] Figure 37 shows example transformations 3700 of a Conv1D layer of a Convolutional Neural Network, according to some implementations. The Conv1D layer of a neural network is a convolution applied over a 1-dimensional (e.g., the X dimension 3708) sequence of data (shown as input 3702, which may include multiple input channels 3712) to obtain output 3704 (with output channels 3714). Each element of an output tensor is a product of a particular subset of the input data 3702 and a slice of a weight tensor called the kernel 3706 (sometimes called a filter kernel or a filter). For T-transforming the Conv1D layer, some implementations evaluate the output tensor shape and construct a layer of SNMs (e.g., the layer 3710) with the same shape (the number of SNMs equals the product of the tensor dimensions). Each of these SNMs has incoming connections weighted according to a relevant slice of the weight tensor and is connected to a relevant subset of input SNMs of the previous layer. The Output-to-Input SNMs relations may depend on Conv1D layer parameters, such as kernel size, stride, padding, and filters. The shape of the output layer SNMs is the same as output tensor shape in mathematical representation, where each SNM has a number of incoming weighted connections from the previous layer. Example Transformation of MaxPooling Layers [00523] Figure 38 shows example transformations 3800 of a MaxPooling layer of a Convolutional Neural Network, according to some implementations. A MaxPooling layer performs a number of calculations of the form ^^ ^^ ^^ ^^ ^^ ^^^ = ^^( ^^ ^^ ^^{ ^^ ^^ ^^ ^^ ^^^}), where “max” refers to computing a maximum of a subset of input values. The subset of input values is defined by pool size and strides parameters (e.g., as specified by a machine learning framework where the original Convolutional Neural Network is defined). Some implementations define dependency relations between each output value and a related subset of input values, and for each output value, build a maximum subtree (sometimes called a Max subtree), thereby converting the MaxPooling layer into a multilayer SNM structure according to following algorithm: a) Define a schematic with 2 layers and 2 SNMs (e.g., SNM 3808 and SNM 3810) performing a max operation (e.g., the Max2 operation 3802) over 2 Input elements (e.g., Input 13804 and Input 23806). b) Define a schematic with 3 layers and 3 SNMs (e.g., SNM 3820, SNM 3822, and SNM 3824) performing a max operation (e.g., the Max3 operation 3812) over 3 Input elements (e.g., Input 13814, Input 23816, and Input 33818). c) Define a schematic with 3 layers and 4 SNMs (e.g., SNM 3836, SNM 3838, SNM 3840, and SNM 3842) performing a max operation (e.g., the Max4 operation 3826) over 4 Input elements (e.g., Input 13828, Input 23830, Input 33832, and Input 43834). d) Because max({x_i}) is symmetric with respect to the arguments (e.g., max(x,y,z) = max(max(x,y), z)), perform transformation of max({Input_i}) calculation into a calculation tree, where each tree node is a Max2, Max3, or Max4 schematic. This tree is built in a manner that minimizes total tree layers and prioritizes the use of the Max4 schematic, according to some implementations. For instance, max(1,2,3,4,5,6,7,8,9) is transformed into max(max(1,2,3,4), max(5,6,7,8), 9) producing a structure of 6 layers with 11 SNMs. e) An activation function other than ReLU can be applied over the output neuron. ReLUs are applied over each SNM without changing the final output value. Human Activity Recognition Using Analog Neuromorphic Computing Hardware [00524] Some implementations include two neural networks in a pipeline. Details on data, model, training, validation main results, for the two neural networks, are described herein. In some implementations, a dataset contains labeled experiments with different types of activity (e.g., more than 50 types of activity), which correspond to one of several basic activity classes. For example, the basic activity classes include ‘SLS’ that indicates a resting state, ‘Walk’ that indicates walking, including on a treadmill, ‘Run’ that indicates running, including on a treadmill, and ‘Intensive’ that indicates various exercises from crossfit workouts, characterized by high load. The two neural networks accomplish two tasks: creation of a descriptor (model converts a time series into a vector, embedding) and classification of base classes of activities (model matches the time series with one of four base classes). The task of creating a descriptor includes training and testing a model (sometimes called a descriptor) that receives data with fixed-length (window) from one or more sensors (e.g., an accelerometer) as input and outputs a vector. Embeddings of windows with the same activity class are likely to be close and can be easily separated and a simple classifier can be built on top of this model. The classification task includes training and testing the classifier for the base classes of activities. [00525] A dataset may correspond to a set of experiments. In some implementations, the dataset includes input for one or more users. For example, one person could participate in several experiments. In some implementations, each person may participate in only one experiment. Each experiment contains information, such as the values of the signal from an accelerometer and activity marks. After processing the dataset, a sequence of windows (data sequences of a fixed length) is obtained. In some implementations, the sequence of windows contains data from 3 channels from the accelerometer, as well as an activity label for each window (or a mode of activity for each window). Training a descriptor model may include two stages: training a neural network model for the descriptor and examining the quality of resulting embeddings. For the first stage, the dataset is divided into two parts: training and validation. The first part is used to train the descriptor, and the second part is used to analyze the quality of these models, according to some implementations. For example, Figure 39 shows an example 3900 of splitting a dataset 3902 to a training dataset 2904 and a validation dataset 3906, based on experiments experiment_1, experiment_2, experiment_3, …, experiment_N, according to some implementations. In this way, the model is tested on a previously unseen experiment, so that it is possible to objectively assess the quality of the model. [00526] In some implementations, special types of activities are selected. These are activities, windows for which are excluded from the training dataset and included in the validation process. To study the quality of the descriptor, five special activities, namely crossfit_airbike, crossfit_row, crossfit_ski, crossfit_situp, walk_treadmill, were selected, and patterns in the accelerometer data included data for these activities. Based on the selected special classes, nine experiments with the largest number of windows with these activities were selected. This resulted in a validation set that contained each specific activity class in at least two experiments. In the course of the experiments, a window length of 125 and a step of 25 were chosen at a sampling frequency of 25 Hz. Example Dataset for Descriptor Validation The descriptor was trained on the previously selected training experiments, but windows for special classes were removed. The validation process includes division of the validation dataset into training or testing samples. Training samples are used to train classifiers (e.g., KNN), and testing samples are used to test the classifiers. To maintain the balance of classes between training and testing, samples are divided as follows. Figure 40 shows two graph plots 4000 and 4002 of activity labels, according to some implementations. The graph plot 4000 corresponds to a mark up of segments for activity labels label_0, label_1, label_2, and label_3. The x-axis in the two graph plots correspond to time periods, and the y-axis corresponds to activity labels. The graph plot 4000 shows a mark up of segments. Segments marked 4004 correspond to activity label label_1 and segments marked 4006 correspond to activity label_2, in an experiment. There are four types of activities (or activity labels) in this experiment. The activity types are selected in turn and periods with the selected activity are identified. For example, for the label_2 activity type, there are two such periods (shown in the graph plot 4002). The two periods for label_2 activity are split into training 4008 and testing 4010 datasets for the selected activity (i.e., label_2). Figures 41A and 41B show an example graph plot 4100 for splitting experiments to training or testing test data, according to some implementations. First half of the periods is included in a training sample 4102, and second half of the periods is included in a test sample 4104. Since the segment size is 125, and the step is 25, the segments have intersections. The techniques described herein allow minimizing the number of intersections between segments from train and test samples. Intersections arise regardless, however. In the transition between different activities, the experiment has an "unknown" label (activity near bottom of they-axis). If an intersection is found between training and testing samples, then the intersecting segments are deleted 4106. Segments belonging to the unknown activity type are prioritized for deletion. As in Figure 40, the y-axis in Figures 41A and 41B show activity labels. In Figures 41A and 41B, the example activities shown are (top to bottom) crossfit_row, walk, standing, sedentary, crossfit_ovrhead_squat, run, pushups, pullups, crossfot_warmup, crossfit_walking_lunges, crosssfit_truster, crossfit_bar_facing_burpees, crossfit_gymnastics, crossfit_situp, and unknown. Example Dataset for Base Class Classifier Validation [00527] In addition to testing descriptors, the quality of the basic activity classification model is also tested. For testing the classifiers, same experiments may be used as for training the descriptor. However, these experiments are no longer labeled into types of activities, but labeled into basic classes of activities. The dataset contains four basic activity classes: SLS, Walk, Run, Intensive. Thus, all previously selected 9 experiments participate in the study of the quality of the classifier of basic classes. However, for a greater variety of activity types in the validation set, two more experiments are added with transport activities, including transport_subway, transport_escalator, transport_bus, transport_tram. As a result, the training dataset consists of 99 experiments (from 64 users) with lengths from 37 minutes to 122 minutes, (in total there are about 338,000 windows), and the validation dataset consists of 11 experiments (from 11 users) with lengths from 38 minutes to 89 minutes (in total there are about 43,000 windows). Example Model [00528] In some implementations, the model consists of three parts: an encoder, a decoder, and a classifier head. The process of training the model is divided into two stages: training the encoder and decoder (sometimes called an autoencoder) and training the classifier. First, the autoencoder is trained, then the autoencoder is frozen and the classifier is trained, which receives embedding from the encoder as input, according to some implementations. Figures 42A, 42B, and 42C show schematic diagrams of an example encoder 4200, an example decoder 4222, and an example classifier 4244, respectively, according to some implementations. Example Descriptor model [00529] As a first step, the autoencoder is trained. The autoencoder includes an encoder (e.g., the encoder 4200) and a decoder (e.g., the decoder 4222). In some implementations, the autoencoder includes one-dimensional convolutions (Conv1d in the encoder and ConvTranspose1d in the decoder), and after each convolutional layer, BatchNorm is used. An example kernel size in Conv1d and ConvTranspose1d layers is 5. In some implementations, the first 4 layers of convolutional layers have stride 1, the remaining layers have stride 2. After each layer, ReLu is used as the activation function. In some implementations, the output of the encoder is a vector of length 16. The result of training this model is an autoencoder that can be used as a descriptor. In some implementations, a window of signals from the accelerometer is input to the descriptor, and at the output, an embedding (a vector of fixed length) is formed for the window, which is characterized by the activity that was performed by the user (similar activities will have close vectors in the resulting vector space). In some implementations, each embedding is a string of 16 bytes. Each of these 16 bytes encodes a feature of a user’s movement. In other words, the 16 bytes together represent a specific movement, and represent a digital fingerprint of the movement. To illustrate, suppose an example embedding includes the following values: -2.3534, -28.4428, 10.2809, - 10.5756, -1.0527, -7.4458, 1.3814, -1.9068, 10.5118, 1.2902, 26.5022, 1.9261, -2.9055, 3.9552, -0.3831, and 13.4464. Each value represents a feature for a specific movement. [00530] In some implementations, as shown in the example shown in Figure 42A, the encoder 4200 includes 3 input channels 4202 (of length 125 each, acc_x, acc_y, and acc_z for the three input channels), a one-dimensional convolution 4204 for 16 channels with stride 1, a one-dimensional convolution 4206 for 16 channels with stride 1, a one-dimensional convolution 4208 for 16 channels with stride 1, a one-dimensional convolution 4212 for 16 channels with stride 1, a one-dimensional convolution 4212 for 32 channels with stride 2, a one-dimensional convolution 4214 for 64 channels with stride 2, a one-dimensional convolution 4216 for 128 channels with stride 2, a flatten layer 4218, and a layer 4220 that generates linear output with length 16 (embedding). [00531] In some implementations, as shown in the example shown in Figure 42B, the decoder 4222 includes input (embedding) 4224 for 1 channel of length 16, a linear layer 4226, a layer 4228 to reshape to 128 channels, a convolution or transpose layer 4230 for 64 channels with stride 2, a convolution or transpose layer 4232 for 32 channels with stride 2, a convolution or transpose layer 4234 for 16 channels with stride 2, a convolution or transpose layer 4236 for 16 channels with stride 1, a convolution or transpose layer 4238 for 16 channels with stride 1, a convolution or transpose layer 4240 for 16 channels with stride 1, and a convolution or transpose layer 4230 for 3 channels of length 125 (which is the reconstructed input). Example Base Class Classifier model [00532] Next, the classifier of the base classes is trained. The encoder trained in the first step (described above) forms an embedding, which is input to a multilayer perceptron, according to some implementations. Some implementations use linear layers with ReLU at the output. In some implementations, the model outputs a vector of length 4 (e.g., a classification into 4 basic activity classes). If the window was marked as unknown activity type, then it received the label 1 and did not participate in training the model and calculating metrics. This is due to the fact that it cannot be said unambiguously which base class of activity took place on a given window. [00533] In some implementations, as shown in the example shown in Figure 42C, the classifier 4244 includes input (embedding) 4246 for 1 channel of length 16, a linear layer 4248 of length 256, a linear layer 4250 of length 128, a linear layer 4252 of length 64, a linear layer 4254 of length 32, and a linear layer 4256 of length 4 (which is the predicted class). Example Training [00534] In some experiments, the training process of the model was carried out on a V100 video card, the consumption of RAM during the training process reached 40GB, the average time of one epoch was 2 minutes 15 seconds. The training was split in two stages: training the autoencoder (for descriptor task) and training the classifier of the basic classes. Training process consisted of 40 epochs for training for the descriptor and 40 epochs for training the classifier, learning rate was 0.001, and for descriptor learning rate scheduler was used (every 10 epochs learning rate was decreased by 2 times). The main metric used to assess the quality of models is F2 score. This metric can be calculated using Equation (1) shown below. In Equation (1), P indicates Precision, and R indicates Recall. ^^2_ ^^ ^^ ^^ ^^ ^^( ^^, ^^) = 5 ∗ ^^ ∗ ^^/(4 ∗ ^^ + ^^) (1) [00535] For descriptor model, some implementations use MSE loss (an example of which is shown in Equation (2) below) as a loss function. In Equation (2), T is true vector, P is predicted vector, and N is vector length.
Figure imgf000137_0001
[00536] For base class classifier model, some implemenations use CrossEntropyLoss (an example of which is shown in Equation (3) below) as a loss function. In this formula, P is the predicted vector of length C, C is a number of classes, and class is an index of truth class. ^^ ^^ ^^ ^^ ( ^^, ^^ ^^ ^^ ^^ ^^ ) = −log
Figure imgf000137_0002
Example Results Descriptor model [00537] Using a model trained on a training sample, embeddings were obtained for all windows of a valid dataset sample. Further, the embeddings were again divided into training or testing using the algorithm described above for training and testing a simple classifier of activity types. KNN (K-Nearest Neighbors) was chosen as the classifier with the number of neighbors 5. This classifier was trained separately for each special type of activity - a binary classification was performed. The predictions from the binary classifier are smoothed using a median filter with a window of 15 predictions (equivalent to 15 seconds). The values of the metrics on the training and testing samples of this classifier are presented below in Tables 1 and 2, respectively. [00538] Table 1: Training scores for KNN model, trained on embeddings
Figure imgf000137_0003
Figure imgf000138_0001
[00539] Table 2: Test scores for KNN model, trained on embeddings
Figure imgf000138_0002
[00540] Figures 43A-43F, 44A-44D, 46A-46F, 47A-47F, and 49A-49D show graphs of results, according to some implementations. In these graphs, the Y-axis corresponds to the type of activity, and the X-axis corresponds to time. Thin lines 4308 indicate the true value of the type of activity, thick or solid lines 4310 indicate prediction of the KNN model, pattern 4312 (with grains) under the graph indicate which sections were assigned to train, and pattern 4314 with stripes under the graph indicate test samples, respectively, for the classifier. These figures show a series of graphs. The problem of binary classification was solved for each individual class, so time series graphs were built for each class. [00541] Figures 43A-43F show graph plots of time series of predictions of the KNN classifier for a set of experiments, according to some implementations. Figures 43A and 43B show a graph plot 4300 that corresponds to data for crossfit_airbike. Note predictions 4302-2, 4302-4, 4302-6, and 4302-8 of crossfit_airbike. For training, accuracy was 0.9919, F1 score was 0.9694, F2 score was 0.9875, precision was 0.9406, and recall was 1.0; and for test, accuracy was 0.9892, F1 score was 0.8879, F2 score was 0.8925, precision was 0.8803, and recall was 0.8957. Figures 43C and 43D show a graph plot 4304 that corresponds to data for crossfit_row. Note predictions 4306-2, 4306-4, 4306-6, 4306-8, and 4306-10 of crossfit_row. For training, accuracy was 0.9871, F1 score was 0.8571, F2 score was 0.8237, precision was 0.9194, and recall was 0.8028; and for test, accuracy was 0.9863, F1 score was 0.795, F2 score was 0.8101, precision was 0.7711, and recall was 0.8205. Figures 43E and 43F show a graph plot 4308 that corresponds to data for crossfit_ski. Note predictions 4310-2, 4310-4, 4310-6, and 4310-8 of crossfit_ski. For training, accuracy was 0.9898, F1 score was 0.8936, F2 score was 0.875, precision was 0.9265, and recall was 0.863; and for test, accuracy was 0.9908, F1 score was 0.8514, F2 score was 0.8182, precision was 0.913, and recall was 0.7975. [00542] Figures 44A-44D show additional graph plots of time series of predictions of the KNN classifier, according to some implementations. Figures 44A and 44B show a graph plot 4400 that corresponds to data for crossfit_row. Note predictions 4402-2, 4402-4, 4402-6, and 4402-8 of crossfit_row. For training, accuracy was 0.9984, F1 score was 0.963, F2 score was 0.942, precision was 1.0, and recall was 0.9286; and for test, accuracy was 0.9977, F1 score was 0.9643, F2 score was 0.9941, precision was 1.0, and recall was 0.931. Figures 44C and 44D show a graph plot 4404 that corresponds to data for crossfit_situp. Note predictions 4406-2, 4406-4, 4406-6, 4406-8, and 4406-10 of crossfit_situp. For training, accuracy was 0.9984, F1 score was 0.9818, F2 score was 0.9712, precision was 1.0, and recall was 0.9643; and for test, accuracy was 0.9948, F1 score was 0.9756, F2 score was 0.9646, precision was 0.9945, and recall was 0.9574. Example Results for Base class classifier model [00543] In some experiments, at the end of training, the loss value on the training dataset stopped at level 0.6993, and on the validation dataset at level 0.7253. Figures 45A and 45B show a table 4500 that presents predictions for the base class activity classifier model on validation experiments described earlier, according to some implementations. The table shows count of predictions of each activity types and base activity classes. Scale 4502 is shown on the right. Columns show base activity classes, and rows show activity types. Values in the first column 4504 and first row 4506 show the number of segments with this activity class and base activity class, respectively. The cells show the number of segments for which the model made this class prediction. Table 3 shown below lists average metrics for the classification task for base activity classes. These metrics were first calculated for each experiment, after which they were averaged between experiments. For this reason, the value of these metrics differs from the metrics shown in Figures 45A and 45B, where they were calculated to add all the experiments together. [00544] Table 3: Mean base class metrics across valid experiments
Figure imgf000140_0001
[00545] Figures 46A-46F show graph plots of time series of predictions of the KNN classifier for a set of experiments, according to some implementations. Figures 46A and 46B show a graph plot 4600 that corresponds to data for crossfit_airbike. Note predictions 4602-2 and 4602-4 of crossfit_airbike. For training, accuracy was 0.9942, F1 score was 0.9286, F2 score was 0.9028, precision was 0.975, and recall was 0.8864; and for test, accuracy was 0.9815, F1 score was 0.8598, F2 score was 0.9388, precision was 0.7541, and recall was 1.0. Figures 46C and 46D show a graph plot 4604 that corresponds to data for crossfit_row. Note predictions 4606-2, and 4606-4 of crossfit_row. For training, accuracy was 0.9933, F1 score was 0.9028, F2 score was 0.853, precision was 1.0, and recall was 0.8228; and for test, accuracy was 0.944, F1 score was 0.5882, F2 score was 0.705, precision was 0.461, and recall was 0.8125. Figures 46E and 46F show a graph plot 4608 that corresponds to data for crossfit_ski. Note predictions 4610 of crossfit_ski. For training, accuracy was 0.9832, F1 score was 0.557, F2 score was 0.4622, precision was 0.8462, and recall was 0.4151; and for test, accuracy was 0.9766, F1 score was 0.7246, F2 score was 0.7062, precision was 0.7576, and recall was 0.6944. [00546] Figures 47A-47F show graph plots of time series of predictions of the KNN classifier for a set of experiments, according to some implementations. Figures 47A and 47B show a graph plot 4700 that corresponds to data for crossfit_row. Note prediction 4702-2 of crossfit_row. For training, accuracy was 0.9988, F1 score was 0.9829, F2 score was 0.9729, precision was 1.0, and recall was 0.9663; and for test, accuracy was 1.0, F1 score was 1.0, F2 score was 1.0, precision was 1.0, and recall was 1.0. Figures 47C and 47D show a graph plot 4704 that corresponds to data for crossfit_ski. Note predictions 4706-2, 4706-4, 4706-6, 4706- 8, 4708-10, and 4706-12 of crossfit_ski. For training, accuracy was 0.9988, F1 score was 0.9875, F2 score was 0.9839, precision was 0.991, and recall was 0.9821; and for test, accuracy was 0.9966, F1 score was 0.9702, F2 score was 0.9677, precision was 0.9744, and recall was 0.9661. Figures 47E and 47F show a graph plot 4708 that corresponds to data for crossfit_situp. Note predictions 4710-2, 4710-4, 4710-6, and 4710-8 of crossfit_situp. For training, accuracy was 0.9949, F1 score was 0.8762, F2 score was 0.8156, precision was 1.0, and recall was 0.7797; and for test, accuracy was 0.9805, F1 score was 0.5122, F2 score was 0.3962, precision was 1.0, and recall was 0.3443. [00547] Figures 46A-46F show a few examples of false recognition of incorrect activity. In Figure 46A, the system predicts crossfit_airbike 4602-4 when the person was at rest (the type of activity is standing). This may be due to arm movements during rest between exercises (arm warm-up). More instances of wrong predictions are shown for crossfit_warmup (e.g., in Figure 46C, the system predicts crossfit_row 4606-2 instead of crossfit_warmup, and in Figure 46D, the system predicts crossfit_row 4606-2 instead of standing). Similarly, in Figure 47F, the system fails to predict crossfit-situp 4712 in some instances. To improve prediction, some implementations use larger training datasets (e.g., use a size large enough to correct such errors noticed during testing). For example, there are public datasets that contain dozens of hours of data. For commercial use, practical applications may use hundreds to thousands of hours of data, and it is possible to collect such data in a practical way. [00548] Figure 48 shows a visualization 4800 for t-distributed stochastic neighbor embedding (t-SNE) by exercise sets (sets 4802 and 4804) for embedding different sets of the crossfit_situp exercise in experiment shown in Figures 47E and 47F, according to some implementations. A set refers to a set of activities for an exercise, performed for a duration, before a user rests. As shown, the last two sets 4802 are far from the previous ones 4804. Nevertheless, embeddings are well grouped into clusters, so with an increase of size of the dataset for training KNN, the quality of classification this type of activity is likely to improve. [00549] Figures 49A-49D show additional graph plots of time series of predictions of the KNN classifier, according to some implementations. Figures 49A and 49B show a graph plot 4900 that corresponds to data for crossfit_ski, without smoothing. Note predictions 4902 of crossfit_ski. Figures 49C and 49D show a graph plot 4904 that corresponds to data for crossfit_ski, with smoothing. Note predictions 4906 of crossfit_ski. Figures 49A-49D show how the presence of smoothing affects the quality of the classification. These graphs show the time series for the crossfit_ski activity binary classifier. When using the median filter (or smoothing, as shown in graphs 49C and 49D), the first set of the exercise 4908 is lost. Without using the median filter (or smoothing), all instances of 4906 are visible. Thus, the use of the filter can degrade the recognition quality of short-set exercises. Therefore, some implementations avoid use of smoothing for short-set exercises. Example Human Activity Recognition Device [00550] Figure 50 shows an example human activity recognition device 5000, according to some implementations. The device includes an integrated circuit 5002 for human activity recognition. The integrated circuit includes an analog network of analog components configured to implement a trained neural network model. The neural network model is trained to generate a plurality of descriptors 5006 for a plurality of predefined human activities based on a plurality of features extracted from a plurality of electrical signals (e.g., signals 5012 and/or signals 5008) from one or more sensors (e.g., external sensors 5010 and/or internal sensors 5008). The device also includes one or more digital components 5004 (e.g., CPUs, DSPs, and associated I/O components) configured to classify human activity as one of the plurality of predefined human activities according to the plurality of descriptors 5006 generated by the integrated circuit 5002. [00551] In some implementations, the human activity recognition device 5000 further includes the one or more sensors (e.g., the internal sensors 5008) configured to collect the plurality of electrical signals during the human activity. [00552] In some implementations, the trained neural network model (implemented by the integrated circuit 5002) is an autoencoder that includes an encoder (e.g., the encoder 4200) and a decoder (e.g., the decoder 4222). [00553] In some implementations, the one or more digital components 5004 implement a trained machine learning classifier that is a KNN (K-Nearest Neighbors) classifier (e.g., the classifier 4244) which can be retrained. In some implementations, wherein a number of neighbors for the KNN classifier equals five. In some implementations, the trained machine learning classifier is trained separately for each of the plurality of predefined human activities using binary classification. [00554] In some implementations, the one or more digital components 5004 are further configured to smooth the output of the trained machine learning classifier to obtain a basic class of activity. [00555] In some implementations, the one or more sensors include one or more of IMUs, cameras, microphones, and biofeedback devices. [00556] In some implementations, the integrated circuit 5002 is fabricated by steps comprising: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model. Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating the integrated circuit, according to the schematic model, using a lithographic process. Examples of related neural network topologies, trained neural networks, analog networks, conversion methods, and fabrication steps, are shown and described above in reference to Figures 1-38, according to some implementations. In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. Example Methods for Recognizing Human Activities [00557] Figure 51 shows a flowchart of an example method 5100 for recognizing (5102) human activities, according to some implementations. The method is performed (5104) by a human activity recognition device (e.g., the device 5000 described above in reference to Figure 50). The method includes using (5106) one or more sensors to track activity of a user, including obtaining a plurality of electrical signals from the one or more sensors. Window length, step size, and/or sampling frequency, for obtaining the electrical signals, may be adjusted, depending on the human activities, according to some implementations. In some implementations, these are adjustable parameters for specific users and/or types of users. In some implementations, the one or more sensors include one or more of IMUs, cameras, microphones, and/or biofeedback devices. [00558] The method also includes forming (5108) a feature vector by extracting a plurality of features from the plurality of electrical signals. The features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities. For example, raw electrical signals from the sensors may be input to a deserializer, and may be sampled (e.g., at 20-25 Hz), to extract features from the electrical signals. The processed signals may be scaled and/or shifted. An accelerometer generates digital signals, which are converted to analog signals before applying transformations. Some implementations select only a single axis or a pair of axes of acceleration data, when extracting features for forming the feature vectors. Some implementations normalize acceleration data, and/or create segments from the acceleration data (sample size may define the number of segments per feature vector). In some implementations, the feature vectors are formed from analog raw signals using a deserializer and sampling using a hold array, where the number of samples and holds equals the number of inputs of an analog neural network. [00559] The method also includes applying (5110) an analog neurocomputing hardware device to the feature vector (in other words, inputting the feature vector to the analog neurocomputing hardware) to generate an embedding vector that specifies a descriptor. The analog neurocomputing hardware device implements the trained neural network model. In some implementations, the trained neural network model is an autoencoder, which includes an encoder and a decoder. An autoencoder is also suitable for keyword spotting, and the exact keywords can be defined after the network is trained. Pictures need 2D convolutions while sound can be done with 1D convolutions, which resemble the example autoencoder architecture described above in reference to Figures 42A and 42B, according to some implementations. The window size may be adjusted depending on the application. [00560] The method also includes applying (5112) a trained machine learning classifier to the embedding vector to classify the activity of the user as one of the predefined human activities. In some implementations, the trained machine learning classifier is a KNN (K-Nearest Neighbors) classifier. In some implementations, the number of neighbors for the KNN classifier equals five. In some implementations, the trained machine learning classifier is trained separately for each of the predefined human activities using binary classification. Some implementations also smooth the output of the trained machine learning classifier to obtain a basic class of activity. Some implementations perform post-processing of the output of the human activity recognition device in an I/O device 5016 which may be a consumer device, such as a smart watch. Smoothing the activity labels may include using average(X) predictions within some time range instead of taking just one prediction during each time step, or using some other more complicated formula. Some implementations perform prediction every second. A user may not need to know the activity every second. For example, the user may want to know that information during a specific time of a day or in certain time periods (e.g., from 1:00 till 1:45, when the user is typically running). Smoothing may be used for that purpose, according to some implementations. [00561] In some implementations, the analog neurocomputing hardware device is fabricated by steps including: obtaining a neural network topology and weights of the trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network model (each element of the weight matrix represents one or more connections between analog components of the equivalent analog network); generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating an integrated circuit, according to the schematic model, using a lithographic process. Examples of these steps are described above in reference to Figures 1-38, according to various implementations. In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix, where each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. In some implementations, the trained machine learning classifier is implemented using one or more digital components and the trained machine learning classifier can be retrained for new users. [00562] Figure 52 shows a flowchart of another example method 5200 for recognizing (5202) human activities, according to some implementations. The method is performed (5204) by a human activity recognition device 5000 described above in reference to Figure 50. The method includes obtaining (5206) a sequence of electrical signals from one or more sensors that track activity of a user. The method also includes forming (5208) a plurality of feature vectors by extracting features from the sequence of electrical signals. The features correspond to inputs for a neural network model trained to generate a plurality of descriptors for a plurality of predefined human activities. The method also includes applying (5210) the analog neurocomputing hardware device to the plurality of feature vectors to generate a plurality of embedding vectors, each of which specifies a corresponding descriptor. [00563] The method also includes using (5212) the plurality of embedding vectors to classify the activity of the user as one of the predefined human activities. In some implementations, the method further includes: receiving (5214), from the user, a set of descriptors that describes specific physical activities; and using the set of descriptors and the plurality of embedding vectors to classify the activity of the user as one of the specific physical activities. In some implementations, the method further includes generating statistics of personal daily routines of the user based on classifying the activity of the user as one of the specific physical activities. [00564] In some implementations, the method further includes: storing (5216), for the user, the plurality of embedding vectors as describing a specific activity; and using the plurality of embedding vectors for classifying subsequent activities of the user as the specific activity. [00565] In some implementations, the method further includes: receiving (5218), from a trainer (not the user), a set of descriptors that describes a specific activity; and providing feedback to the user if the activity matches the specific activity based on the plurality of embedding vectors and the set of descriptors. Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines [00566] Sensors that monitor rotation or reciprocating parts of machines typically generate signals in the range of 0 to 20 kHz. A wideband connection to a control unit is necessary. This makes the sensors consume a lot of power, making it almost impossible to use batteries. An apparatus based on analog neuromorphic hardware is suitable for this application. The apparatus provides embeddings (descriptors) from the analog signal, which reduces the signal bandwidth substantially. Embeddings fully characterize the rotational or reciprocating movements of machine parts and distinguish between normal movements and abnormalities. [00567] Systems and devices according to the techniques described herein may be used to receive raw data signals and add intelligence to various sensors. In some implementations, an architecture contains artificial neurons (the nodes performing computations) and axons (the connections with weights between the nodes) implemented using circuitry elements. The neurons may be implemented as operational amplifiers, and axons may be implemented using thin-film resistors. Some implementations embody the approach of a sparse neural network with only the necessary connections between neurons required for inference. In contrast to in-memory designs, where each neuron is connected to each neighboring neuron, the devices according to the techniques described here simplify chip layout. The techniques work well for Convolutional Neural Networks (CNN), where connections are very sparse, as well as Recurrent Neural Networks (RNNs), transformers, and autoencoders. Techniques for converting a trained and optimized neural network model into a chip structure are described above. The chip architecture offers area utilization close to 100% and may use 8 bits per weight of a trained neural network. Such techniques enable faster time to market, lower technical risks, and better performance. Furthermore, processors according to the techniques described herein may use hybrid cores (using analog and digital hardware) similar to human brain data processing. [00568] Some implementations combine a fixed weights method, which includes complete separation between inference and training, with a fixed chip structure, similar to the human visual nerve and retina, and a flexible function, which differs depending on the application. The flexible function is responsible for further classification of the received embeddings. [00569] In machine learning, it is well-known that, after several hundred training cycles (also known as epochs), a deep convolutional neural network maintains fixed weights and structure for the first 80-90% of the layers, and in the following cycles only the few last layers responsible for classification continue to change weights. This property is also used in transfer learning techniques. This property is used for implementing hybrid analog/digital hardware. In this hybrid approach, a fixed neural network is responsible for pattern detection (e.g., for generating embeddings), and it is combined with a flexible algorithm responsible for the pattern interpretation. The flexible algorithm can be implemented in digital or analog hardware, and may include additional flexible neural networks. In some implementations, the hybrid core includes (i) a fixed neuromorphic analog core that has ultra-low power consumption and low latency, for generating embeddings, and (ii) a flexible digital core for final classification. [00570] Embeddings are representations containing densely packed information about sensory input, and are formed by a neural network or biological nervous system. Embeddings are formed in hidden layers of a neural network, and contain the most significant information about input data. Embeddings may be used as input data for further efficient processing, for data classification, and interpretation. [00571] A classification task may be accomplished in two stages. In the first stage (i), a neural network G is applied to an input vector x from RN to generate an embedding v, where v is a vector with M dimensions and M < N. Applying the neural network G uses a set of trainable parameters WG. This can be expressed concisely as v = G(x, WG), where x is input data, x ^ RN, G is a neural network for building embeddings, WG are trainable parameters of G, v is an embedding, v ^ RM, R represents the set of real numbers, and N and M are positive integers. [00572] In the second stage (ii), a classifier C is applied to the embedding v to compute an output result y, which has one or more components. Applying the classifier C uses trainable parameters WC. This can be expressed concisely as y = C(v, WC), where C is the final classifier, WC are trainable parameters of C, and y is a classification result for input vector x. [00573] As described above, 80-90% of neural network weights do not change after several epochs of training a network and can be subjected to so-called transfer learning. The neural network weights can be a part of new neural network with fixed weights (weights corresponding to 80% of the original neural network). The rest of the neural network (e.g., 20% of the layers) can be trained separately and may be implemented using a flexible hardware system (without fixed weights). This combines fixed parts of a neural network, placed as resistors into a neuromorphic analog signal processor, with flexible parts of the neural network realized in digital circuitry (e.g., in the MCU of the device, at a RISC-V processor, at an FPGA, or at a CPU), which may be integrated (e.g., on the same chip as the neuromorphic analog signal processor). The flexible part of a network may be implemented by means of standard technologies and may be developed using programming languages such as Python, C/C++, Assembler and specialized frameworks (e.g., TensorFlow or Torch) and may run on conventional digital computing units, such as CPUs, GPUs, RISCs, FPGAs. or similar devices, depending on the target device and/or application. The digital computing units may also perform the role of a digital controller, provide signals to interfaces and multiplexing power signals within the neuromorphic analog signal processor. The flexible part may be implemented using compute-in-memory and programmable memory tiles (e.g., SST superflash memory), memristors or other types of programmable memory. The flexible part may also be implemented using a classification neural network or classification algorithms at a CPU of the system, using external computation capabilities, since classification is rather simple and not resource intensive. Since the fixed part is computationally intensive and the flexible part is not, the two parts of a neural network may be logically distributed. [00574] As described above in the Background section, vibrational sensors are commonly used to measure vibrations in machinery, tracks, railway cars, wind turbines, and oil and gas pumps. The signals due to the vibrations may be transferred wirelessly to analytic equipment. Typically, the signals include a large amount of data. The data flow may shorten the battery life of operating sensor nodes. Analog neuromorphic hardware according to the disclosed technology helps reduce the data flow from vibration sensors (sometimes up to 99.9%). [00575] Some implementations use an encoder-decoder approach, and transmit, via long-range technology (sometimes referred to as LoRa or similar low power technology), embeddings extracted from the initial data. The autoencoder systems and embeddings may create new classes, describing new signals of vibration sensors, even if they were not trained to recognize these types of signal patterns. [00576] Analog neuromorphic hardware based on the disclosed technology may be used to implement an encoder neural network, which obtains a range of different vibration signals from various vibration sensors. The output of the analog hardware may be analyzed by a digital system to recognize machine malfunctions. Some implementations detect a predetermined range of frequencies (e.g., up to 20 kHz or up to 60 kHz) and may use an extended sampling rate (e.g., up to 41 kHz). This allows early failure prediction (e.g., predictions up to one month or more in advance) for rotating or moving parts of machinery. The use of the embeddings reduces the amount of data sent to a cloud infrastructure, and addresses the fundamental problem of low bandwidth required by internet-of-things (IoT) systems. [00577] Compressed sensing is one of the technologies used for condition classification of rolling element bearings in rotating machines. This technology allows sampling below the Nyquist sampling rate. It is one of several methods for fault detection and classification in the compressed domain (i.e., without reconstructing the original signal). Compressed sensing may be used for intelligent condition monitoring for bearing faults from highly compressed measurements using sparse over-complete features. Compressed sensing may be used to produce highly compressed measurements of an original bearing dataset. Then a deep neural network (DNN) with an unsupervised feature learning algorithm based on a sparse autoencoder may be used for learning overcomplete sparse representations of these compressed datasets. Finally, the fault classification may be achieved using a softmax regression layer. An autoencoder may be viewed as a source of embeddings that are subsequently processed by a classifier (a softmax regression layer). [00578] Compression of raw data received from sensors enables transmission of a smaller amount of data when compared to uncompressed signals. It is possible to transmit the lower amount of data using low-throughput lines. It is also possible to include redundancy or error-corrections over noisy lines. Additionally, the lower amount of data may be easily processed by a receiver using digital processing techniques. These techniques lead to energy savings, due to which it is possible to have longer battery life and/or include battery-less energy harvesting. Neural networks can be used to extract only relevant features of a large data set. In this way, neural networks may be used to implement a higher quality of signal pre-processing than conventional alternatives of algorithmic data compression. [00579] Figure 53 shows an example sparse autoencoder 5300 for bearing fault classification. Bearing vibration signals 5312 are acquired (in a data acquisition step 5310) using a compressive sampling mechanism to obtain compressed measurements. The measurements are input into a first learning stage 5308, which implements a first encoder for obtaining feature representations v1 having d1 dimensions, with d1 > m (m is the number of input dimensions). The feature representations v1 are input into a second learning stage 5306, which is another encoder for obtaining feature representations v2 having d2 dimensions, with d2 > d1. The feature representations v2 are subsequently input into a softmax regression layer 5304 for classifying bearing health conditions to obtain classification results 5302. In some implementations, the first learning stage 5308 and the second learning stage 5306 are implemented using an analog neuromorphic core using fixed weights and the softmax regression layer 5304 is implemented using a digital circuit. [00580] A ResNet Convolutional Neural Network (CNN) may be used for fault analysis, as shown in Figure 54. A deep ResNet framework with global average pooling (GAP) (sometimes referred to as average pooling) for feature learning and fault diagnosis of rolling bearing has been proposed. See H. Hao, et al., Research on Intelligent Fault Diagnosis of Rolling Bearing Based on Improved Deep Residual Network, 2021. An example of this is shown in Figure 54. The structure of the network 5400 is composed of an input layer 5402, a convolution layer 5404, a max-pooling layer 5406, eight residual blocks 5408, 5410, 5412, 5414, 5416, 5418, 5420, and 5422, followed by a global average pooling (GAP) layer 5424 and a softmax output layer 5426. The network may be used to process fault data of a rolling bearing. A one-dimensional time-domain signal of the rolling bearing fault signal may be input to the network 5400 to obtain a probability distribution of each failure type. The ResNet CNN may be used for bearing fault classification, in which case the output of global average pooling (GAP) layer may be embeddings classified by the softmax output layer. Residual blocks may include Conv1D, Conv2D, BatchNorm, ReLU, and/or add layers. The ResNet CNN may include a dense layer or fully-connected layer, in addition to the GAP layer. [00581] To effectively detect, locate, and identify faults in rolling bearings, a stacked noise reduction autoencoder may be utilized for abstracting characteristics from the original vibration signals, and then, the characteristics may be provided as input for a backpropagation (BP) network classifier as described in Y. Gu, et al., A Denoising Autoencoder-Based Bearing Fault Diagnosis System for Time-Domain Vibration Signals, 2021. The results output by this classifier represent different fault categories. [00582] Figure 55 shows a flowchart for an example method 5500 for deep autoencoder training for bearing fault classification. A deep autoencoder (DAE) may include four layers. The size of the first layer may be identical to the feature dimension size of the training data, which is 2048. The shape of the entire DAE may be, for example, 2048-500- 500-10-500-500-2048, where the structure of the encoder and the structure of the decoder are symmetric. The classification may be performed by removing the decoder from the pretrained DAE and by adding a soft-max layer for the back-propagation network. In Figure 55, in a data acquisition step 5502, original vibration signal data in a training set may be organized (5510). Next, in a pretraining step 5504, the organized data may be used to build (5512) a stacked DAE with N hidden layers. The DAE may be trained for N iterations (steps 5514, 5516, and 5518). The DAE may be finetuned in a next step 5506. Dimensionality of the output layer of the DAE may be determined (5520). The number of dimensions may be equal to the number of possible health conditions of a machine. The parameters of the DAE may be fine-tuned (5522) through a BP network. Once the DAE is trained, the trained model may be applied (5524) to test data and diagnosis results 5508 may be output (5526). [00583] As described above, analog neuromorphic hardware may be used for predictive maintenance of rotating and reciprocating parts of machines. The hardware may be used to generate embeddings (descriptors) from an analog 20 kHz bandwidth signal, which may be packed into an embedding having a size of 1000 bits, and then transferred by a low bandwidth channel (e.g., LoRa or other LPWAN (Low-Power Wide-Area Network) technology). [00584] Some implementations separately generate embeddings using analog neuromorphic hardware using fixed weights, according to the techniques described above. Subsequently, the signal obtained may be analyzed in a cloud or an edge cloud. The cloud’s infrastructure may use a neural network (or layers of a neural network) that uses flexible weights (e.g., the neural network may be retrained). In this way, it is possible to apply fixed weights using an analog neuromorphic hardware for predictive maintenance tasks. [00585] Some implementations train a neural network (or a portion thereof) so that the trained neural network uses analog neuromorphic hardware according to techniques described above, and generates descriptors for vibrational or reciprocating signals, which enables distinguishing abnormal operation of machines from normal operation. [00586] Some implementations use a hybrid core including analog neuromorphic hardware according to techniques described above. The analog neuromorphic hardware implements a fixed part, in a chip that is close to the sensors. The chip can use battery power due to extremely low power consumption. Hardware and/or software for analysis may be implemented in the cloud after transmitting the signal output of the analog neuromorphic hardware through a low bandwidth wireless connection, like LoRa. [00587] In some implementations, an apparatus or a system that includes the hybrid core may be used for control of bearings or any rotating parts. Bearings or other rotating parts (or any movable part) may generate vibrations caused by rotation during operation. The movable part may include any rotating or reciprocating part, a transverse moving part, feed mechanisms, or auxiliary parts of a machine. Periodicity in the vibrations may be used for predicting machine maintenance problems. Some implementations use a vibration wireless sensor node, which is a device that measures the amplitude and frequency of vibration in equipment. Such nodes may be installed on or near parts of a machine and may consist of a vibration sensor (with analog or digital output), an analog-digital converter (ADC), a communication module, and a battery. Such devices may work according to schedule, but may be time-limited because of high power consumption of a wide band communication module. The communication module may transmit a large amount of raw data to analytic equipment for analysis and interpretation. Instead of, or in addition to, such hardware, analog neuromorphic hardware or a hybrid core containing such analog hardware may be used as a part of a sensor node, which receives raw data coming from a sensor. Such hardware may be used to pre- process raw signals and to extract embeddings. Pre-processed data may be 99.9% less when compared to the original input data size. LPWAN technology like LoRa can be used instead of traditional wide band communication technology (for example Wi-Fi or Bluetooth). At the start of any deviation from a normal working mode, there are typically changes to the vibration pattern of a bearing or rotating part of the equipment. After collecting sufficient information, signals may be analyzed to determine the source of the vibration, make failure predictions, and suggest appropriate maintenance. Use of ultra-low power chips provides much longer life from a battery. In addition, energy harvesting technology may be used to collect power from the vibration source. In this way, the techniques described here may be used to collect and transmit vibration data much more frequently and it is possible to make earlier and more precise analytical predictions. [00588] In some implementations, hybrid hardware may be used for the control of vibration in tires. A smart tire sensor node is a device that includes a vibration sensor (with analog or digital output), a neuromorphic analog signal processor, a LoRa communication module (or other LPWAN), and a battery with or without an energy harvesting module. This device may be disposed in or on a tire and may collect vibration data while a vehicle is moving. Initial data coming to the neuromorphic analog signal processor or chip may be pre-processed and/or transformed into embeddings. This reduces data volume by as much as 99.9%. The reduced data may be transmitted by an LPWAN module to a gateway and then to an application (e.g., an application hosted in a cloud) for further analytics and interpretation. The embeddings output by such hardware may be used to predict road surface and physical conditions, as well as monitor tire and suspension condition or a time-to-failure of the vibration source (or a corresponding machine). [00589] Figure 56 shows an example system 5600 for predictive maintenance of a machine, according to some implementations. The system may include a hardware apparatus 5620. The hardware apparatus 5620 may include a vibration sensor 5608 configured to sense vibrations 5612 of a vibration source 5610 (e.g., a machine part) and an analog circuit 5602 comprising a plurality of operational amplifiers and a plurality of resistors. The analog circuit 5602 may be coupled to the vibration sensor 5608, and configured to (i) receive an analog signal 5614 from the vibration sensor 5608 and (ii) compute an output 5616 based on the analog signal, by performing a portion of a trained neural network (e.g., the example ResNet CNN shown in Figure 54). [00590] In some implementations, the hardware apparatus 5620 further includes a transceiver 5606 coupled to the analog circuit 5602 and configured to receive the output 5616 from the analog circuit 5602 and transmit the output 5616 over a low power wide area network (LPWAN) 5618. [00591] In some implementations, the vibration sensor 5608 is disposed adjacent to a movable part (e.g., a rotating or reciprocating part) of a machine 5610 (sometimes called a vibration source), and the vibration sensor 5608 is configured to collect vibration signals 5612 from the rotating or reciprocating part. In some implementations, the rotating part includes a ball bearing of the machine. [00592] In some implementations, the output 5616 of the analog circuit 5602 represents embeddings used for at least one of: defining a source of vibration, predicting failures of a machine coupled to the vibration source, or generating suggestions for maintenance of the machine. [00593] In some implementations, the vibration sensor 5608 is disposed in or on a tire and is configured to collect vibration data for the tire, which is an example of a vibration source. [00594] In some implementations, the output 5616 represents embeddings used to predict at least one of: a road surface, a physical condition, a tire condition, a suspension condition, or a time-to-failure of the vibration source. [00595] In some implementations, the vibration sensor 5608 is configured to sample signals in a range of 0 to 20 kilohertz (kHz). [00596] In some implementations, the vibration sensor 5608 is configured to sample signals up to 41 kHz. [00597] In some implementations, the vibration sensor 5608 is configured to sample signals below a Nyquist sampling rate for fault detection and classification in a compressed domain. [00598] In some implementations, the vibration sensor 5608 is configured to sample signals for compressed sensing (CS) for condition classification of rolling element bearings in rotating machines. [00599] In some implementations, the trained neural network comprises a plurality of layers of neurons including a first set of layers (e.g., all the layers in the ResNet CNN shown in Figure 54 except the softmax layer) and a second set of layers (e.g., the softmax layer) and the analog circuit 5602 is configured to implement the first set of layers. In some implementations, the second set of layers consists of a last layer of the trained neural network. [00600] In some implementations, the trained neural network includes a deep neural network for unsupervised learning based on a sparse autoencoder (e.g., the example sparse autoencoder shown and described above in reference to Figure 53). [00601] In some implementations, the trained neural network comprises a ResNet Convolutional Neural Network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings. [00602] In some implementations, the vibration sensor 5608 is configured to sample and output a one-dimensional time domain signal of a rolling bearing fault signal. [00603] In some implementations, the trained neural network comprises a stacked noise reduction autoencoder. [00604] In some implementations, the analog circuit 5602 is configured to be powered by vibrations of the vibration source 5610. [00605] In some implementations, the hardware apparatus 5620 further includes a power harvesting circuit configured to harvest power from vibrations 5612 of the vibration source 5610 and supply power to the analog circuit 5602. [00606] In some implementations, the plurality of operational amplifiers is configured to implement neurons of the portion of the trained neural network, and the plurality of resistors is configured to implement axons or connections between neurons of the portion of the trained neural network. [00607] In some implementations, the analog circuit 5602 is configured to implement an optimized neural network corresponding to the trained neural network. [00608] In some implementations, values of the plurality of resistors are based on weights of connections of the trained neural network. [00609] In some implementations, the plurality of resistors is configured to connect the plurality of operational amplifiers. [00610] In some implementations, the analog circuit 5602 comprises resistors in the backend-of-the-line (BEOL). [00611] In some implementations, the trained neural network is an autoencoder comprising an encoder portion and a decoder portion. The encoder portion reconstructs an input vector at an output layer after nonlinear transformations performed by hidden layers. The analog circuit 5602 corresponds to the encoder portion of the autoencoder. The encoder portion comprises the hidden layers. The analog circuit 5602 is configured to compute a representation of the input vector in fewer dimensions than an input space of the input vector. [00612] In some implementations, the analog circuit 5602 is configured to generate compressed data that encodes vibration sensor data based on vibration features from the vibration sensor 5608. [00613] In another aspect, the system 5600 includes the hardware apparatus 5620, the transceiver 5606, and a digital circuit 5604. The hardware apparatus 5620 includes the vibration sensor 5608 configured to sense vibrations of the vibration source 5610 of a machine. The hardware apparatus 5620 also includes the analog circuit 5602 comprising a plurality of operational amplifiers and a plurality of resistors. The analog circuit 5602 is coupled to the vibration sensor 5608 and configured to receive an analog signal 5614 from the vibration sensor 5608 and compute an output 5616 based on the analog signal 5614, by performing a portion of a trained neural network. The transceiver 5606 is coupled to the analog circuit 5602 and configured to receive the output 5616 from the analog circuit 5602 and transmit the output 5616 over a low power wide area network (LPWAN) 5618. The analog signal is converted before transmitting. An analog-to-digital converter (ADC) may be included in the hardware apparatus 5620, or may be included as part of the transceiver 5606. The digital circuit 5604 is communicatively coupled to the transceiver 5606 of the hardware apparatus 5620 via the LPWAN. The digital circuit 5604 is configured to receive the output from the analog circuit 5602 (via the transceiver 5606) and predict the state 5624 of the machine for maintenance (e.g., by a device 5622 for maintaining the machine), based on the output. [00614] In some implementations, the digital circuit 5604 includes one or more digital computing units, including one or more of: CPUs, GPUs, RISCs, FPGAs, and ASICs. [00615] In some implementations, the digital circuit 5604 comprises a processor configured to perform data classification. In some implementations, the data classification is performed by a neural network that is distinct from the trained neural network. In some implementations, the data classification is performed using k-nearest neighbors (k-NN). [00616] In some implementations, the output of the analog circuit 5602 represents embeddings and the digital circuit 5604 is configured to use the embeddings to classify the analog signal 5614. [00617] In some implementations, a method is provided for vibration sensing. The method includes sensing vibrations of a vibration source using a vibration sensor (e.g., the vibration sensor 5608) to obtain an analog signal (e.g., the analog signal 5614). The method also includes computing an output (e.g., the output 5616) based on the analog signal, by performing a portion of a trained neural network, using an analog circuit (e.g., the analog circuit 5602) comprising a plurality of operational amplifiers and a plurality of resistors. The method also includes transmitting the output over a low power wide area network (LPWAN) 5618 using a transceiver 5606. [00618] The terminology used in the description of the invention herein is for the purpose of describing particular implementations only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. [00619] The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various implementations with various modifications as are suited to the particular use contemplated.

Claims

What is claimed is: 1. A system comprising: a hardware apparatus comprising: a vibration sensor configured to sense vibrations of a vibration source of a machine; an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors, wherein the analog circuit is coupled to the vibration sensor and configured to: receive an analog signal from the vibration sensor; and compute an output based on the analog signal, by performing a portion of a trained neural network; a transceiver coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN); and a digital circuit communicatively coupled to the transceiver of the hardware apparatus via the LPWAN, wherein the digital circuit is configured to: receive the output from the analog circuit; and predict a state of the machine for maintenance, based on the output.
2. The system of claim 1, wherein the digital circuit comprises one or more digital computing units selected from the group consisting of: CPUs, GPUs, RISCs, FPGAs, and ASICs.
3. The system of claim 1, wherein the digital circuit comprises a processor configured to perform data classification.
4. The system of claim 3, wherein the data classification is performed by a neural network that is distinct from the trained neural network.
5. The system of claim 3, wherein the data classification is performed using k-nearest neighbors (k-NN).
6. The system of claim 1, wherein the output of the analog circuit represents embeddings and the digital circuit is configured to use the embeddings to classify the analog signal.
7. A hardware apparatus comprising: a vibration sensor configured to sense vibrations of a vibration source; an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors, wherein the analog circuit is coupled to the vibration sensor and configured to: receive an analog signal from the vibration sensor; and compute an output based on the analog signal, by performing a portion of a trained neural network.
8. The hardware apparatus of claim 7, further comprising: a transceiver coupled to the analog circuit, wherein the transceiver is configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN).
9. The hardware apparatus of claim 7, wherein the vibration sensor is disposed adjacent to a movable part of a machine, and the vibration sensor is configured to collect vibration data for the movable part.
10. The hardware apparatus of claim 9, wherein the movable part includes a ball bearing of the machine.
11. The hardware apparatus of claim 7, wherein the output of the analog circuit represents embeddings used for at least one of: defining a source of vibration, predicting failures of a machine coupled to the vibration source, and generating suggestions for maintenance of the machine.
12. The hardware apparatus of claim 7, wherein the vibration sensor is disposed in or on a tire and is configured to collect vibration data for the tire.
13. The hardware apparatus of claim 7, wherein the output represents embeddings used to predict at least one of: a road surface, a physical condition, a tire condition, a suspension condition, or a time-to-failure of the vibration source.
14. The hardware apparatus of claim 7, wherein the vibration sensor is configured to sample signals in a range of 0 to 20 kilohertz (kHz).
15. The hardware apparatus of claim 7, wherein the vibration sensor is configured to sample signals up to 41 kHz.
16. The hardware apparatus of claim 7, wherein the vibration sensor is configured to sample signals below a Nyquist sampling rate for fault detection and classification in a compressed domain.
17. The hardware apparatus of claim 7, wherein the vibration sensor is configured to sample signals for compressed sensing (CS) for condition classification of rolling element bearings in rotating machines.
18. The hardware apparatus of claim 7, wherein the trained neural network comprises a plurality of layers of neurons including a first set of layers and a second set of layers and the analog circuit is configured to implement the first set of layers.
19. The hardware apparatus of claim 18, wherein the second set of layers consists of a last layer of the trained neural network.
20. The hardware apparatus of claim 7, wherein the trained neural network comprises a deep neural network for unsupervised learning based on a sparse autoencoder.
21. The hardware apparatus of claim 7, wherein the trained neural network comprises a ResNet Convolutional Neural Network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings.
22. The hardware apparatus of claim 7, wherein the vibration sensor is configured to sample and output a one-dimensional time domain signal of a rolling bearing fault signal.
23. The hardware apparatus of claim 7, wherein the trained neural network comprises a stacked noise reduction autoencoder.
24. The hardware apparatus of claim 7, wherein the analog circuit is configured to be powered by vibrations of the vibration source.
25. The hardware apparatus of claim 24, further comprising a power harvesting circuit configured to harvest power from vibrations of the vibration source and supply power to the analog circuit.
26. The hardware apparatus of claim 7, wherein the plurality of operational amplifiers is configured to implement neurons of the portion of the trained neural network, and wherein the plurality of resistors is configured to implement connections between neurons of the portion of the trained neural network.
27. The hardware apparatus of claim 7, wherein the analog circuit is configured to implement an optimized neural network corresponding to the trained neural network.
28. The hardware apparatus of claim 7, wherein values of the plurality of resistors are based on weights of connections of the trained neural network.
29. The hardware apparatus of claim 7, wherein the plurality of resistors is configured to connect the plurality of operational amplifiers.
30. The hardware apparatus of claim 7, wherein the analog circuit comprises resistors in a backend-of-the-line (BEOL).
31. The hardware apparatus of claim 7, wherein: (i) the trained neural network is an autoencoder comprising an encoder portion and a decoder portion; (ii) the encoder portion reconstructs an input vector at an output layer after nonlinear transformations performed by hidden layers; (iii) the analog circuit corresponds to the encoder portion of the autoencoder, the encoder portion comprising the hidden layers; and (iv) the analog circuit is configured to compute a representation of the input vector in fewer dimensions than an input space of the input vector.
32. The hardware apparatus of claim 7, wherein the analog circuit is configured to generate compressed data that encodes vibration sensor data based on vibration features from the vibration sensor.
33. A method comprising: sensing vibrations of a vibration source using a vibration sensor to obtain an analog signal; computing an output based on the analog signal, by performing a portion of a trained neural network, using an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors; and transmitting the output over a low power wide area network (LPWAN) using a transceiver.
PCT/US2023/031692 2022-09-02 2023-08-31 Neuromorphic analog signal processor for predictive maintenance of machines Ceased WO2024049998A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2025507806A JP2025531660A (en) 2022-09-02 2023-08-31 Neuromorphic Analog Signal Processor for Machinery Predictive Maintenance
CN202380053827.6A CN119547080A (en) 2022-09-02 2023-08-31 Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines
EP23772700.3A EP4581521A1 (en) 2022-09-02 2023-08-31 Neuromorphic analog signal processor for predictive maintenance of machines

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US17/902,757 2022-09-02
US17/902,757 US20230081715A1 (en) 2020-06-25 2022-09-02 Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines

Publications (1)

Publication Number Publication Date
WO2024049998A1 true WO2024049998A1 (en) 2024-03-07

Family

ID=88093894

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2023/031692 Ceased WO2024049998A1 (en) 2022-09-02 2023-08-31 Neuromorphic analog signal processor for predictive maintenance of machines

Country Status (4)

Country Link
EP (1) EP4581521A1 (en)
JP (1) JP2025531660A (en)
CN (1) CN119547080A (en)
WO (1) WO2024049998A1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112862059A (en) * 2019-11-28 2021-05-28 华为技术有限公司 Long-short term memory LSTM network computing device and computing device
CN118033432A (en) * 2024-04-12 2024-05-14 中国科学院大连化学物理研究所 Battery state of charge estimation method and device and computer equipment
CN118647134A (en) * 2024-07-09 2024-09-13 诚瑞电路有限公司 Multilayer circuit board and its manufacturing process
CN118886380A (en) * 2024-07-09 2024-11-01 重庆惠程未来智能电气有限公司 Current sampling and correction system and method applied to electric motorcycles through physical modeling
CN119443356A (en) * 2024-10-09 2025-02-14 国家能源集团科学技术研究院有限公司 Wind power generation prediction method and electronic equipment based on deep learning
CN119509787A (en) * 2024-10-28 2025-02-25 中北大学 Chip design method and system for silicon piezoresistive pressure sensor

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200166922A1 (en) * 2018-05-07 2020-05-28 Strong Force Iot Portfolio 2016, Llc Methods and systems for data collection, learning, and streaming of machine signals for analytics and predicted maintenance using the industrial internet of things

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200166922A1 (en) * 2018-05-07 2020-05-28 Strong Force Iot Portfolio 2016, Llc Methods and systems for data collection, learning, and streaming of machine signals for analytics and predicted maintenance using the industrial internet of things

Non-Patent Citations (10)

* Cited by examiner, † Cited by third party
Title
ABDELBAKI H ET AL: "Analog hardware implementation of the random neural network model", NEURAL NETWORKS, 2000. IJCNN 2000, PROCEEDINGS OF THE IEEE-INNS-ENNS I NTERNATIONAL JOINT CONFERENCE ON 24-27 JULY 2000, PISCATAWAY, NJ, USA,IEEE, vol. 4, 24 July 2000 (2000-07-24), pages 197 - 201, XP010506431, ISBN: 978-0-7695-0619-7 *
ANONYMOUS: "https://scienceprog.com/thoughts-on-interfacing-piezo-vibration-sensor/", 22 August 2013 (2013-08-22), XP093101249, Retrieved from the Internet <URL:https://scienceprog.com/thoughts-on-interfacing-piezo-vibration-sensor/> [retrieved on 20231114] *
CONVOLUTIONAL NEURAL NETWORKS FOR HUMAN ACTIVITY RECOGNITION USING MOBILE SENSORS
H. HAO ET AL., RESEARCH ON INTELLIGENT FAULT DIAGNOSIS OF ROLLING BEARING BASED ON IMPROVED DEEP RESIDUAL NETWORK, 2021
MUSCAT ANDREW ET AL: "Electromagnetic Vibrational Energy Harvesters: A Review", SENSORS, vol. 22, no. 15, 25 July 2022 (2022-07-25), pages 5555, XP093104577, DOI: 10.3390/s22155555 *
REAL-TIME DEEP LEARNING AT THE EDGE FOR SCALABLE RELIABILITY MODELING OF SI-MOSFET POWER ELECTRONICS CONVERTERS
SHEA SHARON: "What is LPWAN (low-power wide area network)? | Definition from TechTarget", 30 September 2017 (2017-09-30), XP093101276, Retrieved from the Internet <URL:https://www.techtarget.com/iotagenda/definition/LPWAN-low-power-wide-area-network> [retrieved on 20231114] *
T. KOUNOVSKYJ. MALEK, SINGLE CHANNEL SPEECH ENHANCEMENT USING A CONVOLUTIONAL NEURAL NETWORK, 2017
WANG QIANG ET AL: "Compressive sensing reconstruction for vibration signals based on the improved fast iterative shrinkage-thresholding algorithm", MEASUREMENT, vol. 142, 31 August 2019 (2019-08-31), pages 68 - 78, XP085692250, ISSN: 0263-2241, DOI: 10.1016/J.MEASUREMENT.2019.04.012 *
Y. GU ET AL., A DENOISING AUTOENCODER-BASED BEARING FAULT DIAGNOSIS SYSTEM FOR TIME-DOMAIN VIBRATION SIGNALS, 2021

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112862059A (en) * 2019-11-28 2021-05-28 华为技术有限公司 Long-short term memory LSTM network computing device and computing device
CN112862059B (en) * 2019-11-28 2025-06-17 华为技术有限公司 Long short-term memory (LSTM) network computing device and computing device
CN118033432A (en) * 2024-04-12 2024-05-14 中国科学院大连化学物理研究所 Battery state of charge estimation method and device and computer equipment
CN118647134A (en) * 2024-07-09 2024-09-13 诚瑞电路有限公司 Multilayer circuit board and its manufacturing process
CN118886380A (en) * 2024-07-09 2024-11-01 重庆惠程未来智能电气有限公司 Current sampling and correction system and method applied to electric motorcycles through physical modeling
CN119443356A (en) * 2024-10-09 2025-02-14 国家能源集团科学技术研究院有限公司 Wind power generation prediction method and electronic equipment based on deep learning
CN119509787A (en) * 2024-10-28 2025-02-25 中北大学 Chip design method and system for silicon piezoresistive pressure sensor

Also Published As

Publication number Publication date
EP4581521A1 (en) 2025-07-09
CN119547080A (en) 2025-02-28
JP2025531660A (en) 2025-09-25

Similar Documents

Publication Publication Date Title
US20230081715A1 (en) Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines
JP7678052B2 (en) Analog Hardware Realization of Neural Networks
US12327182B2 (en) Optimizations for analog hardware realization of trained neural networks
US20220280072A1 (en) Systems and Methods for Human Activity Recognition Using Analog Neuromorphic Computing Hardware
WO2024049998A1 (en) Neuromorphic analog signal processor for predictive maintenance of machines
WO2022191879A1 (en) Analog hardware realization of trained neural networks for voice clarity
JP7770589B2 (en) Systems and methods for human activity recognition
US20220268229A1 (en) Systems and Methods for Detonation Control in Spark Ignition Engines Using Analog Neuromorphic Computing Hardware
WO2021259482A1 (en) Analog hardware realization of neural networks
WO2023220437A1 (en) Systems and methods for human activity recognition using analog neuromorphic computing hardware
US20210406662A1 (en) Analog hardware realization of trained neural networks for voice clarity
US20240005140A1 (en) Quantization Algorithms for Analog Hardware Realization of Neural Networks
US11823037B1 (en) Optocoupler-based flexible weights in neuromorphic analog signal processors
US12347421B2 (en) Sound signal processing using a neuromorphic analog signal processor
RU2796649C2 (en) Analogue hardware implementation of neural networks

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23772700

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202417104806

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 202380053827.6

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2025507806

Country of ref document: JP

WWP Wipo information: published in national office

Ref document number: 202380053827.6

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 202417104806

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2023772700

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2023772700

Country of ref document: EP

Effective date: 20250402

WWP Wipo information: published in national office

Ref document number: 2023772700

Country of ref document: EP