WO2025165752A1 - 3d photonic-electronic neuromorphic computing - Google Patents

Abstract

A combination of photonic neural networks and electronic neural networks integrated in 3D to form 3D Electronic-Photonic Integrated Circuits (3D EPICs) are described toward enabling new brain-derived neuromorphic hardware with energy-efficiency, connectivity, density, and scalability. Described are the construction of the optoelectronic (OE) neurons in the photonic neural network (PNN) including photonic-memristive dendrites, photonic-memristive synapses, photonic axons, and nano-electronic somas. These OE neurons and their hierarchical interconnections are described in constructing 3D EPICs. The use of a scalable PNN neuromorphic computing simulator is described, as well as PNN training.

Description

3D PHOTONIC-ELECTRONIC NEUROMORPHIC COMPUTING

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to, and the benefit of, U.S. provisional patent application serial number 63/626,513 filed on January 29, 2024, incorporated herein by reference in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with Government support under Grant No. FA9550-18-1-0186 and under Grant No. FA9550-22-1-0532, awarded by the Air Force Office of Scientific Research. The Government has certain rights in the invention.

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

[0003] A portion of the material in this patent document may be subject to copyright protection under the copyright laws of the United States and of other countries. The owner of the copyright rights has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the United States Patent and Trademark Office publicly available file or records, but otherwise reserves all copyright rights whatsoever. The copyright owner does not hereby waive any of its rights to have this patent document maintained in secrecy, including without limitation its rights pursuant to 37 C. F. R. § 1 .14.

BACKGROUND

[0004] 1. Technical Field

[0005] The technology of this disclosure pertains generally to neural based artificial intelligence, and more particularly to a hierarchical photonic and electronic neural network consisting of ionic dendrites, electronic somas, electronic axons, and ionic synapses for the electronic neural networks, and photonic memristive synapses, photonic dendrites, photonic somas, and photonic axons for photonic neural networks.

[0006] 2. Background Discussion

[0007] Artificial Intelligence research using conventional electronics-based methods, Electronic Integrated Circuitries (EICs) have failed to match the scalability, energy efficiency, and self-supervised learning capabilities of the human brain. Current approaches in Artificial Intelligence (Al) are based on computational architectures that are loosely inspired by the brain. The human brain, in contrast, is capable of remarkably fast learning in a manner that is flexible and enables generalization to new situations and tasks, and it does so with a remarkably low level of energy consumption relative to traditional computational hardware.

[0008] Accordingly, a need exists for a brain-derived architecture that can provide the processing power and task flexibility of the brain, while operating at low power levels. The present disclosure fulfills that need and provides additional benefits over existing systems.

BRIEF SUMMARY

[0009] This disclosure describes 3D Electronic Photonic Integrated Circuitry (3D EPIC) that combines electronic Neuromorphic circuits and photonic (optoelectronic) neuromorphic circuits. The described 3D EPICs offer high density and high connectivity with extreme efficiency at scale while supporting hierarchical learning in optical nano/macro-circuits and electronic nano/micro- circuits. 3D integration of electronic neural networks and photonic neural networks together with learning algorithms into a 3D EPIC neural network facilitates human-like hierarchical learning capability.

[0010] In one embodiment, an artificial intelligence machine comprises: (a) photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (b) electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; and (c) interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

[0011] In another embodiment, a method of creating an artificial intelligence machine comprises: (a) providing photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (b) providing electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; and (c) providing interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

[0012] In still another embodiment, an artificial intelligent machine comprises: a plurality of modules of electronic neuromorphic computing circuits and photonic neuromorphic computing circuits wherein each module specializes in artificial intelligent functions, mimicking those of hippocampus, thalamus, and others in a biological brain.

[0013] Further aspects of the technology described herein will be brought out in the following portions of the specification, wherein the detailed description is for the purpose of fully disclosing preferred embodiments of the technology without placing limitations thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The technology described herein will be more fully understood by reference to the following drawings which are for illustrative purposes only:

[0015] FIG. 1 and FIG. 2 are schematics of nanoscale optoelectronic spiking neurons according to at least one embodiment of the present disclosure.

[0016] FIG. 3 and FIG. 4 are a comparison between SPICE simulation and experimental results for the nanoscale optoelectronic spiking neurons.

[0017] FIG. 5 is a schematic of a synaptic PNN according to at least one embodiment of the present disclosure.

[0018] FIG. 6 and FIG. 7 are representations of 2x2 photonic memristive-MZI synapse between each layer as utilized according to at least one embodiment of the present disclosure.

[0019] FIG. 8 and FIG. 9 are pictorial view of a hybrid neuromorphic computing platform according to at least one embodiment of the present disclosure.

[0020] FIG. 10 is a pictorial view of a hybrid nano-electronic-photonic neuromorphic cores according to at least one embodiment of the present disclosure. [0021] FIG. 11 is a pictorial view of a nano-electronic neural network cores according to at least one embodiment of the present disclosure.

[0022] FIG. 12 is a pictorial view of nano-electronic neural network cores stacked with CMOS photonics according to at least one embodiment of the present disclosure.

[0023] FIG. 13 is a schematic of a hierarchical photonic and electronic neural network according to at least one embodiment of the present disclosure.

[0024] FIG. 14 is a cross section of one plane of the 3D EPIC according to at least one embodiment of the present disclosure.

[0025] FIG. 15A and FIG. 15B are a schematic of a neural network training setup according to at least one embodiment of the present disclosure.

[0026] FIG. 16 and FIG. 17 schematically show 3D EPIC examples utilizing optical TSVs and electrical TSVs assuming each layer to be either a monolithic silicon CMOS-photonic die (e.g., GF45SPCLO) or a 3D DBI bonded EIC-PIC layer, according to embodiments of the present disclosure.

[0027] FIG. 18 schematically shows an example of heterogeneous 3D EPIC chiplets integrated on a reconfigurable optoelectronic interposers according to at least one embodiment of the present disclosure.

[0028] FIG. 19A and FIG. 19B are side view diagrams for the interlayer coupling using grating couplers and TSOVs based on flip chip bonding and chip stacking to evaluate the coupling performance according to at least one embodiment of the present disclosure.

[0029] FIG. 20A and FIG. 20B is a schematic diagram of two upside-up gratings for interlayer coupling, along with a chart showing corresponding coupling efficiency as a function of misalignment between the gratings, according to at least one embodiment of the presented technology.

[0030] FIG. 21 A and FIG. 21 B is a schematic diagram showing one upside-up and one upside-down ratings for interlayer coupling, along with a chart showing corresponding coupling efficiency as a function of misalignment between the gratings, according to at least one embodiment of the presented technology.

[0031] FIG. 22A and FIG. 22B are schematics of an ultra-compact vertical U- shaped coupler comprising a TSV and two 45 degree reflectors, according to at least one embodiment of the present disclosure.

[0032] FIG. 23 is an image rendition of an upside-down grating according to at least one embodiment of the present disclosure.

[0033] FIG. 24 is a schematic showing the GDS layout for a TSOV according to at least one embodiment of the present disclosure.

[0034] FIG. 25 is a schematic diagram showing the GDS layout for an integrated structure of a TSOV and a grating coupler according to at least one embodiment of the present disclosure.

[0035] FIG. 26 is a schematic of a GDS layout for the grating couplers and TSOVs used to evaluate the interlayer coupling performance based on flip chip bonding and chip stacking, according to at least one embodiment of the present disclosure.

[0036] FIG. 27 is a schematic of chip-to-chip coupling using an upside-up grating and an upside-down grating based on chip stacking according to at least one embodiment of the present disclosure.

DETAILED DESCRIPTION

[0037] 1. Introduction

[0038] 1.1. Opto-electronic Neurons and Interconnects

[0039] The construction of optoelectronic neurons in a photonic neural network (PNN) includes photonic-memristive dendrites, photonic-memristive synapses, photonic axons, and nano-electronic SOMAs.

[0040] FIG. 1 through FIG. 4 illustrate example embodiments of nanoscale optoelectronic spiking neurons and results.

[0041] In FIG. 1 is illustrated an embodiment 10 of a nanoscale optoelectronic spiking neuron consisting of nanoscale nanophotonic detectors 16 shown for the sake of simplicity as Ge/Si photonic crystal enhanced photodiodes. The nanophotonic detectors allow receiving an excitatory input 12, and inhibitory input 14. The device is shown with nanoelectronics having extremely low parasitics, such as less than 0.5 fF, for less than 1 fJ/spike (i.e. , approximately 10 fJ/spike for a fanout exceeding 80) efficiency integrated on a silicon- photonic platform exploiting quantum impedance conversion capable of exhibiting both excitatory and inhibitory neuron behaviors by using nanophotonic detectors PD1 and PD2, respectively. In addition, a nanolayer is incorporated as the output device. The circuit can also integrate K+, Na+, and Ca++ ion-channels.

[0042] Two Field Effect Transistors (Nano FETs) 18, 20 are shown on a Silicon-On-lnsulator (SOI) substrate 21 for thresholding and spiking signal generation by triggering a quantum-dot laser (Nanolaser) 22 with photonic crystal patterns etched for in-plane emission 24. Passive circuit elements in biasing and controlling the FETs and nanolaser are shown with R1 26, R2 28 and C1 30, The photonic crystal cavity laser in this example is fabricated on silicon utilizing hybrid integration by transfer-printing to realize hybrid III- V/silicon nanophotonic devices. According to at least one embodiment, the nanolasers utilize a hybrid InAs/AIGaAs quantum-dot on SOI structure with photonic crystal patterns etched in on silicon. The hybrid InP Multi-Quantum- Well or silicon semiconductor optical amplifier can be utilized in a similar fabrication process. The absence of capacitive charge associated with the interconnect wires can drastically reduce future nanophotonic neurons’ power consumption.

[0043] In a lower portion of FIG. 1 is seen a symbol used herein for an optoelectronic neuron 40 with photonic dendrite 42 coupled to a photo-detector (PD) 44 to SOMA 46, whose output is to a nano laser device (LD) 48 outputting to a photonic axon 49. Thus, each optoelectronic neuron consists of a photodetector 44, a nonlinear electronic circuit (soma) 46, and a laser diode 49. The optical Through Si Vias (TSOVs) are a new class of transparent chalcogenide Phase Change Material (PCM) based on optical waveguides which can be utilized as photonic dendrites operating similarly to the manner of electronic dendrites.

[0044] In FIG. 2 is shown a schematic of a spiking optoelectronic nano neuron circuit design using Cadence circuit-level simulation using the Verilog-A model, and is a slightly more elaborate neuron circuit than seen in FIG. 1.

[0045] This model is based on an Izhikevich model and introduces the following three Eqs. (1 )-(3) that govern the optoelectronic neurons design of FIG. 5. [0046] The membrane potential of the neuron is expressed as: whereas the refractory potential can be realized as:

A directly modulated laser implements the output of the optoelectronic neuron. The amplitude of the laser output signal is determined by baser, which can be approximated as: baser = K₂max {0, v - V_th2}² (3) wherein values R1 , R2, C1 , and C2 are the values of the resistors and the capacitors. Values K1 , K2 and K3 are transconductance gain of the fieldeffect transistor (FETs). Vth1 , Vth2, and Vth3 are the threshold of the fieldeffect transistor (FETs) shown in FIG. 2. To efficiently utilize the model in the neural network simulations, it neglects the subthreshold behaviors of the transistors and the lasers.

[0047] In FIG. 2 is seen the working principle of the optoelectronic neuron. The operating points of all transistors is set to a saturation condition. The optical spiking inputs detected by the excitatory photodetector PD1_exe 52 generate photocurrents 58 to be integrated 60 by in the membrane potential circuit (MPC) 60 having capacitor C1 and discharged through its resistor R1 . As the voltage (the membrane potential) of the MPC builds-up to the threshold of FET1 64 and the combination of FET2 66 and FET6 76 to drive current through the nanolaser 78, it will fire output spikes, and capacitance C2 in the Refractory Potential Circuit (RPC) 70 will start to charge up in response to receiving current 62 through the combined operation of FET1 64, FET4 72 and FET5 74. As the refractory potential builds up to the threshold of FET3 67, the membrane potential is reset and kept at the reset voltage level until the refractory potential discharges below the threshold of FET3.

[0048] In addition to the excitatory connection, a critical feature included in the designed neuron model is the inhibitory connection. The inhibitory connection can be interpreted as a negative input to the neuron. In SNNs, inhibitory input signals decrease the accumulated membrane potential and make the neuron less responsive to the excitatory inputs. Contrary to the perceptron’s negative inputs in ANNs, inhibitory inputs do not affect the neuron behaviors when the neurons are at the resting state. They are only effective when the neurons are excited beforehand.

[0049] In the optoelectronic neuron hardware, excitatory and inhibitory inputs are received by the photodetector pair (PD1_exc and PD2_inh) biased at the voltage Vd in reference to the ground. Therefore, inhibitory inputs are ineffective when the optoelectronic neuron is at its resting state v(t) = 0, where v(t) is measured at the Membrane Potential.

[0050] There are two principle differences between the original Izhikevich model and the presented optoelectronic neuron model. First, the quadratic positive feedback part of the membrane potential in the Izhikevich model is truncated because it leads to instabilities in the (optoelectronic neuron) circuit when the input signal is large. However, in the depicted neuron the Izhikevich model considers the spike output solely as v(t). In contrast, for optoelectronic neurons, another stage of the circuit involving FET2 is required to map the membrane potential voltage to drive electrical current into the laser. The second difference lies in its threshold behavior, as shown in Eq. (3). In the original Izhikevich model, both variables, v(t) and u(t) are immediately set to their reset values when the membrane potential reaches the threshold voltage. However, in the depicted optoelectronic neuron model there is a gradual reset of the two variables over time in response to the interaction of FET3, R2, and C2 in the circuit.

[0051] In FIG. 3 is shown HSpice simulation results 90 of neuron spiking behavior with both excitatory and inhibitory signal inputs. A voltage buildup can be seen as depicted by the dark vertical spikes, while the light colored spikes represent the optical excitatory input.

[0052] In FIG. 4 is shown experimental results 110 of neuron behavior with optical I/O as obtained from prototyped optoelectronic neurons fabricated utilizing off-the-shelf FETs, detectors, and lasers. It can be seen that the experimental results of FIG. 4 have a high degree of matching with the SPICE simulation.

[0053] Future 3D integration of photonic networks can benefit from TSOVs utilizing Wavelength Division Multiplexing (WDM) to support high fan-in and fan-out interconnections which mimic the brain. Silicon photonic TSOVs have been demonstrated exploiting 45 degree anisotropic etching of silicon.

[0054] For achieving photonic interposer-based 3D integration of 3D Electronic-Photonic Integrated-Circuits (EPICs), a method has been recently demonstrated of attaching free-form micro-optical reflectors onto in-plane waveguides to redirect and collimate the output light from the waveguide to a surface-normal direction. Both TSOVs achieve less than 1-dB loss across optical bandwidths exceeding 300 nm, enabling WDM 3D photonic interconnection between the EPIC planes and between 3D EPIC chips.

[0055] It should be appreciated that an Ising machine can be realized on a Photonic Neural Network (PNN) to achieve scale-free distributed learning. It will be noted that the creation of optical Ising machines should open up the ability to solve complex optimization problems with an optical hardware acceleration advantage. This can be achieved by utilizing Nano-Oscillators as a soma integrated with a nanolaser for Electronic to Optical (E/O) conversion, and nanodetectors for Optical to Electronic (O/E) conversion to complete a parametrically-driven optical oscillator neuron 40 as seen in the lower portion of FIG. 1 . The working principle of emergent dynamics allows creation of scale-free hierarchical photonic-electronic Continuous-Time Dynamical Systems (CTDS), based on collective dynamics of OE devices and their coupled networks to achieve distributed learning.

[0056] FIG. 5 through FIG. 7 illustrate an initial design of brain-like PNN chips which leverage Phase Change Material (PCM)-based photonic RAM (PRAM) memristive weights, Si-photonic-based dendrites, and optoelectronic or electronic neurons.

[0057] In FIG. 5 is shown an example embodiment 210 of a brain-derived synaptic PNN as a self-optimizing nanophotonic neural network. This single wavelength example of a Nanophotonic Neural Network utilizes the nanoscale optoelectronic neurons shown in FIG. 1 and FIG. 2.

[0058] A plurality of nanoscale optoelectronic spiking neurons 212 are shown as the input side 212 coupled to a network layer(s) to an output side 214. The photonic synaptic network is shown consisting of optical couplers 216 and tunable phase shifters 218 with interconnecting optoelectronic neurons to form an optical neural network. The photonic synapses consist of photonic phase change materials (PCMs). Along one neural path is a Mach-Zehnder Interferometer (MZI) mesh 220. Output of the PNN is through another plurality of nanoscale optoelectronic spiking neurons 214.

[0059] In FIG. 6 and FIG. 7 is shown an example embodiment 250, 270 of a 2x2 photonic memristive MZI synapse between each layer incorporating photonic memristors as part of silicon photonic waveguides in MZI. In particular Phase Change Material (PCM) 252 is shown used as a photonic- memristive material serving as phase shifters in a Mach-Zehnder Interferometer (MZI) mesh, shown with contact material 254 and n+ 256 and n regions 258.

[0060] Wavelength Division Multiplexing (WDM) plays an important role in scalability to perform weighted addition computations, and WDM photonic tensor-core-decomposition allows scaling far beyond the typical limitations of photonic circuits. WDM enhances synaptic interconnect fanout by approximately 100 times with a limited number of waveguides and allows individual assignment of weight values on each wavelength, while supporting a parallel “addition” operation by routing parallelized WDM signals onto a single photodetector, which outputs an electrical current representing the sum of total power. The use of photonic tensor core decomposition can emulate ‘sparse’ synaptic interconnection of the brain to realize interconnection of a large number of neurons (e.g., 1024) with orders of magnitude fewer synapses (e.g., about 582 times fewer) while maintaining high accuracies (e.g., 99%). In the brain, the sparsity contributes significantly to its compactness and energy-efficiency.

[0061] Recently we have also demonstrated a number of Machine-Learning (ML) applications including hand-writing recognition on the proposed EPICs, and this included benchmarking (energy-efficiency, throughput, and accuracy of various learning modalities) the photonic/optoelectronic neural networks guided by relating the neuron’s fan-in/out to quantum noise in the PIC network, and to the synaptic weight sensitivity. This PNN simulator is then used in the 3D EPIC simulator described below, which is utilized in the multi- 30 EPIC simulator, also described below. [0062] 2. 3D Integrated Neuromorphic Computing Circuits and Architecture

[0063] 2.1 . Requirement of Neuromorphic Computing

[0064] Neuromorphic computing requires extremely high scalability and large connectivity while minimizing energy consumption and size. Human brains are structured in 3D, and they consist of multiple regions interconnected with each other facilitating hierarchical learning. Previous attempts of neuromorphic computing research failed to reverse-engineer the brain due to lack of scalable and energy-efficient interconnecting circuits for brain-like structured hierarchical learning.

[0065] Objectives of the present disclosure include pursuing 3D EPICs that offer high density and high connectivity with extreme efficiency at scale, while supporting hierarchical learning in optical macro-circuits and electronic microcircuits. These designs will 3D integrate electronic neural networks and photonic neural networks together with learning algorithms into a 3D EPIC neural network towards human-like hierarchical learning capability. A simulator is being implemented for 3D EPICs in conducting simulation and experimental studies.

[0066] In the inventive 3D EPIC neural networks, the constituent electronic neural networks and photonic neural networks can work independently, in parallel, or in hierarchy. The optoelectronic neurons are essentially electronic neurons with photonic/electronic dendrite terminals (photodetectors) and electronic/photonic axon terminals (lasers or modulators); hence, the electronic and photonic neural networks can work together or separately depending on the electronic synaptic connections between the OE neurons and electronic neurons. Hence, the learning process (e.g. Hebbian learning) can take place across the electronic and photonic neural networks.

[0067] 2.1 .1 . The disclosed 3D EPIC pursues a new 3D platform that combines low-noise, scalability, wavelength-parallelism, high-throughput, and dynamic memristive plasticity of photonics, as well as high-density, agility, and dynamic plasticity of electronics. While the electronic neuromorphic computing approaches alone are unable to achieve the vast connectivity (e.g., approximately 8000 synaptic connections per neuron), at scale (e.g., billions of neurons), at the energy efficiency of the human brain and while the photonic neuromorphic computing approaches alone are unable to achieve the high density (100 billion neurons in a 3 liter volume), the 3D EPIC is expected to achieve brain-derived neuromorphic computing with performance, efficiency, and a density approaching that of a human brain.

[0068] 2.1.2. 3D EPICs built on brain-derived principles support scalable hierarchical learning networks with self-learning (e.g., predictive-error learning) resembling the human brain.

[0069] 2.1.3. 3D EPICs with photonic tensor cores also adopt sparsity of the human brain to further enhance scaling and energy-efficiency. It will be noted that the human brain’s energy efficiency and scalability are enhanced by pruning of synaptic interconnects through iterative predictive-learning processes and empirical experiences.

[0070] 2.2. Technical Approach

[0071] The procedure for creating the 3D EPIC neuromorphic computing system includes efforts in the following three areas: (1 ) hardware design, fabrication and integration; (2) establishing the learning process; and (3) simulator development.

[0072] 2.2.1. 3D EPIC Hardware Design, Fabrication, and Integration

[0073] FIG. 8 through FIG. 12 illustrate a neuromorphic system having a hybrid neuromorphic computing platform in a stacked 3D hierarchy.

[0074] In FIG. 8 is shown an example embodiment 310 of a wafer-scale nano- optoelectronic neural network intimately integrated with heterogeneous nanoelectronic neural networks over an electronic neural network die. Each of the nanoelectronic neural networks are in each section (approximately 5 mm by 5 mm) on the wafer-scale nano-optoelectronic neuromorphic processor. Each section is interconnected to each other via M x N nanophotonic synaptic interconnects (e.g., with M and N each having approximately 1000, or greater interconnects). Each wafer is 3D stacked utilizing 3D optical vias (TSOVs). Each TSOV can potentially carry at least 100 wavelengths, potentially achieving the ability to support thousands of optical interconnections, or more neurons from each neuron on multiple TSOVs.

[0075] The layers in this example are depicted as a Meta data plane 312 forming a type of artificial hippocampus based on memristive materials (e.g. resistive RAM), beneath which are shown multiple (e.g., four) hybrid neural net planes 314, 316, 318 and 320. It should be appreciated that each plane can be heterogeneously different, as is noted by HNN Plane3 316. Each of the neural net planes can consist of the combined photonic and electronic neural networks each conducting the required neuromorphic processing.

[0076] In FIG. 9 is shown an example embodiment of optical connections between the layers shown using TSOVs 330. In the figure can be seen a top Si waveguide (WG) 332 with a 45 degree reflector 334, an interlayer Si WG 336 and a bottom Si WG 338.

[0077] In FIG. 10 is shown an example embodiment 350 of hybrid nano- electronic-photonic neuromorphic cores 352, exemplified having multiple (e.g., four in this example) nano-electronic neural network cores 354 (shown in detail in FIG. 11 , and connections 356. It will be noted that the perimeter connections 356 provide for interconnecting multiple neural networks or cores to form larger neural networks. In the lower portion of the figure, one of these cores 352 is shown connecting through nanophotonic memristive wavelength routing synaptic interconnects 360 into a wafer scale neuromorphic computing plane 358.

[0078] In FIG. 11 is shown an example embodiment 354 of a nano-electronic neural network core, and depicting a portion 360 of that core with its electronic ionic synapses, shown with gate 362, channel 364 and write signal 366 and read signal 368.

[0079] In FIG. 12 is shown an example embodiment 390 of the nano- optoelectronic neural network core on silicon CMOS photonics, with an exploded view 392 of the photonic memristive synapses. In this view is seen a side view of a stack of these cores 352. It should be noted that all of the cores are interconnected with each other.

[0080] Each plane of the hybrid neuromorphic computing platform consists of the wafer-scale nano-optoelectronic neuromorphic macro-circuits intimately integrated with heterogeneous nanoelectronic neuromorphic micro-circuits integrated vertically by Back-End-Of-Line (BEOL) processes. Recently, BEOL compatible wide band gap semiconducting oxide transistors have been reported to have ultra-low off-state leakage current of less than 10 fA/pm, which is more than 10,000 times lower than that of silicon transistors, enabling learning and inference without relying on external non-volatile memories.

[0081] Each electronic plane links to each section of the nano-optoelectronic macro-circuit, and multiple such sections are interconnected to each other in parallel wavelengths using an M x N photonic wavelength routing WDM synaptic interconnects. The resulting hybrid neural network planes are vertically stacked in 3D utilizing TSOVs. Such wavelength routing capabilities and 3D photonic-electronic hierarchical circuits enable the neuromorphic computing networked system to support hierarchical learning.

[0082] The following describes additional aspects of this brain-derived, hierarchical, and reconfigurable neuromorphic computing system architecture co-designed with intelligent materials, devices, and learning algorithms. The disclosed brain-derived neuromorphic system consists of the hybrid neuromorphic computing platform shown in FIG. 8 through FIG. 12 operates according to the following biological principles. (1 ) Each electronic crossbar implements the learnable synapses for one projection within the hierarchical network architecture, with N sending neurons and M receiving dendritic branches. (2) The integrated conductance for each dendritic branch per neuron is then integrated according to configurable nonlinearities, to be determined by algorithm research, resulting in a net conductance contribution to the soma, which integrates this over time to produce the somatic membrane potential, Vm. (3) The somatic V_m value then drives discrete spiking via threshold-crossing, which initiates broadcast of the spike to all relevant electronic crossbar circuits. (4) In parallel, the optical interconnect broadcasts shortcut connections widely across the PNN using high-bandwidth WDM or optical couplers.

[0083] FIG. 13 through FIG. 14 illustrate an example embodiment of the hierarchical photonic and electronic neural network.

[0084] In FIG. 13 is shown an example logical hierarchical network 410 of photonic and electronic neural networks depicted in two different hierarchical planes 412, 414, although they are physically in the same hybrid plane, as seen in FIG. 14. It should be appreciated that the neurons are oblivious to the two different physical neural networks, photonic neural networks will achieve long-range communications between the remote neurons at far lower energy, latency, and noise. The electronic neurons include electronic (e.g., ionic polymer) dendrites with or without nano-photodetector receptors (excitatory or inhibitory), electronic axons with axon terminals with or without nanolasers, so that electronic somas can interface either with multiple planes of the ionic memristive synaptic interconnection networks or with photonic synaptic interconnection networks. The interface to the photonic synaptic interconnection networks includes photonic dendrites that are optical couplers to the photonic transmitters (nano-lasers) with wavelength multiplexers or simple optical power couplers. The ionic dendrite trees achieve dendrite computing, essential for low-energy, high-density, and scalable computing.

[0085] An optoelectronic neuron example was shown in FIG. 1 and FIG. 2 having a simple leaky-integrate-and-fire (LIF) electronic soma. The photonic synaptic interconnection networks and photonic neurons constitute photonic spiking neural networks. The proposed hierarchical neural network achieves symbiotic and synergistic neuromorphic computing across photonic and electronic neural networks.

[0086] As exemplified in FIG. 2, nonlinear and linear dynamics of all elements can utilize Verilog-A and HSPICE modeling for co-design of photonic, electronic, and ionic components. In particular, interfacing between nanotransistors (FETs) and photonics can be realized by building the photonic Verilog-A models and importing them into electrical circuits on HSPICE simulations. This optoelectronics-based modeling technique can bridge the OE conversion and simulate soma and dendrites’ space and time information. In the disclosed electronic neural networks, models are included based on advanced low-noise FETs with full Process Design Kits (PDKs) capable of conducting HSPICE modeling for abstraction for large-scale circuit modeling.

[0087] Although photonic neural networks and electronic neural networks can operate independently, it should be appreciated that electronic-only neural networks cannot scale without excessive noise and power-penalty by themselves, while photonic-only neural networks cannot achieve high density or easily include time-delays. [0088] FIG. 14 illustrates 510 a plane of a 3D EPIC consisting of wafer-scale nano-optoelectronic neuromorphic macro-circuits which are intimately integrated with heterogeneous nanoelectronic neuro-morphic micro-circuits.

[0089] The platform already contains the auxiliary silicon CMOS circuits and silicon photonics incorporating dynamic photonic materials, and BEOL postfabrication on top of the metal layer to allow integration of multiple layers of electronic neural networks consisting of dynamic electronic/ionic materials and devices.

[0090] The figure depicts these layers as being a handling wafer layer 512 over which is box layer 514 upon which is the silicon photonic and FET layer 516, shown toward the right side with fabricated N and P MOSFETs 532. In addition, in this layer is formed photonic memristive devices 524.

[0091] Now describing the figure from left to right, is seen a fabricated TSV 520, a silicon photonic waveguide 522, photonic memristive devices 524, one of multiple “3D” TSOVs 526, a nano-laser 528, a nano-detector 530, and FeFET/ionic memristive devices 534.

[0092] Each wafer can be 3D stacked utilizing 3D TSOVs which achieve low- loss vertical coupling between photonic layers to complete 3D EPIC hybrid neural networks such as depicted in FIG. 8.

[0093] 2.2.2. Learning algorithm integration with 3D EPIC: On-Line, Self-

Supervised Learning in hierarchical 3D EPIC Neural Networks

[0094] Neuromorphic learning algorithms require only information that is local to a given synaptic connection. Extended Contrastive Attractor Learning (XCAL), and other functionally equivalent learning rules, have been proven to be equivalent to error-driven learning in recurrent neural networks. The XCAL rule is technology-agnostic and allows end-to-end training of a neural network using only the activity of the sending and receiving neuron for each weight w = ((xy - 0_p) if xy > 9_p 9_d

= — xy(l — 9_d)/9_d otherwise) where 9_p is a floating threshold parameter, 9_d is a constant, and xy represents a moving average of activity for the sending and receiving neuron respectively. All weight updates can be calculated in parallel without the complexity and explicit knowledge of any gradients within the network. [0095] FIG. 15A and FIG. 15B illustrate an example embodiment of a neural network being trained 610 in a supervised manner using a single feedforward and feedback pathway between each layer. The figure depicts the following elements. Optical inputs 611 configured for receipt at an optical neural input layer 612 coupled to a W1 layer 614 with MZI mesh, whose output are connected through another neural layer 616, which outputs to a W2 layer 618 in FIG. 15B, which is connected to an output neural layer 620, from which optical outputs 622 are generated. The self learning process involves these elements and interactions, so that predictive error driven learning can take place.

[0096] More particularly, an optical matrix fabric 614, 618 (W1 and W2 layers) is shown comprising Mach-Zehnder interferometer meshes. Optical inputs are seen 652 are seen from matrix ~W1 644 to the optical inputs of the input neurons 612, which provide spike traces as pre-synaptic input 654 to matrix 646, which also receives post synaptic input 640, and parallel updates 642. Matrix 646 receives pre and post synaptic inputs 654, 640, as well as parallel updates 642, and generates outputs 650 to matrix ~W1 644 as well as outputs 648 to matrix fabric W1 614. Output of W1 614 is to optoelectronic neurons 616, which receive optical inputs 636, and provide post synaptic feedback 640, and laser optical output 638 to ~W1 644.

[0097] Optoelectronic neurons 616 are connected to the structures in FIG. 15B to matrix fabric 618 which is connected to a layer of optoelectronic neurons 620, which output 626 signals from the post-synaptic neurons to matrix 630. Output 622 are waveguides carrying the outputs from the layer of optoelectronic neurons 620, and which are also connected via parallel feedback pathways 624 to matrix 628 which updates the weight values taking into account differences 634 in the weight values and computing the phase values; with matrix 628 outputting parallel feedback pathways 636 to presynaptic pathways. Matrix 630 communicates 640 bidirectional signals between presynaptic neurons and synapses for updating weight values, and performs parallel updates 642, while also generating outputs 632 for updating weight matrix W2 618.

[0098] A training sample is first fed to the network from the input layer 612 and the network activity is allowed to stabilize. Next, the desired output activity is forced onto the output layer and network activity settles into a new pattern. Differences in activity over time pair with the XCAL learning rule to drive error- driven learning.

[0099] In human brains, the observation of causal outcomes in the environment follows this same general pattern and allows for predictive, error- driven learning. Typical interactions with the environment occur significantly slower than the reconfiguration speed of optoelectronics. As such, weight updates can be determined in an abstract manner and then quickly mapped to the implemented hardware to form intelligent devices. In this figure electrical signals carry spike timing information from optoelectronic neurons to an XCAL arithmetic control unit which maps weight updates onto the phase-shifters for each Mach-Zehnder interferometer of the local mesh.

[0100] 2.2.3. Comprehensive 3D EPIC Neuromorphic Computing Simulators

[0101] A comprehensive 3D EPIC simulator constructed based on the electronic and photonic simulators of the previous tasks will help design the neuromorphic computing system of the present disclosure. Since at least one simulator addresses every level of bio-plausible neural networks (neurons and synapses, circuits, micro networks, and structured networks involving hippocampus, prefrontal cortex, and so forth), this task pursues the creation of a similar comprehensive simulator for 3D EPIC neuromorphic computing systems where each 3D EPIC can emulate hippocampus, prefrontal cortex, and so forth, with neurons, synapses, and circuits created by dynamic photonic, electronic, and ionic materials. In addition, this process will further simulate learning processes and conduct experiments on 3D EPICs, test hierarchical learning algorithms, and conduct benchmarking studies (energyefficiency, throughput, accuracy, and so forth) for various Al applications.

[0102] 3. Examples

[0103] 3.1. Example 1 : Interlayer Coupling Performance Using Different Chip

Bonding Strategies

[0104] Efforts towards realizing 3D Electronic Integrated Circuits (3D EICs) have been propelled by the desire to reduce lengthy lateral electrical wires used for interconnecting various circuits and modules laid out in 2D and to circumvent the shore-bandwidth I/O bottlenecks seen in typical 2D EICs. Moving to a 3D structure allows EICs to achieve: (a) lower power consumption wherein short wiring with lower capacitance and resistance reduces the skin-effect RF loss and the need for repeaters and equalizers, (b) lower noise wherein shorter interconnects with shorter capacitance provides less noise, less jitter with fewer repeaters, (c) high packing density for a given footprint, and (d) higher I/O bandwidth overcoming the shore-bandwidth limitations at the perimeter of the die. However, these 3D EICs fundamentally rely on copper electrical TSVs between the layers and pBumps at the bottom that limit the bandwidth and energy-efficiency. For instance, 12 stacks of 12- Hi HBM2 must time-multiplex and share the bandwidth offered by a given set of TSVs (hence 1/12 bandwidth or 12x latency per stack) and are subject to the electrical parasitics of the package.

[0105] Recently, the first silicon photonic TSVs have been proposed and demonstrated for 3D Electronic Photonic Integrated Circuit (3D EPIC) applications. Utilizing such optical TSVs can potentially revolutionize heterogeneous chiplet integration on optoelectronic interposers by offering high-bandwidth and wavelength routing by photonics and power and control signal delivery by electronics utilizing TSVs.

[0106] FIG. 16 and FIG. 17 illustrate example embodiments of 3D EPIC circuits utilizing optical TSVs and electrical TSVs assuming each layer to be either a monolithic silicon CMOS-photonic die (e.g. GF45SPCLO) or a 3D Direct Bond Interconnect (DBI) bonded ElC-Photonic Integrated Circuit (PIC) layer. In FIG. 16 is seen a first view 710 showing DRAM layers 712, optical TSVs 714, optical fiber array Input/Outputs (I/O) 716, a processor layer 718 and TSVs 720. In FIG. 17 is seen a second embodiment 810 showing optical TSVs 714, optical fiber array Input/Outputs (I/O) 716, a processor layer 718 and TSVs 720, SiPh PDs 722, and SiPH modulators 724, and routing layer 726.

[0107] FIG. 18 illustrates an example embodiment 910 of heterogeneous 3D EPIC chiplets 914, 916 integrated on a reconfigurable optoelectronic interposer 912. Recent publications have emphasized the impact of 3D EPICs and optical switches in energy-efficient and high-throughput reconfigurable computing.

[0108] 3.2. Example 2: Interlayer Coupling Performance Using Different Chip

Bonding Strategies

[0109] Interlayer couplers are typically required to transmit the light vertically between the waveguides in different layers. Therefore, an interlayer coupler with compact size, low-loss and high integration compatibility is highly desired.

[0110] FIG. 19A and FIG. 19B illustrate embodiments 1010, 1030 using flip chip bonding and chip stacking to realize chip-to-chip coupling. As illustrated in FIG. 19A, flip chip bonding uses either two upside-up grating couplers 1012, 1014 or two Through-Silicon Optical Vias (TSOVs) 1016, 1018. In contrast, chip stacking as illustrated in FIG. 19B requires one upside-up grating coupler 1032 and one upside-down grating coupler 1034.

[0111] 3.3. Example 3: Design, Fabrication and Testing of Two Types of

Through-Silicon Optical Vias (TSOVs)

[0112] The key enablers for 3D Photonic Integrated Circuitries (PICs) include optical TSVs and other means to realize 3D photonic interconnections capable of achieving higher throughput and energy-efficiency compared to 3D Electronic Integrated Circuitries (EICs) relying on conventional electrical TSVs. There are two main methods, one with paired gratings and another with paired reflectors.

[0113] 3.3.1. Optical-TSVs with Paired-Gratings

[0114] Perhaps the simplest method to achieve the near-vertical optical coupling is by using custom-designed grating couplers using Process-Design- Kits (PDKs) from commercial silicon photonic foundries. This disclosure has introduced "upside-up" and "upside-down" grating couplers constructed using a custom-etched silicon with a poly-silicon overlay that is compatible with foundry processes, including with the GF45SPCLO foundry process.

[0115] To optimize both the grating emission and grating-to-grating coupling efficiency, the duty cycles for the etched silicon and poly-silicon overlay were simultaneously varied. In at least one embodiment, the grating period and length were intentionally optimized to 0.7 pm and 4.9 pm to maintain the same emission angle of approximately 17° as the upside-up gratings. However, this resulted in a slightly lower downward efficiency of approximately 0.6.

[0116] FIG. 20A and FIG. 20B as well as FIG. 21 A and FIG. 21 B provide schematics 1050, 1110 for the interlayer couplings. In FIG. 20A is shown this interlayer coupling between two upside-up grating couplers 1052a, 1052b, shown with offset 1054. In FIG. 20B is shown 1070 a plot of coupling efficiency with respect to offset, showing highest coupling efficiency in the range between 1.5 and 2 microns of offset. In FIG. 21 A is shown 1110 this interlayer coupling between one upside-up 1102a and one upside-down 1102b grating couplers, with offset 1104. In FIG. 21 B is shown 1130 a plot of coupling efficiency with respect to offset, showing highest coupling efficiency at about 1 .5 microns of offset.

[0117] Calculations of coupling efficiency under various grating misalignments indicate an ability to achieve a loss of 3.93 dB between two upside-up grating couplers and 5.38 dB between an upside-up and an upside-down grating coupler. Measurement results reveal a grating-to-grating coupling loss of 2.5 dB between the gratings for interlayer coupling between the upside-up gratings. These findings emphasize the significant potential in the development of compact and low-loss grating-based optical TSVs.

[0118] 3.3.2. Optical-TSVs with Paired-45 Degree Mirrors

[0119] Recent demonstrations of optical TSVs offer new solutions to 3D EPICs.

[0120] FIG. 22A and FIG. 22B illustrate 1150, 1170 in-depth depictions of an optical TSV comprising two 45° reflectors 1154a, 1154b meticulously connecting to distinct silicon waveguiding layers 1152a, 1152b, in addition to a pivotal inter-layer vertical optical via 1156. Notably, in at least one embodiment the choice of this silicon waveguide thickness in FIG. 22A is compatible with the GF45SPCLO foundry process (preferred range of approximately 220 to 500 nm thickness). In FIG. 22B a Finite-Difference Time-Domain (FDTD) simulation was utilized to facilitate a visual representation 1170 of light transmission from the lower silicon layer to the upper tier. When the design parameters are fine-tuned and optimized, the U- shaped coupler demonstrates the potential to attain an impressively minimal coupling loss of a mere 1 .42 dB even for 10 pm vertical distance. [0121] FIG. 23 visually demonstrates 1190 the final fabricated vertical U- shaped structures 1190, revealing the results after the partial removal of the oxide cladding. Transmission loss of cascaded vertical U-shaped couplers with different offsets was measured to achieve a minimum loss of 8.4 dB per coupler at an offset of 130 nm (bottom) and 425 nm (top).

[0122] 3.4. Example 4: Experimental Results

[0123] Using the configuration of FIG. 22A, the gap between the two TSOVs was set to 10 pm, as seen in FIG. 22B. The structural parameters were optimized to increase the interlayer coupling efficiency, with the results noted in FIG. 22B for a gap of 10 pm, a WG width of 1 .934 pm, a via width of 1 .717 pm, d=0.33 pm, and an offset of 0.203 pm, for which the loss was simulated as 1.42 dB.

[0124] FIG. 24 illustrates an example 1250 for the GDS layout of a TSOV structure generated by GlobalFoundries PDK. Since TSOVs will require postprocessing for 45-degree etching, upside-up and upside-down grating couplers were utilized for interlayer coupling in this initial demonstration. The figure depicts the TSOV structure with a tapered section 1252, leading to a straight waveguide (WG) section 1254 whose end 1256 provides for 45 degree etching, and this is within layers 1258 that prevent metal fills. By way of example, and not limitation, the taper is shown being approximately 100 pm in length and being about 350 nm at the narrowest point, with the WG end of a diameter of 1 .868 pm. The layer to prevent metal fills 1258 is shown with a length of 100 pm, and a thickness of 10 pm.

[0125] FIG. 25 illustrates 1310 an integrated structure designed to comprise a TSOV and a grating coupler to facilitate an easy transfer from grating couplers to TSOVs for interlayer coupling once the fabrication process for 45-degree etching is ready. This figure depicts a tapered section 1312 which is 350 nm at the narrowest point, and progresses to larger diameters along 40 pm, into a layer 1315 configured to prevent metal fills, a first end of which comprises a straight WG for the 45 degree etching 1314 (showing alignment marks). Within layer 1315, the WG then has a taper 1318, leading to a section 1322 having grating coupler 1324.

[0126] FIG. 26 illustrates an example GDS design 1350 after optimization of the coupler 1352 discussed above, with bonding pads 1356, were prepared having different test structures. By way of example and not limitation, the GDS structure is shown as being 1 .5 x 1.5 mm. Test structures with different numbers of couplers were arranged on the layout to estimate interlayer coupling efficiency. As the distance between the two grating couplers is uncertain and the emitting direction of grating is not vertical, the grating couplers with different misalignment were prepared as well. Several alignment marks 1358 and Vernier scales 1354 were placed in the layout to help the alignment during the chip bonding and estimate the misalignment between different layers. In this example the bonding pads 1356 are exemplified as being 100 pm square, with the alignment marks 1358 of the same size.

[0127] 3.5. Example 5: Photonic TSV Fabrication and 3D EPIC Integration

[0128] 3.5.1 . Back Substrate Removal of Global Foundry (GF) Die for

Grating-to-Grating Coupling Experiment

[0129] One of the primary aims of this disclosure is to provide grating-to- grating coupling utilizing two Global Foundry (GF) dies. One die will be positioned upside down after the removal of its back substrate, while the other GF die will be optically coupled by placing it atop the inverted first die. Consequently, this necessitates the availability of a GF die devoid of its back substrate.

[0130] To achieve the removal of the back substrate from a GF die, a specific procedure was followed. In at least one embodiment, this process initially reduces the thickness of the back substrate, for example from 685 pm to 270 pm and the dimension of the die was reduced to 1.5 mm by 5 mm through mechanical polishing. Subsequently, a BCB solution is employed to bond the thinned substrate in an upside-down orientation to a handling substrate. Finally, the removal of the back substrate is accomplished using XeF2 gas.

[0131] 3.5.2. Wafer Reconstitution of GF Dies on Handle Wafers

[0132] To reconstitute a wafer from a Global Foundry (GF) die, an intricate procedure was employed. Initially, the GF die was affixed to a 6-inch handle wafer using a Benzocyclonbute (BCB) solution. This solution, comprised of a BCB and mesitylene solvent at a ratio of 1 :10, underwent a meticulous preparation process. The prepared solution was spun onto the wafer using a spin-coating technique (spread at 5000 RPM for 5 seconds and spun at 3000 RPM for 40 seconds) followed by baking at 115 °C. Subsequently, the mechanically thinned GF die was bonded to the handle wafer. The confirmation of successful bonding was validated by assessing the infrared camera image.

[0133] A deposition process ensued, commencing with the application of an 8 pm layer of SiO2 on the bonded die-to-wafer structure through the HDPCVD method. Additionally, a 200 nm layer of SiN was deposited as a protective coating, employing the same HDPCVD technique. Subsequent steps involved spinning photoresist on top of the die-to-wafer bonding sample, achieving a thickness of 3.5 pm. Precise removal of the photoresist solely from the top of the die was executed using clean room swabs and IPA (isopropyl alcohol). This was followed by the etching of SiN and oxide layers from the die's surface via inductively coupled plasma.

[0134] Presently, the ongoing process involves the utilization of XeF2 gas for the removal of the GF die's substrate, marking a crucial step in this reconstitution procedure.

[0135] 3.6. Example 6: 3D EPIC by Interlayer Coupling Using Upside-Up and

Upside-Down gratings

[0136] FIG. 27 illustrates an example embodiment 1370 demonstrating an interlayer coupling using an upside-up grating 1374 and an upside-down grating 1372 based on chip stacking. It can be seen that SMF-28® optical fiber sections from Corning® are coupled at each end 1372, 1376. Given that the chip stacking technology is still in the developmental phase for interlayer coupling demonstrations, the two Global Foundries (GF) chips were manually aligned to assess the coupling performance between upside-up and upsidedown gratings. The chip was fabricated with upside-up gratings. Regarding the upside-down gratings, the substrate was removed from another identical chip to unveil the upside-down gratings on the bottom side. A measurement setup was fabricated for assessing interlayer coupling using upside-up and upside-down gratings. To achieve precise alignment of the two grating couplers, two 5-axis stages were utilized to accurately calibrate the angles of two GF chips and minimize the air gap between them. Following the same alignment procedure detailed in the prior report, visible and infrared cameras from different perspectives were utilized to optimize grating-to-grating coupling. However, the side view from the microscope image revealed a bending issue on the surface of the GF die with substrate removal. This poses a challenge in bringing the upside-up and upside-down gratings into close proximity for characterizing their coupling efficiency. Therefore, further refinement of the substrate removal process on the GF chip would be appropriate.

[0137] 4. General Scope of Embodiments

[0138] Embodiments of the present technology may be described herein with reference to flowchart illustrations of methods and systems according to embodiments of the technology, and/or procedures, algorithms, steps, operations, formulae, or other computational depictions, which may also be implemented as computer program products. In this regard, each block or step of a flowchart, and combinations of blocks (and/or steps) in a flowchart, as well as any procedure, algorithm, step, operation, formula, or computational depiction can be implemented by various means, such as hardware, firmware, and/or software including one or more computer program instructions embodied in computer-readable program code. As will be appreciated, any such computer program instructions may be executed by one or more computer processors, including without limitation a general purpose computer or special purpose computer, or other programmable processing apparatus to produce a machine, such that the computer program instructions which execute on the computer processor(s) or other programmable processing apparatus create means for implementing the function(s) specified.

[0139] Accordingly, blocks of the flowcharts, and procedures, algorithms, steps, operations, formulae, or computational depictions described herein support combinations of means for performing the specified function(s), combinations of steps for performing the specified function(s), and computer program instructions, such as embodied in computer-readable program code logic means, for performing the specified function(s). It will also be understood that each block of the flowchart illustrations, as well as any procedures, algorithms, steps, operations, formulae, or computational depictions and combinations thereof described herein, can be implemented by special purpose hardware-based computer systems which perform the specified function(s) or step(s), or combinations of special purpose hardware and computer-readable program code.

[0140] Furthermore, these computer program instructions, such as embodied in computer-readable program code, may also be stored in one or more computer-readable memory or memory devices that can direct a computer processor or other programmable processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory or memory devices produce an article of manufacture including instruction means which implement the function specified in the block(s) of the flowchart(s). The computer program instructions may also be executed by a computer processor or other programmable processing apparatus to cause a series of operational steps to be performed on the computer processor or other programmable processing apparatus to produce a computer- implemented process such that the instructions which execute on the computer processor or other programmable processing apparatus provide steps for implementing the functions specified in the block(s) of the flowchart(s), procedure (s) algorithm(s), step(s), operation(s), formula(e), or computational depiction(s).

[0141] It will further be appreciated that the terms "programming" or "program executable" as used herein refer to one or more instructions that can be executed by one or more computer processors to perform one or more functions as described herein. The instructions can be embodied in software, in firmware, or in a combination of software and firmware. The instructions can be stored local to the device in non-transitory media, or can be stored remotely such as on a server, or all or a portion of the instructions can be stored locally and remotely. Instructions stored remotely can be downloaded (pushed) to the device by user initiation, or automatically based on one or more factors.

[0142] It will further be appreciated that as used herein, the terms processor, hardware processor, computer processor, central processing unit (CPU), and computer are used synonymously to denote a device capable of executing the instructions and communicating with input/output interfaces and/or peripheral devices, and that the terms processor, hardware processor, computer processor, CPU, and computer are intended to encompass single or multiple devices, single core and multicore devices, and variations thereof.

[0143] From the description herein, it will be appreciated that the present disclosure encompasses multiple implementations of the technology which include, but are not limited to, the following:

[0144] An artificial intelligence apparatus, comprising: (a) photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (b) electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (c) interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits; and (d) hybrid neuromorphic computing circuits realized by hybrid or hierarchical operation of the photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

[0145] An artificial intelligent apparatus, comprising a plurality of modules of electronic neuromorphic computing circuits and photonic neuromorphic computing circuits wherein each module specializes in artificial intelligence functions, mimicking those of hippocampus, thalamus, and others in a biological brain through hardware architecture and topology and through the plasticity of the module by training.

[0146] An artificial intelligence machine, comprising: (a) photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (b) electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; and (c) interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

[0147] A method of creating an artificial intelligence machine, comprising: (a) providing photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (b) providing electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (c) providing interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits; and (d) providing hybrid neuromorphic computing circuits realized by hybrid or hierarchical operation of the photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

[0148] A method of creating an artificial intelligence machine, comprising: (a) providing photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; (b) providing electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses; and (c) providing interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

[0149] An artificial intelligent machine, comprising a plurality of modules of electronic neuromorphic computing circuits and photonic neuromorphic computing circuits wherein each module specializes in artificial intelligent functions, mimicking those of hippocampus, thalamus, and others in a biological brain.

[0150] An artificial intelligent apparatus, comprising a plurality of modules of electronic neuromorphic computing circuits and photonic neuromorphic computing circuits wherein each module specializes in artificial intelligent functions, mimicking those of hippocampus, thalamus, and others in a biological brain.

[0151] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and lasers.

[0152] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and optical modulators.

[0153] The apparatus or method of any preceding implementation, wherein the neural networks consist of electronic neural networks and photonic neural networks.

[0154] The apparatus or method of any preceding implementation, wherein the neural networks consist of electronic neural networks and photonic neural networks which can operate independently, in parallel, or in hierarchy. [0155] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons are electronic neurons with photonic/electronic dendrite terminals comprising photodetectors and electronic/photonic axon terminals comprising lasers or modulators.

[0156] The apparatus or method of any preceding implementation, wherein the electronic and photonic neural networks can operate together, or separately, depending on the electronic synaptic connections between the optoelectronic neurons and electronic neurons.

[0157] The apparatus or method of any preceding implementation, wherein training and inference processes in the artificial intelligence machine exploits hierarchical learning and inference processes of those in the photonic neuromorphic computing circuits and those in the electronic neuromorphic computing circuits.

[0158] The apparatus or method of any preceding implementation, wherein end-to-end learning processes exploit predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

[0159] The apparatus or method of any preceding implementation, wherein end-to-end learning processes exploit predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

[0160] The apparatus or method of any preceding implementation, wherein the photonic spikes are utilized in the photonic neuromorphic computing circuits and the electronic spikes are utilized in the electronic neuromorphic computing circuits for communications.

[0161] The apparatus or method of any preceding implementation, wherein multiple wavelengths are utilized for parallel interconnections between photonic neural networks.

[0162] The apparatus or method of any preceding implementation, wherein multiple wavelengths are used for parallel interconnections between optoelectronic neurons.

[0163] The apparatus or method of any preceding implementation, wherein photonic neuromorphic computing circuits and electronic neuromorphic computing circuits are integrated into a 3D electronic-photonic-integrated- circuit artificial intelligent machine.

[0164] The apparatus or method of any preceding implementation, wherein electronic neuromorphic computing circuits utilize ferroelectric transistors for electronic neuromorphic computing circuits.

[0165] The apparatus or method of any preceding implementation, wherein electronic neuromorphic computing circuits utilize resistive random access memory (RRAM) for electronic neuromorphic computing circuits.

[0166] The apparatus or method of any preceding implementation, wherein scalable interconnections of 100 billion neurons can be achieved utilizing the photonic and electronic interconnections.

[0167] The apparatus or method of any preceding implementation, wherein energy-efficiency comparable to human brain can be achieved.

[0168] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and lasers.

[0169] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and optical modulators.

[0170] The apparatus or method of any preceding implementation, wherein the neural networks consist of electronic neural networks and photonic neural networks.

[0171] The apparatus or method of any preceding implementation, wherein the neural networks consist of electronic neural networks and photonic neural networks which can operate independently, in parallel, or in hierarchy.

[0172] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons are electronic neurons with photonic/electronic dendrite terminals (photodetectors) and electronic/photonic axon terminals (lasers or modulators).

[0173] The apparatus or method of any preceding implementation, wherein the electronic and photonic neural networks can operate together, or separately, depending on the electronic synaptic connections between the optoelectronic neurons and electronic neurons. [0174] The apparatus or method of any preceding implementation, wherein training and inference processes in the artificial intelligence machine exploits hierarchical learning and inference processes of those in the photonic neuromorphic computing circuits and those in the electronic neuromorphic computing circuits.

[0175] The apparatus or method of any preceding implementation, wherein end-to-end learning processes exploit predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

[0176] The apparatus or method of any preceding implementation, wherein end-to-end learning processes exploit predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

[0177] The apparatus or method of any preceding implementation, wherein the photonic spikes are utilized in the photonic neuromorphic computing circuits and the electronic spikes are utilized in the electronic neuromorphic computing circuits for communications.

[0178] The apparatus or method of any preceding implementation, wherein multiple wavelengths are utilized for parallel interconnections between photonic neural networks.

[0179] The apparatus or method of any preceding implementation, wherein multiple wavelengths are used for parallel interconnections between optoelectronic neurons.

[0180] The apparatus or method of any preceding implementation, wherein photonic neuromorphic computing circuits and electronic neuromorphic computing circuits are integrated into a 3D electronic-photonic-integrated- circuit artificial intelligent machine.

[0181] The apparatus or method of any preceding implementation, wherein electronic neuromorphic computing circuits utilize ferroelectric transistors for electronic neuromorphic computing circuits.

[0182] The apparatus or method of any preceding implementation, wherein scalable interconnections of 100 billion neurons can be achieved utilizing the photonic and electronic interconnections. [0183] The apparatus or method of any preceding implementation, wherein energy-efficiency comparable to human brain can be achieved.

[0184] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and lasers.

[0185] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and optical modulators.

[0186] The apparatus or method of any preceding implementation, wherein the neural networks consist of electronic neural networks and photonic neural networks.

[0187] The apparatus or method of any preceding implementation, wherein the neural networks consists of electronic neural networks and photonic neural networks can work independently, in parallel, or in hierarchy.

[0188] The apparatus or method of any preceding implementation, wherein the optoelectronic neurons are essentially electronic neurons with photonic/electronic dendrite terminals (photodetectors) and electronic/photonic axon terminals (lasers or modulators).

[0189] The apparatus or method of any preceding implementation, wherein the electronic and photonic neural networks can work together or separately depending on the electronic synaptic connections between the optoelectronic neurons and electronic neurons.

[0190] The apparatus or method of any preceding implementation, wherein training and inference processes in the artificial intelligence machine exploits hierarchical learning and inference processes of those in the photonic neuromorphic computing circuits and those in the electronic neuromorphic computing circuits.

[0191] The apparatus or method of any preceding implementation, wherein end-to-end learning processes exploits predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

[0192] The apparatus or method of any preceding implementation, wherein end-to-end learning processes exploits predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

[0193] The apparatus or method of any preceding implementation, wherein the photonic spikes are used in the photonic neuromorphic computing circuits and the electronic spikes are used in the electronic neuromorphic computing circuits for communications.

[0194] The apparatus or method of any preceding implementation, wherein multiple wavelengths are used for parallel interconnections between photonic neural networks.

[0195] The apparatus or method of any preceding implementation, wherein multiple wavelengths are used for parallel interconnections between optoelectronic neurons.

[0196] The apparatus or method of any preceding implementation, wherein photonic neuromorphic computing circuits and electronic neuromorphic computing circuits are integrated into 3D electronic-photonic-integrated-circuit artificial intelligent machine.

[0197] The apparatus or method of any preceding implementation, wherein electronic neuromorphic computing circuits utilize ferroelectric transistors for electronic neuromorphic computing circuits.

[0198] The apparatus or method of any preceding implementation, wherein scalable interconnections on the order of 100 billion neurons can be achieved utilizing the photonic and electronic interconnections.

[0199] The apparatus or method of any preceding implementation, wherein energy-efficiency comparable to human brain can be achieved.

[0200] The apparatus or method of any preceding implementation, wherein the human like flexible learning and inference can be achieved.

[0201] Accordingly, by way of example and not of limitation, the technology of this disclosure generally comprises 3D electronic photonic integrated circuitry (3DEPIC) that combines electronic Neuromorphic circuits and photonic (optoelectronic) neuromorphic circuits.

[0202] As used herein, the term "implementation" is intended to include, without limitation, embodiments, examples, or other forms of practicing the technology described herein. [0203] As used herein, the singular terms "a," "an," and "the" may include plural referents unless the context clearly dictates otherwise. Reference to an object in the singular is not intended to mean "one and only one" unless explicitly so stated, but rather "one or more."

[0204] Phrasing constructs, such as “A, B and/or C”, within the present disclosure describe where either A, B, or C can be present, or any combination of items A, B and C. Phrasing constructs indicating, such as “at least one of” followed by listing a group of elements, indicates that at least one of these groups of elements is present, which includes any possible combination of the listed elements as applicable.

[0205] References in this disclosure referring to “an embodiment”, “at least one embodiment” or similar embodiment wording indicates that a particular feature, structure, or characteristic described in connection with a described embodiment is included in at least one embodiment of the present disclosure. Thus, these various embodiment phrases are not necessarily all referring to the same embodiment, or to a specific embodiment which differs from all the other embodiments being described. The embodiment phrasing should be construed to mean that the particular features, structures, or characteristics of a given embodiment may be combined in any suitable manner in one or more embodiments of the disclosed apparatus, system, or method.

[0206] As used herein, the term "set" refers to a collection of one or more objects. Thus, for example, a set of objects can include a single object or multiple objects.

[0207] Relational terms such as first and second, top and bottom, upper and lower, left and right, and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0208] The terms "comprises," "comprising," "has", "having," "includes", "including," "contains", "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, apparatus, or system, that comprises, has, includes, or contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, apparatus, or system. An element proceeded by "comprises... a", "has... a", "includes... a", "contains... a" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, apparatus, or system, that comprises, has, includes, contains the element.

[0209] As used herein, the terms "approximately", "approximate", "substantially", "substantial", "essentially", and "about", or any other version thereof, are used to describe and account for small variations. When used in conjunction with an event or circumstance, the terms can refer to instances in which the event or circumstance occurs precisely as well as instances in which the event or circumstance occurs to a close approximation. When used in conjunction with a numerical value, the terms can refer to a range of variation of less than or equal to ± 10% of that numerical value, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1 %, less than or equal to ±0.5%, less than or equal to ±0.1 %, or less than or equal to ±0.05%. For example, "substantially" aligned can refer to a range of angular variation of less than or equal to ±10°, such as less than or equal to ±5°, less than or equal to ±4°, less than or equal to ±3°, less than or equal to ±2°, less than or equal to ±1 °, less than or equal to ±0.5°, less than or equal to ±0.1 °, or less than or equal to ±0.05°.

[0210] Additionally, amounts, ratios, and other numerical values may sometimes be presented herein in a range format. It is to be understood that such range format is used for convenience and brevity and should be understood flexibly to include numerical values explicitly specified as limits of a range, but also to include all individual numerical values or sub-ranges encompassed within that range as if each numerical value and sub-range is explicitly specified. For example, a ratio in the range of about 1 to about 200 should be understood to include the explicitly recited limits of about 1 and about 200, but also to include individual ratios such as about 2, about 3, and about 4, and sub-ranges such as about 10 to about 50, about 20 to about 100, and so forth.

[0211] The term "coupled" as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

[0212] Benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or element of the technology described herein or any or all the claims.

[0213] In addition, in the foregoing disclosure various features may be grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Inventive subject matter can lie in less than all features of a single disclosed embodiment.

[0214] The abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

[0215] It will be appreciated that the practice of some jurisdictions may require deletion of one or more portions of the disclosure after the application is filed. Accordingly, the reader should consult the application as filed for the original content of the disclosure. Any deletion of content of the disclosure should not be construed as a disclaimer, forfeiture, or dedication to the public of any subject matter of the application as originally filed.

[0216] All text in a drawing figure is hereby incorporated into the disclosure and is to be treated as part of the written description of the drawing figure.

[0217] The following claims are hereby incorporated into the disclosure, with each claim standing on its own as a separately claimed subject matter.

[0218] Although the description herein contains many details, these should not be construed as limiting the scope of the disclosure, but as merely providing illustrations of some of the presently preferred embodiments. Therefore, it will be appreciated that the scope of the disclosure fully encompasses other embodiments which may become obvious to those skilled in the art.

[0219] All structural and functional equivalents to the elements of the disclosed embodiments that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element herein is to be construed as a "means plus function" element unless the element is expressly recited using the phrase "means for". No claim element herein is to be construed as a "step plus function" element unless the element is expressly recited using the phrase "step for".

Claims

CLAIMS What is claimed is:

1 . An artificial intelligence apparatus, comprising:

(a) photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses;

(b) electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses;

(c) interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits; and

(d) hybrid neuromorphic computing circuits realized by hybrid or hierarchical operation of the photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

2. The apparatus of claim 1 , wherein the optoelectronic neurons comprise photodetectors, nonlinear electronic circuits, and lasers.

3. The apparatus of claim 1 , wherein the optoelectronic neurons comprise of photodetectors, nonlinear electronic circuits, and optical modulators.

4. The apparatus of claim 1 , wherein the neural networks comprise of electronic neural networks and photonic neural networks.

5. The apparatus of claim 1 , wherein the neural networks comprise of electronic neural networks and photonic neural networks which can operate independently, in parallel, or in hierarchy.

6. The apparatus of claim 1 , wherein the optoelectronic neurons are electronic neurons with photonic/electronic dendrite terminals (photodetectors) and electronic/photonic axon terminals (lasers or modulators).

7. The apparatus of claim 1 , wherein the electronic and photonic neural networks can operate together, or separately, depending on the electronic synaptic connections between the optoelectronic neurons and electronic neurons.

8. The apparatus of claim 1 , wherein training and inference processes in the artificial intelligence apparatus utilizes hierarchical learning and inference processes of those in the photonic neuromorphic computing circuits and those in the electronic neuromorphic computing circuits.

9. The apparatus of claim 1 , wherein end-to-end learning processes utilize predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

10. The apparatus of claim 1 , wherein end-to-end learning processes utilize predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

11 . The apparatus of claim 1 , wherein the photonic spikes are utilized in the photonic neuromorphic computing circuits and the electronic spikes are utilized in the electronic neuromorphic computing circuits for communications.

12. The apparatus of claim 1 , wherein multiple wavelengths are utilized for parallel interconnections between photonic neural networks.

13. The apparatus of claim 1 , wherein multiple wavelengths are used for parallel interconnections between optoelectronic neurons.

14. The apparatus of claim 1 , wherein photonic neuromorphic computing circuits and electronic neuromorphic computing circuits are integrated into a 3D electronic-photonic-integrated-circuit artificial intelligent apparatus.

15. The apparatus of claim 1 , wherein electronic neuromorphic computing circuits utilize ferroelectric transistors for electronic neuromorphic computing circuits.

16. The apparatus of claim 1 , wherein electronic neuromorphic computing circuits utilize resistive random access memory (RRAM) for electronic neuromorphic computing circuits.

17. The apparatus of claim 1 , wherein scalable interconnections on the order of 100 billion neurons can be achieved utilizing the photonic and electronic interconnections.

18. The apparatus of claim 1 , wherein energy-efficiency comparable to a human brain may be achieved.

19. An artificial intelligent apparatus, comprising a plurality of modules of electronic neuromorphic computing circuits and photonic neuromorphic computing circuits wherein each module performs specialized artificial intelligence functions, mimicking those of hippocampus, thalamus, and others in a biological brain through hardware architecture and topology and through the plasticity of the module by training.

20. The apparatus of claim 19, wherein human-like flexible learning and inference can be achieved.

21 . A method of creating an artificial intelligence machine, comprising:

(a) providing photonic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses;

(b) providing electronic neuromorphic computing circuits comprising optoelectronic neurons and photonic synapses;

(c) providing interconnections between photonic neuromorphic computing circuits and electronic neuromorphic computing circuits; and

(d) providing hybrid neuromorphic computing circuits realized by hybrid or hierarchical operation of the photonic neuromorphic computing circuits and electronic neuromorphic computing circuits.

22. The method of claim 21 , wherein the optoelectronic neurons consist of photodetectors, nonlinear electronic circuits, and lasers.

23. The method of claim 21 , wherein the optoelectronic neurons comprise photodetectors, nonlinear electronic circuits, and optical modulators.

24. The method of claim 21 , wherein the neural networks comprise electronic neural networks and photonic neural networks.

25. The method of claim 21 , wherein the neural networks comprise electronic neural networks and photonic neural networks which can operate independently, in parallel, or in hierarchy.

26. The method of claim 21 , wherein the optoelectronic neurons are electronic neurons with photonic/electronic dendrite terminals (photodetectors) and electronic/photonic axon terminals (lasers or modulators).

27. The method of claim 21 , wherein the electronic and photonic neural networks can operate together, or separately, depending on the electronic synaptic connections between the optoelectronic neurons and electronic neurons.

28. The method of claim 21 , wherein training and inference processes in the artificial intelligence apparatus utilize hierarchical learning and inference processes of those in the photonic neuromorphic computing circuits and those in the electronic neuromorphic computing circuits.

29. The method of claim 21 , wherein end-to-end learning processes comprise predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

30. The method of claim 21 , wherein end-to-end learning processes comprise predictive-error learning across the photonic neuromorphic computing circuits and the electronic neuromorphic computing circuits.

31 . The method of claim 21 , wherein the photonic spikes are utilized in the photonic neuromorphic computing circuits and the electronic spikes are utilized in the electronic neuromorphic computing circuits for communications.

32. The method of claim 21 , wherein multiple wavelengths are utilized for parallel interconnections between photonic neural networks.

33. The method of claim 21 , wherein multiple wavelengths are utilized for parallel interconnections between optoelectronic neurons.

34. The method of claim 21 , wherein photonic neuromorphic computing circuits and electronic neuromorphic computing circuits are integrated into a 3D electronic-photonic-integrated-circuit artificial intelligent machine.

35. The method of claim 21 , wherein electronic neuromorphic computing circuits utilize ferroelectric transistors for electronic neuromorphic computing circuits.

36. The method of claim 21 , wherein scalable interconnections on the order of 100 billion neurons may be achieved utilizing the photonic and electronic interconnections.

37. The method of claim 21 , wherein energy-efficiency comparable to that of a human brain may be achieved.