CN111753977A - Optical neural network convolution layer chip, convolution computing method and electronic device - Google Patents

Abstract

An optical neural network convolution layer chip, which is applied to the field of artificial intelligence, includes a first coupler, a first beam splitter, a plurality of photon calculation modules and a convolution summation module connected in sequence; is used to couple the received optical signal into the first beam splitter; the first beam splitter includes a plurality of output ports, and the beam splitter is used to split the coupled optical signal to obtain multiple beams of optical signals. The optical signals are input to each of the photon calculation modules through each of the output ports; the photon calculation module is used to perform amplitude modulation and phase modulation on each beam of the optical signal, so that each beam of modulated optical signal represents an input data and a convolution kernel parameter, and convert all modulated optical signals into electrical signals; the convolution summation module is used to convolve and sum all the electrical signals to complete all input data and convolution kernel parameters The photon convolution operation. Photons have the characteristics of high speed, high bandwidth and low power consumption. Using photons to realize convolution computing can greatly improve the computing speed and reduce computing energy consumption.

Description

Optical neural network convolution layer chip, convolution computing method and electronic device

technical field

The present application relates to the field of artificial intelligence, and in particular, to an optical neural network convolution layer chip, a convolution computing method and an electronic device.

Background technique

A neural network refers to a series of mathematical models that are inspired by biology. In recent years, neural network technology has become an important driving force for the development of artificial intelligence technology, and is widely used in image processing, speech recognition, natural language processing and other fields. Convolution operation is an important type of computation in neural networks, especially in convolutional neural networks.

Chip technology provides powerful computing power for neural networks for intelligent processing, supporting the development of neural network technology. With the sharp increase in the amount of data to be processed and the more complex neural network models, the bottleneck of the von Neumann architecture of traditional electronic chip technology and the bottleneck of CMOS technology and devices have become more prominent, resulting in chip power consumption and performance improvement. Problems and so on restrict the application and popularization of neural network technology. Optical neural network chips provide a new solution for dedicated neural network accelerator chips. Compared with electrons, photons have the advantages of high speed, high bandwidth and low power consumption, so optical neural network chips can be more than ten times faster than traditional electrical chips. , the power consumption is about one tenth or less, and it has a good application prospect. By designing the optical neural network convolution layer chip, high-speed, low-power convolution operations can be realized, and the application and development of neural network technology can be promoted.

SUMMARY OF THE INVENTION

The main purpose of this application is to provide an optical neural network convolution layer chip, a convolution calculation method and an electronic device.

In order to achieve the above purpose, a first aspect of the embodiments of the present application provides an optical neural network convolution layer chip, including a first coupler, a first beam splitter, a plurality of photon calculation modules and a convolution summation module connected in sequence;

the first coupler for coupling the received optical signal into the first beam splitter;

The first beam splitter includes a plurality of output ports, and the beam splitter is used for splitting the coupled optical signals to obtain multiple beams of optical signals, and the multiple beams of the optical signals are input through each of the output ports one by one to each of the photon computing modules;

The photon calculation module is used to perform amplitude modulation and phase modulation on each beam of the optical signal, so as to represent an input data and a convolution kernel parameter through each beam of modulated optical signal, and convert all the modulated optical signals into is an electrical signal;

The convolution and summation module is configured to perform convolution and summation on all the electrical signals, and complete the photon convolution operation of all input data and convolution kernel parameters.

Optionally, the photon calculation module includes a second beam splitter, an input data modulation module, a convolution kernel parameter modulation module, a second coupler and a balanced photodetector connected in sequence;

The second beam splitter is used to divide the input optical signal into two beams of optical signals, the second beam splitter includes two output ports, one output port is connected to the input data modulation module, and is used to convert the input data into two beams. One of the optical signals is transmitted to the input data modulation module, and the other output port is connected to the convolution kernel parameter modulation module for transmitting the other optical signal to the convolution kernel parameter modulation module;

The input data modulation module is configured to perform amplitude modulation and phase modulation on one of the two beams of optical signals according to one input data, and represent the one input data by the modulated optical signal;

The convolution kernel parameter modulation module is configured to perform amplitude modulation and phase modulation on the other one of the two optical signals according to one convolution kernel parameter, and the modulated optical signal represents the one convolution kernel parameters;

the second coupler, configured to couple the optical signal modulated by the input data modulation module and the optical signal modulated by the convolution kernel parameter modulation module, and output two beams of optical signals;

The balanced photodetector is used for balanced detection of the two optical signals to obtain the electrical signal.

Optionally, the input data modulation module includes P1 optical amplitude modulation units and Q1 optical phase modulation units; the convolution kernel parameter modulation module includes P2 optical amplitude modulation units and Q2 optical phase modulation units;

Among them, P1, Q1, P2, Q2 are non-negative integers.

The optical amplitude modulation unit is used for amplitude modulation on the received optical signal; the optical phase modulation unit is used for phase modulation on the received optical signal.

Optionally, the optical signal is emitted by a laser light source;

The first coupler is an end face coupler or a grating coupler;

The second coupler is a 2*2 coupler;

The first beam splitter is a directional coupler or a multimode interference coupler.

Optionally, the optical amplitude modulation unit is a Mach-Zehnder interferometer, a micro-ring resonator or an optical waveguide deposited with a phase-change material;

The optical phase modulation unit is an electro-optic phase modulator or a thermo-optic phase modulator.

A second aspect of the embodiments of the present application provides an optical neural network convolution calculation method, including:

Coupling the received optical signal;

Splitting the coupled optical signals to obtain multiple optical signals;

performing amplitude modulation and phase modulation on each beam of the optical signal, so that one input data and one convolution kernel parameter are represented by each beam of the modulated optical signal;

Convert all modulated optical signals into electrical signals;

Perform convolution and summation on all the electrical signals to complete the photon convolution operation of all input data and convolution kernel parameters.

Optionally, performing amplitude modulation and phase modulation on each beam of the optical signal to represent one input data and one convolution kernel parameter by each beam of modulated optical signal includes:

Dividing each beam of the optical signal into two beams of optical signals to obtain multiple groups of two beams of optical signals;

In each group of two optical signals, amplitude modulation and phase modulation are performed on one of the two optical signals according to one input data, and the one input data is represented by the modulated optical signal, and, according to a convolution For the kernel parameter, amplitude modulation and phase modulation are performed on the other beam of the two optical signals, and the one convolution kernel parameter is represented by the modulated optical signal.

Optionally, before converting all modulated optical signals into electrical signals, the steps include:

The two beams of optical signals after amplitude modulation and phase modulation in each group are coupled to output two beams of optical signals.

Optionally, the converting all modulated optical signals into electrical signals includes:

Balance detection is performed on the two optical signals in each group to obtain the electrical signals.

A third aspect of an embodiment of the present application provides an electronic device, including an optical neural network convolution layer chip provided in the first aspect of an embodiment of the present application.

It can be seen from the above-mentioned embodiments of the present application that the optical neural network convolution layer chip, the convolution calculation method and the electronic device provided by the present application include a first coupler, a first beam splitter, a plurality of photon calculation modules and a volume connected in sequence. The product-summation module, the first coupler, is used for coupling the received optical signal into the first beam splitter, the first beam splitter includes a plurality of output ports, and the beam splitter is used for the coupled optical signal. Split beams to obtain multiple beams of optical signals, and the multiple beams of the optical signals are input to each of the photon calculation modules through each of the output ports, and the photon calculation module is used to perform amplitude modulation and phase modulation on each beam of the optical signal to pass Each modulated optical signal represents an input data and a convolution kernel parameter, and converts all modulated optical signals into electrical signals. The convolution and summation module is used to convolve and sum all the electrical signals, and complete Photon convolution operation for all input data and convolution kernel parameters. Photons have the characteristics of high speed, high bandwidth and low power consumption. Using photons to realize convolution computing can greatly improve the computing speed and reduce computing energy consumption.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic structural diagram of an optical neural network convolution layer chip provided by an embodiment of the application;

2 is a schematic structural diagram of a photonic computing module of an optical neural network convolutional layer chip provided by an embodiment of the application;

3 is a schematic diagram of the output of a photon computing module provided by an embodiment of the application when a convolution operation is performed;

FIG. 4 is a schematic flowchart of an optical neural network convolution calculation method provided by an embodiment of the present application.

Detailed ways

In order to make the application purpose, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. The embodiments described above are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of this application.

其中，W_uv为卷积核参数，X_{i-u+1，j-v+1}为输入数据，Y_ij为输出数据。Convolution operations exist in large numbers in convolutional neural networks, which are the multiplication and addition operations of input data and convolution kernel parameters, which can be expressed as

Among them, W _uv is the convolution kernel parameter, X _{i-u+1, j-v+1} is the input data, and Y _ij is the output data.

Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of an optical neural network convolution layer chip provided by an embodiment of the present application. A light source 1 emits an optical signal to the optical neural network convolution layer chip to realize convolution operation. The chip includes a first coupler 2, a first beam splitter 3, a plurality of photon calculation modules 4 and a convolution summation module 5 connected in sequence; the first coupler 2 is used to couple the optical signal to the first split beam The first beam splitter 3 includes a plurality of output ports, and the beam splitter is used to split the coupled optical signal to obtain multiple beams of optical signals, and the multiple beams of optical signals are input to each photon calculation through each output port one by one. Module 4; Photon calculation module 4, for performing amplitude modulation and phase modulation on each beam of optical signal, so as to represent an input data and a convolution kernel parameter through each beam of modulated optical signal, and convert all modulated optical signals into is an electrical signal; the convolution and summation module 5 is used for convolution and summation of all electrical signals to complete the photon convolution operation of all input data and convolution kernel parameters.

In this embodiment, the light emitted by the light source 1 is coupled into the first beam splitter 3 through the first coupler 2. The beam splitter has multiple output ports, which are connected to the multiple photon calculation modules 4 in one-to-one correspondence. The beam splitter 3 divides the input light into multiple beams, which are respectively input to each photon calculation module 4, and the photon product operation of an input data and a convolution kernel parameter is completed in the photon calculation module 4. The outputs of different photon calculation modules 4 are summed, and then the photon convolution operation between the input data and the convolution kernel parameters is completed. Photons have the characteristics of high speed, high bandwidth and low power consumption. Using photons to realize convolution calculation can greatly improve the Computing speed and reducing computing power consumption.

In one embodiment of the present application, as shown in FIG. 2 , FIG. 2 is a schematic structural diagram of a photonic computing module 4 of an optical neural network convolutional layer chip provided by an embodiment of the present application. The photonic computing module 4 includes a second A beam splitter 41, an input data modulation module 42, a convolution kernel parameter modulation module 43, a second coupler 44 and a balanced photodetector 45; the second beam splitter 41 is used to divide the input optical signal into two beams of light signal, the second beam splitter 41 includes two output ports, one output port is connected to the input data modulation module 42 for transmitting an optical signal to the input data modulation module 42, and the other output port is connected to the convolution kernel parameter modulation module 43 are connected, and are used to transmit another optical signal to the convolution kernel parameter modulation module 43; the input data modulation module 42 is used to perform amplitude modulation and phase modulation on one of the two beams of optical signals according to an input data. The modulated optical signal represents one input data; the convolution kernel parameter modulation module 43 is used to perform amplitude modulation and phase modulation on the other of the two optical signals according to a convolution kernel parameter, and the modulated optical signal Represents a convolution kernel parameter; the second coupler 44 is used to couple the optical signal modulated by the input data modulation module 42 and the optical signal modulated by the convolution kernel parameter modulation module 43, and output two beams of optical signals; balance photoelectric The detector 45 is used to perform balanced detection on the two beams of optical signals to obtain the electrical signals.

Specifically, the second beam splitter 41 divides the input optical signal into two beams of optical signals, one beam of optical signal passes through the input data modulation module 42, and the optical amplitude modulation unit and the optical phase modulation unit in the input data modulation module 42 perform amplitude modulation on the beam. Modulation and phase modulation, the positive and negative and the numerical value of the input data X _i are represented by optical information, and another beam of optical signal passes through the convolution kernel parameter modulation module 43, the optical amplitude modulation unit and the optical phase modulation in the convolution kernel parameter modulation module 43. The unit performs amplitude modulation and phase modulation on the light beam, and the positive and negative values and the numerical value of the _convolution kernel parameter Wi are represented by optical information. The modulated two beams of optical signals are coupled in the second coupler 44 to output two beams of optical signals, The balanced photodetector 45 performs balanced detection on the two beams of light signals to obtain the electrical signal, and the obtained electrical signal is related to the amplitude information and phase information of the output beam, and then completes the product operation W _i *X _i of the input data and the convolution kernel parameter and output.

There are convolution kernels of different sizes in convolutional neural networks, such as 1*1 convolution kernel and 3*3 convolution kernel. In the optical neural network convolution layer chip provided in the embodiment of the present application, different sizes of convolution kernel calculations can be realized through different combinations of adjacent photon computing modules 4, and multiple convolution kernels can be completed at one time through spatial multiplexing. calculation to improve computational efficiency.

Exemplarily, please refer to FIG. 3 , which is a schematic diagram of the output of a photon computing module 4 according to an embodiment of the present application when a convolution operation is performed. As shown in FIG. 3 , a photon calculation module 4 provided in the embodiment of the present application is a multiple of 9, and the calculation results of the adjacent 9 photon calculation modules 4 are summed and output to complete a 3*3 convolution kernel. For the convolution operation with the input data, through spatial multiplexing, the convolution operation of N/9 3*3 convolution kernels can be completed at one time, which improves the computational efficiency.

In one of the embodiments of the present application, the input data modulation module 42 includes P1 optical amplitude modulation units and Q1 optical phase modulation units; the convolution kernel parameter modulation module 43 includes P2 optical amplitude modulation units and Q2 optical phase modulation units an optical amplitude modulation unit for amplitude modulation of the received optical signal; an optical phase modulation unit for phase modulation of the received optical signal.

Among them, P1, Q1, P2, Q2 are all non-negative integers.

In one of the embodiments of the present application, the light source 1 is a laser light source; the first coupler 2 is an end face coupler or a grating coupler; the second coupler is a 2*2 coupler; the first beam splitter 3 is a directional coupler or multimode interference couplers.

In one of the embodiments of the present application, the optical amplitude modulation unit is a Mach-Zehnder interferometer, a micro-ring resonator or an optical waveguide deposited with phase change material; the optical phase modulation unit is an electro-optical phase modulator or a thermo-optical phase modulator.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of an optical neural network convolution calculation method provided by an embodiment of the present application. The method can be performed by the optical neural network convolution layer chip shown in the above-mentioned FIGS. 1 to 3. The method is mainly Include the following steps:

S101. Coupling the received optical signal;

S102, splitting the coupled optical signals to obtain multiple optical signals;

S103, performing amplitude modulation and phase modulation on each beam of the optical signal, so that an input data and a convolution kernel parameter are represented by each beam of the modulated optical signal;

S104, converting all modulated optical signals into electrical signals;

S105, perform convolution and summation on all the electrical signals, and complete the photon convolution operation of all input data and convolution kernel parameters.

It can be understood that step S101 is performed by the first coupler 2 , step S102 is performed by the first beam splitter 3 , step S103 is performed by the photon calculation module 4 , and step S105 is performed by the convolution and summation module 5 .

In one of the embodiments of the present application, step S104 includes: dividing each beam of the optical signal into two beams of optical signals to obtain multiple groups of two beams of optical signals; in each group of two beams of optical signals, according to an input data, for the One of the two optical signals is amplitude-modulated and phase-modulated, the one input data is represented by the modulated optical signal, and, according to a convolution kernel parameter, the other of the two optical signals is amplitude-modulated and summed. Phase modulation, the one convolution kernel parameter is represented by the modulated optical signal.

In one of the embodiments of the present application, before step S105 , the method includes: coupling each group of two optical signals after amplitude modulation and phase modulation, and outputting two optical signals.

In one of the embodiments of the present application, the step S105 includes: performing balanced detection on the two optical signals in each group to obtain the electrical signals.

Wherein, the optical signal is emitted by a light source, a light source with a suitable wavelength and a suitable power is selected, and the light is output to the first beam splitter through the first coupler.

Among them, the input data is preprocessed, the amplitude information and phase information of the light are used to represent the preprocessed input data, and the characteristics of the optical amplitude modulation unit and the optical phase modulation unit are used to calculate the required setting of the input data modulation module. parameters and set the parameters of the input data modulation module.

Among them, according to the application scenario, select the appropriate convolutional neural network model and complete the neural network training, use the amplitude information and phase information of the light to represent the parameters of the convolution kernel in the trained neural network, combine the optical amplitude modulation unit with the light The characteristics of the phase modulation unit are calculated, the parameters required to be set in the convolution kernel parameter modulation module are calculated and the parameters of the convolution kernel parameter modulation module are set.

More, through the stochastic gradient descent method, particle swarm optimization algorithm or other optimization algorithms, the parameter settings of the convolution kernel parameter modulation module can be adjusted, the output results of the optical neural network convolution layer chip can be optimized, and the optical neural network convolution layer can be improved. The calculation accuracy of the chip.

Embodiments of the present disclosure also provide an electronic device, which includes the above-mentioned optical neural network convolution layer chip.

The electronic equipment includes data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cloud servers, cameras, video cameras, projectors, watches, Headphones, mobile storage, wearable devices, vehicles, home appliances and/or medical equipment. The vehicles include airplanes, ships and/or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lamps, gas stoves, and range hoods; the medical equipment includes nuclear magnetic resonance instruments, B-ultrasound and/or electrocardiograph.

It should be noted that each functional module in each embodiment of the present disclosure may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules.

If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or in part that contributes to the prior art, or all or part of the technical solutions.

It should be noted that, for the convenience of description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. As in accordance with the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily all necessary to the present invention.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

The above is a description of an optical neural network convolution layer chip, a convolution calculation method and an electronic device provided by the present invention. For those skilled in the art, according to the idea of the embodiment of the present invention, in terms of specific implementation and application scope There will be changes. In conclusion, the content of this specification should not be construed as a limitation to the present invention.

Claims (10)

1. an optical neural network convolution layer chip, is characterized in that, comprises the first coupler, the first beam splitter, a plurality of photon calculation modules and convolution summation modules connected in turn; the first coupler for coupling the received optical signal into the first beam splitter; The first beam splitter includes a plurality of output ports, and the beam splitter is used for splitting the coupled optical signals to obtain multiple beams of optical signals, and the multiple beams of the optical signals are input through each of the output ports one by one to each of the photon computing modules; The photon calculation module is used to perform amplitude modulation and phase modulation on each beam of the optical signal, so as to represent an input data and a convolution kernel parameter through each beam of modulated optical signal, and convert all the modulated optical signals into is an electrical signal; The convolution and summation module is configured to perform convolution and summation on all the electrical signals, and complete the photon convolution operation of all input data and convolution kernel parameters. 2. The optical neural network convolution layer chip according to claim 1, wherein the photon calculation module comprises a second beam splitter, an input data modulation module, a convolution kernel parameter modulation module, a second beam splitter connected in sequence, and a second Couplers and balanced photodetectors; The second beam splitter is used to divide the input optical signal into two beams of optical signals, the second beam splitter includes two output ports, one output port is connected to the input data modulation module, and is used to convert the input data into two beams. One of the optical signals is transmitted to the input data modulation module, and the other output port is connected to the convolution kernel parameter modulation module for transmitting the other optical signal to the convolution kernel parameter modulation module; The input data modulation module is configured to perform amplitude modulation and phase modulation on one of the two beams of optical signals according to one input data, and represent the one input data by the modulated optical signal; The convolution kernel parameter modulation module is configured to perform amplitude modulation and phase modulation on the other one of the two optical signals according to one convolution kernel parameter, and the modulated optical signal represents the one convolution kernel parameters; the second coupler, configured to couple the optical signal modulated by the input data modulation module and the optical signal modulated by the convolution kernel parameter modulation module, and output two beams of optical signals; The balanced photodetector is used for balanced detection of the two optical signals to obtain the electrical signal. 3. The optical neural network convolution layer chip according to claim 2, wherein the input data modulation module comprises P1 optical amplitude modulation units and Q1 optical phase modulation units; the convolution kernel parameter modulation module Including P2 optical amplitude modulation units and Q2 optical phase modulation units; Among them, P1, Q1, P2, Q2 are non-negative integers. The optical amplitude modulation unit is used for amplitude modulation on the received optical signal; the optical phase modulation unit is used for phase modulation on the received optical signal. 4. The optical neural network convolution layer chip according to any one of claims 1 to 3, wherein the optical signal is emitted by a laser light source; The first coupler is an end face coupler or a grating coupler; The second coupler is a 2*2 coupler; The first beam splitter is a directional coupler or a multimode interference coupler. 5. The optical neural network convolution layer chip according to claim 3, wherein the optical amplitude modulation unit is a Mach-Zehnder interferometer, a micro-ring resonator or an optical waveguide deposited with phase change materials; The optical phase modulation unit is an electro-optic phase modulator or a thermo-optic phase modulator. 6. An optical neural network convolution calculation method, characterized in that, comprising: Coupling the received optical signal; Splitting the coupled optical signals to obtain multiple optical signals; performing amplitude modulation and phase modulation on each beam of the optical signal, so that one input data and one convolution kernel parameter are represented by each beam of the modulated optical signal; Convert all modulated optical signals into electrical signals; Perform convolution and summation on all the electrical signals to complete the photon convolution operation of all input data and convolution kernel parameters. 7 . The method according to claim 6 , wherein, performing amplitude modulation and phase modulation on each beam of the optical signal, so that each beam of modulated optical signal represents an input data and a convolution kernel parameter comprising: 8 . : Dividing each beam of the optical signal into two beams of optical signals to obtain multiple groups of two beams of optical signals; In each group of two optical signals, amplitude modulation and phase modulation are performed on one of the two optical signals according to one input data, and the one input data is represented by the modulated optical signal, and, according to a convolution For the kernel parameter, amplitude modulation and phase modulation are performed on the other beam of the two optical signals, and the one convolution kernel parameter is represented by the modulated optical signal. 8. The method according to claim 7, wherein before converting all modulated optical signals into electrical signals, the method comprises: The two beams of optical signals after amplitude modulation and phase modulation in each group are coupled to output two beams of optical signals. 9. The method according to claim 8, wherein the converting all modulated optical signals into electrical signals comprises: Balance detection is performed on the two optical signals in each group to obtain the electrical signals. 10. An electronic device, characterized by comprising the optical neural network convolution layer chip according to claims 1 to 5.

Images