WO2019165940A1 - Integrated circuit chip apparatus, board card and related product - Google Patents

Abstract

Provided are an integrated circuit chip apparatus and a related product. The integrated circuit chip apparatus comprises: a main processing circuit and a plurality of basic processing circuits, wherein the main processing circuit comprises a first mapping circuit, and at least one of the plurality of basic processing circuits comprises a second mapping circuit, the first mapping circuit and the second mapping circuit both being used to perform compression processing on each piece of data in a neural network operation. The technical solution provided in the present disclosure has the advantages of a small calculation amount and low power consumption.

Description

Integrated circuit chip devices, boards and related products Technical field

The present disclosure relates to the field of neural networks, and more particularly to an integrated circuit chip device, a board, and related products.

Background technique

Artificial Neural Network (ANN) is a research hotspot in the field of artificial intelligence since the 1980s. It abstracts the human brain neuron network from the perspective of information processing, establishes a simple model, and forms different networks according to different connection methods. In engineering and academia, it is often referred to directly as a neural network or a neural network. A neural network is an operational model consisting of a large number of nodes (or neurons) connected to each other. The calculation of the existing neural network is based on a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) to implement the operation of the neural network. Such calculations have a large amount of calculation and high power consumption.

Summary of the invention

Embodiments of the present disclosure provide an integrated circuit chip device, a board, and related products, which can improve the processing speed of the computing device and improve efficiency.

In a first aspect, an integrated circuit chip device is provided, the integrated circuit chip device comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, wherein at least one of the plurality of basic processing circuits A circuit includes a second mapping circuit, each of the first mapping circuit and the second mapping circuit for performing compression processing of each data in a neural network operation;

The main processing circuit is configured to perform each successive operation in the neural network operation and transmit data with the basic processing circuit;

The plurality of basic processing circuits are configured to perform operations in the neural network in parallel according to data transmitted by the main processing circuit, and transmit the operation results to the main processing circuit.

In some possible implementations, the integrated circuit chip device further includes: a branch processing circuit disposed between the main processing circuit and the at least one basic processing circuit; the branch processing circuit for performing the main processing Forwarding data between the circuit and the at least one basic processing circuit; wherein the branch processing circuit includes a plurality of branch processing circuits, the main processing circuit and the plurality of branch processing circuits being respectively connected, each branch processing circuit and at least one The basic processing circuit is connected.

In a second aspect, an integrated circuit chip device is provided. The integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, and at least the plurality of basic processing circuits A circuit includes a second mapping circuit, each of the first mapping circuit and the second mapping circuit for performing compression processing of each data in a neural network operation;

The main processing circuit is configured to acquire an input data block, a convolution kernel data block, and a convolution instruction, and divide the input data block into a broadcast data block according to the convolution instruction, and divide the convolution kernel data block into Generating a data block; determining, according to the operation control of the convolution instruction, starting the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and And the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the convolution instruction;

The plurality of basic processing circuits are configured to determine, according to the operation control of the convolution instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. Calculating an operation result, and transmitting the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing circuit determined by the basic processing circuit a transmitted data block, the second data block being associated with the processed first data block;

The main processing circuit is configured to process the operation result to obtain an instruction result of the convolution instruction.

In a third aspect, an integrated circuit chip device is provided, the integrated circuit chip device comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, wherein at least one of the plurality of basic processing circuits A circuit includes a second mapping circuit, each of the first mapping circuit and the second mapping circuit for performing compression processing of each data in a neural network operation;

The main processing circuit is configured to acquire an input data block, a weight data block, and a multiplication instruction, and divide the input data block into a data block according to the multiplication instruction, and divide the weight data block into a broadcast data block. And determining, according to the operation control of the multiplication instruction, starting the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast a data block; transmitting, according to the multiplication instruction, the processed first data block to at least one of the basic processing circuits connected to the main processing circuit;

The plurality of basic processing circuits are configured to determine, according to the operation control of the multiplication instruction, whether to start the second mapping circuit to process the second data block, and perform the operation in the neural network in parallel according to the processed second data block. Obtaining an operation result, and transmitting the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is sent by the main processing circuit determined by the basic processing circuit Data block, the second data block is associated with the processed first data block;

The main processing circuit is configured to process the operation result to obtain an instruction result of the multiplication instruction.

In a fourth aspect, an integrated circuit chip device is provided for performing a neural network forward operation, the neural network includes n layers, and the main processing circuit includes a first mapping circuit, the plurality of At least one of the basic processing circuits includes a second mapping circuit, each of the first mapping circuit and the second mapping circuit for performing compression processing of each data in the neural network operation:

The main processing circuit is configured to receive a forward operation instruction, and parse the forward operation instruction to obtain a first operation instruction included in an ith layer of the forward operation instruction in the forward operation of the neural network An input data block and a weight data block required by the first operation instruction, wherein the range of i is an integer greater than or equal to 1, and less than or equal to n, and if the i is greater than or equal to 2, the input data block is Output data block of layer i-1;

The main processing circuit is configured to divide the input data block into a broadcast data block according to the first operation instruction, and divide the convolution kernel data block into a data block; according to the first forward instruction The operation control determines to start the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast data block; The product instruction sends the processed first data block to at least one of the basic processing circuits connected to the main processing circuit;

The plurality of basic processing circuits are configured to determine, according to the operation control of the first operation instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. The operation obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing determined by the basic processing circuit a data block sent by the circuit, the second data block being associated with the processed first data block;

The main processing circuit is configured to process the operation result to obtain an instruction result of the first operation instruction, and complete an operation of the first operation instruction included in the ith layer.

In a fifth aspect, an integrated circuit chip device is provided for performing training of a neural network, where the neural network includes n layers, and the n value ranges from an integer greater than or equal to 2, wherein The integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, and at least one of the plurality of basic processing circuits includes a second mapping circuit, A mapping circuit and the second mapping circuit are both used to perform compression processing of respective data in a neural network operation;

The integrated circuit chip device is configured to receive a training instruction, determine first layer input data and first layer weight group data according to the training instruction, and perform a neural network on the first layer input data and the first layer weight group data. The n-th layer forward operation obtains the nth output result of the forward operation:

The main processing circuit is further configured to obtain an nth output result gradient according to the nth output result, and obtain an nth reverse operation nth inverse operation instruction and the nth inverse operation according to the training instruction The nth layer input data and the nth layer weight group data required by the instruction; dividing the nth output result gradient, the nth layer input data, and the nth layer weight group data according to the nth reverse operation instruction a broadcast data block and a distribution data block; determining, according to the operation control of the nth reverse operation instruction, starting the first mapping circuit to process the first data block, and obtaining the processed first data block; the first data block Include the distribution data block and/or the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the nth reverse operation instruction Processing circuit

The plurality of basic processing circuits are configured to determine, according to the operation control of the nth reverse operation instruction, whether to start the second mapping circuit to process the second data block, and execute the nerve in parallel according to the processed second data block An operation in the network obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is determined by the basic processing circuit to receive the a data block sent by the main processing circuit, the second data block being associated with the processed first data block;

The main processing circuit is further configured to process the operation result to obtain an nth layer weight group gradient and an nth layer input data gradient, and apply the nth layer weight group gradient to the nth layer weight group data. Update

The integrated circuit chip device is further configured to perform an n-1 layer inverse operation on the nth layer input data gradient as an n-1th output gradient of the n-1th layer to obtain an n-1 layer weight group gradient, and apply The n-1 layer weight group gradient updates the weight group data of the corresponding layer, and the weight group data includes at least two weights.

According to a sixth aspect, a neural network processor board is provided, the neural network processor board includes: a neural network chip package structure, a first electrical and non-electrical connection device, and a first substrate; the neural network chip package structure The method includes: a neural network chip, a second electrical and non-electrical connection device, and a second substrate, the second substrate carries the neural network chip, and the second substrate passes the second electrical and non-electrical connection device Neural network chip connection;

The neural network chip includes the integrated circuit chip device provided by any of the above aspects to the fifth aspect.

According to a seventh aspect, a neural network computing device is provided, the neural network computing device comprising the integrated circuit chip device provided by any one of the first to fifth aspects.

According to an eighth aspect, a combined processing apparatus is provided, where the combined processing apparatus includes: a neural network computing apparatus provided by the seventh aspect, a universal interconnection interface, and a general processing apparatus;

The neural network computing device is coupled to the general purpose processing device via the universal interconnect interface.

In a ninth aspect, a chip is provided, the chip integrating the apparatus provided in any one of the first to eighth aspects above.

In a tenth aspect, an electronic device is provided, the electronic device comprising the chip of the ninth aspect.

In an eleventh aspect, a method for computing a neural network is provided, the method being applied to an integrated circuit chip device, the integrated circuit chip device comprising: the integrated circuit of any one of the first aspect to the fifth aspect A chip device for performing an operation of a neural network.

It can be seen that, by using the embodiment of the present disclosure, the compression mapping circuit is provided to compress the data block and then perform operations, thereby saving transmission resources and computing resources, so that it has the advantages of low power consumption and small calculation amount.

DRAWINGS

1a is a schematic structural view of an integrated circuit chip device.

FIG. 1b is a schematic structural view of another integrated circuit chip device.

Figure 1c is a schematic structural view of a basic processing circuit.

Figure 2 is a schematic diagram of a matrix multiplied by a vector flow.

Figure 2a is a schematic diagram of a matrix multiplied by a vector.

Figure 2b is a schematic diagram of a matrix multiplied by a matrix flow.

Figure 2c is a schematic diagram of matrix Ai multiplied by vector B.

Figure 2d is a schematic diagram of matrix A multiplied by matrix B.

Figure 2e is a schematic diagram of matrix Ai multiplied by matrix B.

Figure 3a is a schematic diagram of neural network training.

Figure 3b is a schematic diagram of a convolution operation.

4 is a schematic structural diagram of a neural network chip provided by the flow of the embodiment of the present disclosure;

5a-5b are schematic structural diagrams of two mapping circuits provided by an embodiment of the present application.

Figure 6a is a schematic diagram of the forward operation of a neural network.

Figure 6b is a schematic diagram of a neural network operation.

Figure 7a is a schematic diagram of convolution input data.

Figure 7b is a schematic diagram of a convolution kernel.

Figure 7c is a schematic diagram of the operation window of a three-dimensional data block of input data.

Figure 7d is a schematic diagram of another operational window of a three-dimensional data block of input data.

Figure 7e is a schematic diagram of still another operational window of a three-dimensional data block of input data.

FIG. 8 is a schematic diagram of a training method of a neural network.

Figure 9a is a schematic diagram of neural network training.

Figure 9b is a schematic diagram of another neural network training.

Figure 9c is a schematic diagram of the forward and reverse operations of the neural network.

Figure 9d is a schematic diagram of a multi-layer structure of neural network training.

9e is a schematic structural diagram of a neural network chip provided by the flow of the embodiment of the present disclosure.

FIG. 10a is a schematic structural diagram of a combined processing apparatus according to the present disclosure.

FIG. 10b is another schematic structural diagram of a combined processing apparatus according to the present disclosure.

FIG. 11 is a schematic structural diagram of a neural network processor card according to an embodiment of the present disclosure;

FIG. 11b is a schematic structural diagram of a neural network chip package structure according to an embodiment of the present disclosure;

12a is a schematic diagram of a neural network chip package structure provided by the flow of the embodiment of the present disclosure;

FIG. 12b is a schematic diagram of another neural network chip package structure provided by the flow of the embodiment of the present disclosure.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present disclosure. It is a partial embodiment of the disclosure, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present disclosure without departing from the inventive scope are the scope of the disclosure.

In the apparatus provided by the first aspect, the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, and at least one of the plurality of basic processing circuits ( That is, part or all of the basic processing circuit) includes a second mapping circuit, each of which is used to perform compression processing of each data in the neural network operation;

The main processing circuit is configured to perform each successive operation in the neural network operation and transmit data with the basic processing circuit;

The plurality of basic processing circuits are configured to perform operations in the neural network in parallel according to data transmitted by the main processing circuit, and transmit the operation results to the main processing circuit.

In the apparatus provided by the first aspect, the integrated circuit chip device further includes: a branch processing circuit disposed between the main processing circuit and the at least one basic processing circuit; the branch processing circuit configured to Data is forwarded between the main processing circuit and the at least one underlying processing circuit.

In the apparatus provided by the first aspect, the main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; The distribution data block and the pre-stored identification data block associated with the distribution data block are subjected to split processing to obtain a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; and the plurality of basic data are The block and the identification data block associated with each of the plurality of basic data blocks are distributed to a base processing circuit connected thereto; the broadcast data block and the identification data block associated with the broadcast data block are broadcasted to a base processing circuit connected thereto.

The identifier data block may be specifically represented by a direct index or a step index, and an optional List of Lists (LIL), a Coordinate list (COO), and a compressed sparse line (Compressed). Sparse Row (CSR), Compressed Sparse Column (CSC), (ELL Pack, ELL), and Hybrid (HyB) are not limited in this application.

For example, the identification data block is represented by a direct index, and the identification data block may specifically be a data block composed of 0 and 1, wherein 0 represents data included in the data block (such as weight or input nerve). The absolute value of the element is less than or equal to the first threshold, and 1 indicates that the absolute value of the data (such as the weight or the input neuron) contained in the data block is greater than the first threshold, and the first threshold is a custom random setting on the user side or the device side. , such as 0.05, 0, and so on.

In order to save the data transmission amount and improve the data transmission efficiency, in the process of the main processing circuit transmitting the data to the basic processing circuit, the target data in the plurality of basic data blocks and the plurality of basic data may be specifically The identification data blocks associated with the blocks are distributed to the basic processing circuit connected thereto; optionally, the target data in the processed broadcast data block and the identification data block associated with the broadcast data block may also be broadcasted to the base connected thereto. Processing circuit. The target data refers to data in which the absolute value of the data block is greater than the first threshold, or refers to non-zero data in the data block (here, specifically, the processed distribution data block or the processed broadcast data block).

Correspondingly, the basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the broadcast data block and the identification data associated with the basic data block; Processing, by the block, the broadcast data block and the basic data block to obtain a processed broadcast data block and a basic data block; performing an inner product operation on the processed broadcast data block and the basic data block to obtain an operation result, and Transmitting the result of the operation to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction.

第一阈值为0.05，则该基本数据块关联的标识数据块为

关于第一映射电路和第二映射电路对数据块的处理将在后文进行具体阐述。 For example, the distribution data block is a matrix of M1 rows and N1 columns, and the basic data block is a matrix of M2 rows and N2 columns, where M1>M2, N1>N2. Correspondingly, the identification data block associated with the distribution data block is also a matrix of M1 rows and N1 columns, and the identification data block associated with the basic data block is also a matrix of M2 rows and N2 columns. Take a matrix with a basic data block of 2*2 as an example.

The first threshold is 0.05, and the identifier data block associated with the basic data block is

The processing of the data block with respect to the first mapping circuit and the second mapping circuit will be specifically described later.

In the apparatus provided by the first aspect, the main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; And the first mapping circuit is configured to process the distribution data block and the broadcast data block to obtain a processed distribution data block and an identification data block associated with the distribution data block, the processed broadcast data block, and the broadcast data block. Associated identification data block; splitting the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks associated with the basic data blocks, respectively The plurality of basic data blocks and the identification data blocks respectively associated with the plurality of basic data blocks are distributed to the basic processing circuit connected thereto, and the broadcast data block and the identification data block associated with the broadcast data block are broadcasted to the connected Basic processing circuit;

The basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the broadcast data block and the identification data associated with the basic data block; and according to the connection identification data block pair Processing the broadcast data block and the basic data block to obtain a processed broadcast data block and a basic data block; performing an inner product operation on the processed broadcast data block and the basic data block to obtain an operation result, and performing the operation The result is sent to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction.

In an optional embodiment, the main processing circuit is further configured to split the broadcast data block or the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data. a block and an identification data block associated with each of the plurality of partial broadcast data blocks; broadcasting the plurality of partial broadcast data blocks and the identification data blocks associated with each of the plurality of partial broadcast data blocks by the one or more broadcasts to the a base processing circuit; wherein the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.

Correspondingly, the basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the partial broadcast data block and the identification data block associated with the basic data block; The connection identification data is processed to process the partial broadcast data block and the basic data block to obtain a processed partial broadcast data block and the processed basic data block; and the processed partial broadcast data block and the processed The basic data block performs an inner product operation.

The connection identification data block is a data block obtained by performing an element-by-element operation on the identification data block associated with the basic data block and the identification data block associated with the partial broadcast data block. Optionally, the connection identifier data block is used to represent data in which two data blocks (specifically, a basic data block and a broadcast data block) are larger than an absolute value. The details will be described in detail later.

部分广播数据块关联的标识数据块为2*2的矩阵

则对应获得的连接标识数据块为

For example, the identification data block associated with the distribution data block is a 2*3 matrix.

The identification data block associated with a part of the broadcast data block is a matrix of 2*2

Corresponding to the obtained connection identification data block is

In the apparatus provided by the first aspect, the main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; Initiating the first mapping circuit to process the distribution data block to obtain a processed distribution data block and an identification data block associated with the distribution data block, or to initiate the first mapping circuit to associate according to the pre-stored distribution data block The identification data block processes the distribution data block to obtain the processed distribution data block; and the processed distribution data block and the identification data block associated with the distribution data block are split to obtain a plurality of basic data blocks and Identifying data blocks associated with each of the basic data blocks, distributing the plurality of basic data blocks and the identification data blocks associated with the plurality of basic data blocks to a basic processing circuit connected thereto, and broadcasting the broadcast data block To the basic processing circuit connected to it;

The basic processing circuit is configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, to obtain a processed broadcast data block; and the processed broadcast Performing an inner product operation on the data block and the processed basic data block to obtain an operation result, and transmitting the operation result to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction.

In an optional embodiment, the main processing circuit is further configured to: split the broadcast data block to obtain a plurality of partial broadcast data blocks; and broadcast the plurality of partial broadcast data blocks by one or more broadcasts. Giving the basic processing circuit; wherein the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.

Correspondingly, the basic processing circuit is configured to process the partial broadcast data block according to the identifier data block associated with the basic data block to obtain a processed partial broadcast data block; and to process the basic data block and the processing The latter part of the broadcast data block performs an inner product operation.

In the apparatus provided by the first aspect, the main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; And the first mapping circuit is configured to process the broadcast data block to obtain a processed broadcast data block and an identifier data block associated with the broadcast data block, or start the first mapping circuit according to the pre-stored broadcast data block. Processing, by the associated identification data block, the broadcast data block to obtain a processed broadcast data block; performing split processing on the distributed data block to obtain a plurality of basic data blocks; and distributing the plurality of basic data blocks to the connected a basic processing circuit, broadcasting the processed broadcast data block and the identification data block associated with the broadcast data block to a basic processing circuit connected thereto;

The basic processing circuit is configured to start, by the second mapping circuit, processing the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and processing the processed broadcast data. Performing an inner product operation on the block and the processed basic data block to obtain an operation result, and transmitting the operation result to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction.

In an optional embodiment, the main processing circuit is further configured to split the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and the And an identifier data block associated with the plurality of partial broadcast data blocks; and the identification data blocks associated with the plurality of partial broadcast data blocks and the plurality of partial broadcast data blocks are broadcast to the basic processing circuit by one or more times; And the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.

Correspondingly, the basic processing circuit is configured to process the basic data block according to the identifier data block associated with the partial broadcast data block to obtain a processed basic data block; and the processed basic data block and the The partial broadcast data block performs an inner product operation.

In the apparatus provided by the first aspect, the main processing circuit is specifically configured to send the broadcast data block (specifically, the broadcast data block or the processed broadcast data block) by one broadcast to the connected Basic processing circuit.

In the apparatus provided by the first aspect, the basic processing circuit is specifically configured to perform inner product processing on the basic data block (which may be the basic data block or the processed basic data block) and the broadcast data block. The inner product processing result is obtained, the inner product processing result is accumulated to obtain an operation result, and the operation result is sent to the main processing circuit.

In the apparatus provided by the first aspect, the main processing circuit is configured to accumulate the operation result and obtain an accumulation result when the operation result is the result of the inner product processing, and arrange the accumulation result to obtain the The data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided by the first aspect, the main processing circuit is specifically configured to divide the broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the basic processing by multiple times. a circuit; the plurality of partial broadcast data blocks are combined to form the broadcast data block.

In the apparatus provided by the first aspect, the basic processing circuit is specifically configured to perform an inner product of the partial broadcast data block (specifically, a partial broadcast data block or a processed partial broadcast data block) and the basic data block. After processing, an inner product processing result is obtained, and the inner product processing result is accumulated to obtain a partial operation result, and the partial operation result is sent to the main processing circuit. The inner product processing may be specifically as follows: the element of the partial broadcast data block is the first two elements of the matrix B, that is, b10 and b11; the basic data block is the first two elements of the first row of the input data matrix A, that is, For a10 and a11, the inner product = a10*b10+a11*b11.

For example, the basic data block here takes the kernel 3*3 as an example. The partial broadcast data block takes the 3*3 matrix as an example, and the *3 matrix and the core 3*3 perform the multiplication of the corresponding positions, then the corresponding inner product result is There are three inner product processing results, and the three inner product processing results are accumulated to obtain partial operation results. The results of the three inner product processing Out0 (the inner product of the 0th row of the 3*3 matrix and the 0th row of the core 3*3), the Out1 (the inner product of the 1st row of the 3*3 matrix and the 1st row of the core 3*3), Out2 (the inner product of the 2nd line of the 3*3 matrix and the 2nd line of the 3*3 core) can be specifically:

Out0=r00*k0[0]+r01*k0[1]+r02*k0[2]

Out1=r10*k1[0]+r11*k1[1]+r12*k1[2]

Out2=r20*k2[0]+r21*k2[1]+r22*k2[2]

Where r of r00 represents a partial broadcast data block, and 00 represents a 0th column element of the 0th row.

K0[0], k represents the basic data block, and 0[0] represents the 0th column element of the 0th row;

Partial operation result = Out0+Out1+Out2.

In the apparatus provided by the first aspect, the basic processing circuit is specifically configured to multiplex the partial broadcast data block n times to perform the inner product of the partial broadcast data block and the n basic data blocks to obtain n partial processing results ( Alternatively, the result of the n inner product operations may be corresponding to the n basic data blocks, and the n partial processing results are respectively accumulated to obtain n partial operation results, and the n partial operation results are obtained. Sended to the main processing circuit, the n is an integer greater than or equal to 2.

Here, the basic data block takes n cores 3*3 as an example. The partial broadcast data block takes a 3*3 matrix as an example, and the n times 3*3 matrix is multiplexed with the core 3*3 to perform n times corresponding position multiplication, each time. The operation is the corresponding inner product result, that is, there are three inner product results, and the three inner product results form a set of inner product operation results, and the three inner product results of each group in the n group are accumulated to obtain n partial operation results.

The above examples are only for explaining the convolution operation, and the core 3*3 may be replaced by other rules. Of course, the partial input data may also be other regular data blocks.

In the apparatus provided by the first aspect, the main processing circuit includes: a main register or a main on-chip buffer circuit;

The basic processing circuit includes a basic register or a basic on-chip buffer circuit.

In the apparatus provided by the first aspect, the main processing circuit comprises: a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, a first mapping circuit or a data rearrangement circuit. One or any combination of the above.

In the apparatus provided by the first aspect, the branch processing circuit includes a plurality of branch processing circuits, the main processing circuit being respectively connected to the plurality of branch processing circuits, each branch processing circuit being connected to at least one basic processing circuit.

In the apparatus provided by the first aspect, the basic processing circuit is further configured to forward the broadcast data block and the basic data block to other basic processing circuits to perform data processing and then perform an inner product operation to obtain an operation result. Transmitting the result of the operation to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided by the first aspect, the data block may be represented by a tensor, which may specifically be one or any combination of a vector, a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block. The data block may specifically include, but is not limited to, an input data block, a convolution kernel data block, and the like.

In the apparatus provided by the first aspect, the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is a broadcast data block, and the multiplicand data block is a distribution data block;

If the operation instruction is a convolution instruction, the main processing circuit determines that the input data block is a broadcast data block, and the convolution kernel is a distribution data block.

In the method provided by the sixth aspect, the operation of the neural network includes: a convolution operation, a matrix multiplication matrix operation, a matrix multiplication vector operation, a paranoid operation, a full connection operation, a GEMM operation, a GEMV operation, and an activation operation. Or any combination.

The operation instructions include, but are not limited to, a multiplication instruction, a convolution instruction, a forward operation instruction, and a training instruction. The specific embodiments involved in the integrated chip device will be described below by taking the operation instructions as the multiplication instruction and the convolution instruction, respectively. When the operation instruction is a convolution instruction, in the apparatus provided by the first aspect, the main processing circuit includes a first mapping circuit, and at least one of the plurality of basic processing circuits includes a second mapping circuit, the first mapping The circuit and the second mapping circuit are both used to perform compression processing of each data in the neural network operation:

The main processing circuit is configured to acquire an input data block, a convolution kernel data block, and a convolution instruction, and divide the input data block into vertical data blocks according to the convolution instruction, and the convolution kernel data block Dividing into a horizontal data block; determining, according to the operation control of the convolution instruction, starting the first mapping circuit to process the first data block, and obtaining the processed first data block; the first data block including the distribution data block And/or the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the convolution instruction;

The plurality of basic processing circuits are configured to determine, according to the operation control of the convolution instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. Calculating an operation result, and transmitting the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing circuit determined by the basic processing circuit a transmitted data block, the second data block being associated with the processed first data block; the main processing circuit configured to process the operation result to obtain an instruction result of the convolution instruction.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block and a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit, the distribution data block, and the Processing the broadcast data block to obtain the processed distribution data block, the identification data block associated with the distribution data block, the processed broadcast data block, and the identification data block associated with the broadcast data block; and distributing the processed data block and The identification data block associated with the distribution data block is split to obtain a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; and the plurality of basic data blocks and the plurality of basic data blocks are respectively And the associated identification data block is distributed to the basic processing circuit connected thereto, and the broadcast data block and the identification data block associated with the broadcast data block are broadcasted to the basic processing circuit connected thereto;

The basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the basic data block and the identification data block associated with the broadcast data block; and identify the data block according to the connection Processing the basic data block and the broadcast data block, performing a convolution (ie inner product) operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and transmitting the operation result to the The main processing circuit.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block, the main processing circuit is specifically configured to start processing, by the first mapping circuit, the processing data block to be processed. Distributing the data block and the identification data block associated with the distribution data block, or initiating the first mapping circuit to process the distribution data block according to the identifier data block associated with the pre-stored distribution data block to obtain the processed distribution data block; And dividing the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; and the plurality of basic data And the identification data block associated with each of the plurality of basic data blocks is distributed to a basic processing circuit connected thereto; the broadcast data block is broadcasted to a basic processing circuit connected thereto;

The basic processing circuit is configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, and execute the volume on the processed broadcast data block and the basic data block The product operation obtains the operation result, and the operation result is sent to the main processing circuit.

In the apparatus provided by the first aspect, when the first data block includes a broadcast data block, the main processing circuit is specifically configured to start processing, by the first mapping circuit, the broadcast data block to be processed. And the broadcast data block and the identifier data block associated with the broadcast data block, or the first mapping circuit is configured to process the broadcast data block according to the pre-stored identifier data block associated with the broadcast data block to obtain the processed broadcast data block. Splitting the distribution data block to obtain a plurality of basic data blocks; distributing the plurality of basic data to a base processing circuit connected thereto; and identifying the processed broadcast data block and the broadcast data block The data block is broadcast to the base processing circuit connected thereto;

The basic processing circuit is configured to start, by the second mapping circuit, processing the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and processing the processed basic data. The block and the processed broadcast data block perform a convolution operation to obtain an operation result, and the operation result is transmitted to the main processing circuit.

In the apparatus provided by the first aspect, the basic processing circuit is configured to perform a convolution operation on the basic data block and the broadcast data block to obtain a convolution result, and accumulate the convolution result to obtain an operation result, where The operation result is sent to the main processing circuit; the main processing circuit is configured to obtain an accumulation result after accumulating the operation result, and arrange the accumulation result to obtain the instruction result.

When the operation instruction is a multiplication instruction, in the apparatus provided by the first aspect, the main processing circuit includes a first mapping circuit, and at least one of the plurality of basic processing circuits includes a second mapping circuit, the first mapping circuit And the second mapping circuit is configured to perform compression processing of each data in the neural network operation;

The main processing circuit is configured to acquire an input data block, a weight data block, and a multiplication instruction, and divide the input data block into a data block according to the multiplication instruction, and divide the weight data block into a broadcast data block. And determining, according to the operation control of the multiplication instruction, starting the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast a data block; transmitting, according to the multiplication instruction, the processed first data block to at least one of the basic processing circuits connected to the main processing circuit;

The plurality of basic processing circuits are configured to determine, according to the operation control of the multiplication instruction, whether to start the second mapping circuit to process the second data block, and perform the operation in the neural network in parallel according to the processed second data block. Obtaining an operation result, and transmitting the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is sent by the main processing circuit determined by the basic processing circuit The data block, the second data block is associated with the processed first data block; and the main processing circuit is configured to process the operation result to obtain an instruction result of the multiplication instruction.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block and a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit, the distribution data block, and the Processing the broadcast data block to obtain the processed distribution data block, the identification data block associated with the distribution data block, the processed broadcast data block, and the identification data block associated with the broadcast data block; and distributing the processed data block and The identification data block associated with the distribution data block is split to obtain a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; and the plurality of basic data blocks and the plurality of basic data blocks are respectively And the associated identification data block is distributed to the basic processing circuit connected thereto, and the broadcast data block and the identification data block associated with the broadcast data block are broadcasted to the basic processing circuit connected thereto;

The basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the basic data block and the identification data block associated with the broadcast data block; and identify the data block according to the connection Processing the basic data block and the broadcast data block, performing a product operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and transmitting the operation result to the main processing circuit.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block, the main processing circuit is specifically configured to start processing, by the first mapping circuit, the processing data block to be processed. Distributing the data block and the identification data block associated with the distribution data block, or initiating the first mapping circuit to process the distribution data block according to the identifier data block associated with the pre-stored distribution data block to obtain the processed distribution data block; And dividing the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; and the plurality of basic data And the identification data block associated with each of the plurality of basic data blocks is distributed to a basic processing circuit connected thereto; the broadcast data block is broadcasted to a basic processing circuit connected thereto;

The basic processing circuit is configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, and perform a product on the processed broadcast data block and the basic data block. The operation obtains an operation result, and the operation result is transmitted to the main processing circuit.

In the apparatus provided by the first aspect, when the first data block includes a broadcast data block, the main processing circuit is specifically configured to start processing, by the first mapping circuit, the broadcast data block to be processed. And the broadcast data block and the identifier data block associated with the broadcast data block, or the first mapping circuit is configured to process the broadcast data block according to the pre-stored identifier data block associated with the broadcast data block to obtain the processed broadcast data block. Splitting the distribution data block to obtain a plurality of basic data blocks; distributing the plurality of basic data to a base processing circuit connected thereto; and identifying the processed broadcast data block and the broadcast data block The data block is broadcast to the base processing circuit connected thereto;

The basic processing circuit is configured to start, by the second mapping circuit, processing the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and processing the processed basic data. The block and the processed broadcast data block perform a product operation to obtain an operation result, and the operation result is transmitted to the main processing circuit.

In the apparatus provided by the first aspect, the basic processing circuit is configured to perform a product operation on the basic data block and the broadcast data block to obtain a product result, and accumulate the product result to obtain an operation result, and the operation result is obtained. Sending to the main processing circuit; the main processing circuit is configured to obtain an accumulated result after accumulating the operation result, and arrange the accumulated result to obtain the instruction result.

Referring to FIG. 1a, FIG. 1a is a schematic structural diagram of an integrated circuit chip device. As shown in FIG. 1a, the chip device includes: a main processing circuit, a basic processing circuit, and a branch processing circuit (optional). among them,

The main processing circuit may include a register and/or an on-chip buffer circuit, and the main processing circuit may further include: a control circuit, a vector operator circuit, an ALU (arithmetic and logic unit) circuit, an accumulator circuit, and a DMA (Direct Memory). Access, direct memory access) circuits and other circuits, of course, in practical applications, the above main processing circuit can also add, conversion circuits (such as matrix transposition circuits), data rearrangement circuits or activation circuits, and the like;

Optionally, the main processing circuit may include: a first mapping circuit, where the first mapping circuit may be configured to process the received or sent data to obtain the processed data and the identifier mask data associated with the data, the identifier mask data It is used to indicate whether the absolute value of the data is greater than a preset threshold. Optionally, the mask data may be 0 or 1, where 0 indicates that the absolute value of the data is less than or equal to a preset threshold; otherwise, 1 indicates that the absolute value of the data is greater than Preset threshold. The preset threshold is customized on the user side or the terminal device side, for example, 0.1 or 0.05, and the like. In practical applications, data having a data of 0 or not greater than a preset threshold (eg, 0.1) may be culled by the first mapping circuit, or the data may be set to zero. The utility model has the advantages of reducing the amount of data transmitted by the main processing circuit to the basic processing circuit, reducing the calculation amount of the data operation in the basic processing circuit, and improving the data processing efficiency. The invention does not limit the specific form of the first mapping circuit described above. A specific implementation of the first mapping circuit will be set forth below.

经过第一映射电路处理后可获得处理后的矩阵数据块为

该矩阵数据块关联的标识数据块为

关于第一映射电路的具体处理将在后文进行详述。 For example, the input data of the main processing circuit is a matrix data block.

After processing by the first mapping circuit, the processed matrix data block can be obtained as

The identification data block associated with the matrix data block is

The specific processing regarding the first mapping circuit will be described later in detail.

Correspondingly, when the main processing circuit distributes data to the basic processing circuit, only two data of 1 and 0.5 may be transmitted, not the processed matrix data block and 8 data; and the identifier of the matrix data block is also required to be associated. The data blocks are sent together to the basic processing circuit, so that the basic processing circuit correspondingly knows that the two data are located at the position of the original matrix data block according to the received identification data block and the received two data (1 and 0.5). That is, the basic processing circuit can correspondingly restore the processed matrix data block in the main processing circuit according to the received identification data block and the received data.

The main processing circuit further includes a data transmitting circuit, a data receiving circuit or an interface, and the data transmitting circuit can integrate the data distributing circuit and the data broadcasting circuit. Of course, in practical applications, the data distributing circuit and the data broadcasting circuit can also be separately set; in practical applications The above data transmitting circuit and data receiving circuit may also be integrated to form a data transmitting and receiving circuit. For broadcast data, data that needs to be sent to each of the underlying processing circuits. For distributing data, it is necessary to selectively send data to a part of the basic processing circuit, and the specific selection manner may be specifically determined by the main processing circuit according to the load and the calculation manner. For the broadcast transmission mode, the broadcast data is transmitted to each of the basic processing circuits in a broadcast form. (In practical applications, the broadcast data is sent to each of the basic processing circuits by means of one broadcast, and the broadcast data may be sent to each of the basic processing circuits by means of multiple broadcasts. The specific embodiment of the present application does not limit the above. The number of times of broadcasting), for the distribution transmission method, the distribution data is selectively transmitted to the partial basic processing circuit.

When the distribution data is implemented, the control circuit of the main processing circuit transmits data to some or all of the basic processing circuits (the data may be the same or different, and specifically, if the data is transmitted by means of distribution, the basic processing circuit of each received data is received. The data can be different, of course, the data received by some basic processing circuits can be the same;

Specifically, when the data is broadcast, the control circuit of the main processing circuit transmits data to some or all of the basic processing circuits, and the basic processing circuit of each received data can receive the same data.

Optionally, the vector operator circuit of the main processing circuit may perform vector operations, including but not limited to: two vector addition, subtraction, multiplication and division, vector and constant addition, subtraction, multiplication, division, or each element in the vector. Perform arbitrary operations. The continuous operation may specifically be: vector and constant addition, subtraction, multiplication, division, activation, accumulation, and the like.

Each of the basic processing circuits may include a base register and/or a base on-chip buffer circuit; each of the base processing circuits may further include one or any combination of an inner product operator circuit, a vector operator circuit, an accumulator circuit, and the like. The inner product operator circuit, the vector operator circuit, and the accumulator circuit may be integrated circuits, and the inner product operator circuit, the vector operator circuit, and the accumulator circuit may be separately provided circuits.

Optionally, the chip device may further include one or more branch processing circuits, such as a branch processing circuit, wherein the main processing circuit is connected to the branch processing circuit, and the branch processing circuit is connected to the basic processing circuit, and the basic processing circuit is The product operator circuit is configured to perform an inner product operation between the data blocks, the control circuit of the main processing circuit controls the data receiving circuit or the data transmitting circuit to transmit and receive external data, and the control circuit controls the data transmitting circuit to distribute the external data to the branch processing a circuit for transmitting and receiving data of a main processing circuit or a basic processing circuit. The structure shown in Figure 1a is suitable for the calculation of complex data, because for the main processing circuit, the number of connected cells is limited, so it is necessary to add a branch processing circuit between the main processing circuit and the basic processing circuit to achieve more The basic processing circuit is accessed to enable calculation of complex data blocks. The connection structure of the branch processing circuit and the basic processing circuit may be arbitrary, and is not limited to the H-type structure of FIG. 1a. Alternatively, the main processing circuit to the basic processing circuit is a broadcast or distributed structure, and the basic processing circuit to the main processing circuit is a gather structure. Broadcast, distribution and collection are defined as follows. For a distribution or broadcast structure, the number of basic processing circuits at this time is greater than that of the main processing circuit, that is, one main processing circuit corresponds to a plurality of basic processing circuits, that is, from the main processing circuit to a plurality of foundations. The processing circuit is a broadcast or distributed structure, and conversely, the plurality of basic processing circuits to the main processing circuit may be a collection structure.

The basic processing circuit receives the data distributed or broadcast by the main processing circuit and stores it in the on-chip buffer of the basic processing circuit, and can perform the operation to generate the result, and can transmit the data to the main processing circuit. Optionally, the basic processing circuit may first process the received data, save the processed data to the on-chip buffer, and perform processing using the processed data to generate a result, and optionally the processed data. It is sent to other basic processing circuits or main processing circuits, etc., and is not limited in this application.

Optionally, each of the basic processing circuits may include a second mapping circuit, or a second mapping circuit may be configured in the partial basic processing circuit; the second mapping circuit may be configured to process the received or transmitted data (ie, compress processing) ). The present invention does not limit the specific form of the second mapping circuit described above. The specific implementation of the second mapping circuit will be described in detail below.

Optionally, the vector operator circuit of the basic processing circuit can perform vector operations on two vectors (either one or both of the two vectors can be processed vectors), of course, in practical applications, basic processing The inner product operator circuit of the circuit can perform inner product operations on the two vectors, and the accumulator circuit can also accumulate the results of the inner product operations.

In an alternative, the two vectors can be stored in on-chip buffers and/or registers, and the underlying processing circuitry can extract two vectors to perform the operations as needed for the actual computation. The operations include, but are not limited to, inner product operations, multiplication operations, addition operations, or other operations.

In an alternative, the result of the inner product operation can be added to the on-chip buffer and/or the register; the advantage of the alternative is that the amount of data transfer between the basic processing circuit and the main processing circuit is reduced, and the number of data is increased. The efficiency of the operation reduces the power consumption of data transmission.

In an alternative, the result of the inner product operation is not accumulated and directly transmitted as a result; the advantage of this technical solution is that the calculation amount inside the basic processing circuit is reduced, and the operation efficiency of the basic processing circuit is improved.

In an alternative, each of the basic processing circuits may perform inner product operations of multiple sets of two vectors, or may accumulate results of multiple sets of inner product operations separately;

In an alternative, multiple sets of two vector data can be stored in an on-chip buffer and/or register;

In an alternative, the results of the multiple sets of inner product operations may be separately added to the on-chip buffer and/or registers;

In an alternative, the results of the inner product operations of each group may be directly accumulated as a result without accumulating;

In an alternative, each of the basic processing circuits may perform an inner product operation operation of the same vector and the plurality of vectors ("one-to-many" inner product, that is, two vectors of each group in the plurality of inner products. There is a vector in the shared), and the inner product results corresponding to each vector are separately accumulated. The technical solution can realize the same set of weights to perform multiple calculations on different input data, increase data multiplexing, reduce data transmission amount of internal processing circuit internal data, improve calculation efficiency, and reduce power consumption.

Specifically, in the data used to calculate the inner product, the data source shared by each group and the other vector of each group (ie, the vector different between each group) may be different:

In an alternative, when computing the inner product, the vectors shared by the groups are from the broadcast or distribution of the main processing circuit or the branch processing circuit;

In an alternative, when computing the inner product, the vectors shared by each group are from an on-chip cache;

In an alternative, when computing the inner product, the vectors shared by the groups are from registers;

In an alternative, when computing the inner product, another non-shared vector of each group is from the broadcast or distribution of the main processing circuit or the branch processing circuit;

In an alternative, when computing the inner product, another non-shared vector of each group comes from the on-chip cache;

In an alternative, when computing the inner product, another non-shared vector of each group is from a register;

In an alternative, each set of shared vectors retains any number of copies in the on-chip buffer and/or registers of the underlying processing circuitry when performing multiple sets of inner product operations;

In an alternative, the shared vector may be reserved for each inner product of each group;

In an alternative, the shared vector can be kept only one copy;

Specifically, the results of the multiple sets of inner product operations may be separately added to the on-chip buffer and/or the register;

Specifically, the result of the inner product operation of each group may be directly transmitted as a result without being accumulated;

In an alternative, the vector or matrix involved in the basic processing circuit may be a vector or matrix processed by the second mapping circuit, as will be described later.

Referring to the structure shown in Figure 1a, it includes a main processing circuit (which can perform vector operations) and a multi-base processing circuit (which can perform inner product operations). The advantage of this combination is that the device can not only perform matrix and vector multiplication operations using the basic processing circuit, but also perform other arbitrary vector operations using the main processing circuit, so that the device can be completed more quickly under the configuration of limited hardware circuits. More calculations reduce the number of data transmissions outside the device, improve computational efficiency, and reduce power consumption. In addition, the chip may be configured with a first mapping circuit in the main processing circuit to perform processing of data in the neural network, such as culling the first input data that is less than or equal to the preset threshold, and simultaneously obtaining the corresponding first input data. The associated mask data is used to indicate whether the absolute value of the first input data is greater than a preset threshold. For details, refer to the foregoing embodiment, and details are not described herein again. The advantages of this design can reduce the amount of data transmitted to the basic processing circuit, reduce the amount of calculation of the basic processing circuit data, increase the data processing rate, and reduce power consumption.

A second mapping circuit may be disposed in the basic processing circuit to perform processing of data in the neural network, for example, processing the second input data according to mask data associated with the first input data or mask data associated with the first input data and The mask data associated with the two input data selects the first input data whose absolute value is greater than the preset threshold and the second input data performs the corresponding arithmetic operation, and the like. The specific processing of the data by the first mapping circuit and the second mapping circuit can be referred to later in detail.

Optionally, the first mapping circuit and the second mapping circuit are both used to process data, which may be specifically designed into any one or more of the following circuits: a main processing circuit, a branch processing circuit, and Basic processing circuit, etc. In this way, the amount of data calculated can be reduced when performing neural network calculation, and the chip can dynamically allocate which circuit to perform data compression according to the calculation amount (ie, the load amount) of each circuit (mainly the main processing circuit and the basic processing circuit). Processing, which can reduce the complexity of data calculation, reduce power consumption, and dynamic allocation of data processing can achieve the calculation efficiency without affecting the chip. The manner of allocation includes, but is not limited to, load balancing, load minimum allocation, and the like.

Referring to the device shown in FIG. 1b, the device shown in FIG. 1b is a computing device without a branch processing circuit, as shown in FIG. 1b, comprising: a main processing circuit and N basic processing circuits, wherein the main processing circuit ( The specific structure is as shown in FIG. 1c) and can be directly or indirectly connected to the N basic processing circuits. For the indirect connection mode, an optional solution may include N/4 branch processing circuits as shown in FIG. 1a. Each of the branch processing circuits is respectively connected to four basic processing circuits. For the circuits respectively included in the main processing circuit and the N basic processing circuits, reference may be made to the description shown in FIG. 1a, and details are not described herein. The basic processing circuit can also be disposed in the branch processing circuit. In addition, the number of basic processing circuits connected to each branch processing circuit is not limited to four, and the manufacturer can configure according to actual needs. The first processing circuit and the second basic processing circuit may be respectively configured with a first mapping circuit and a second mapping circuit. Specifically, the main processing circuit may include a first mapping circuit, the N basic processing circuits or A portion includes a second mapping circuit; the main processing circuit may include a first mapping circuit and a second mapping circuit, and may also mean that the N basic processing circuits or a part thereof includes a first mapping circuit and a second mapping circuit. The main processing circuit may dynamically allocate an operation entity of the data compression processing step according to the neural network calculation instruction. Specifically, the main processing circuit may determine whether to perform a compression processing step on the received data according to its own load. Specifically, the load may be performed. The value is set to a plurality of sections, and each section corresponds to an execution subject of the allocation data compression processing step. For example, taking three sections as an example, the load value of the section 1 is low, and the data compression processing steps can be performed by the N basic processing circuits. The main processing circuit separately performs the data compression processing step, and the interval 2 load value is located between the interval 1 and the interval 3. The main processing circuit can separately perform the data compression processing step, and the interval 3 has a high load value, which can be performed by the main processing circuit or N The base processing circuit collectively performs the data compression processing steps. In this regard, it may be performed in an explicit manner. For example, the main processing circuit may be configured with a special indication or instruction. When the basic processing circuit receives the special indication or instruction, it determines to perform a data compression processing step, such as the basic processing circuit does not receive When a special instruction or instruction is made, it is determined that the data compression processing step is not performed. As another example, it may be performed in an implied manner, for example, when the basic processing circuit receives the sparse data (ie, includes 0, or includes data smaller than a preset threshold greater than a preset number) and determines that an inner product operation needs to be performed, the sparse The data is compressed.

The data compression processing involved in the present application is specifically performed in the first mapping circuit and the second mapping circuit described above. It should be understood that since the neural network is an algorithm with high computational complexity and high memory access, the more weights, the larger the calculation amount and the memory access amount. In particular, in the case where the weight is small (for example, 0, or less than the weight of the set value), in order to increase the calculation rate and reduce the overhead, it is necessary to compress the data with smaller weights. In practical applications, data compression processing is applied in sparse neural networks, and the effect is most obvious, such as reducing the workload of data calculation, reducing data overhead, and increasing data calculation rate.

Taking the input data as an example, a specific embodiment involved in the data compression processing will be explained. The input data includes, but is not limited to, at least one input neuron and/or at least one weight.

In the first embodiment:

After the first mapping circuit receives the first input data (specifically, the data block to be calculated sent by the main processing circuit, such as a distribution data block or a broadcast data block, etc.), the first mapping circuit may be the first input The data is processed to obtain the processed first input data and the identifier mask data associated with the first input data, the mask data is used to indicate whether the absolute value of the first input data is greater than a first threshold, such as 0.5, 0, etc. Wait.

第一阈值为0.05，则经过第一映射电路处理后可获得处理后的矩阵数据块

与该矩阵数据块关联的标识数据块(也可称为mask矩阵)为

Specifically, when the absolute value of the first input data is greater than the first threshold, the input data is retained; otherwise, the first input data is deleted or the first input data is set to 0. For example, the input matrix data block is

The first threshold is 0.05, and the processed matrix data block can be obtained after being processed by the first mapping circuit.

The identification data block (also referred to as the mask matrix) associated with the matrix data block is

可获知该数据对应的矩阵数据块(即主处理电路中第一映射电路处理后的矩阵数据块)为

Further, in order to reduce the amount of data transmission, when the main processing circuit distributes data to the basic processing circuit connected thereto, the target data in the processed matrix data block may be transmitted (in this example, 1, 0.06) And 0.5) and the identification data block associated with the matrix data block. In a specific implementation, the main processing circuit may distribute the target data in the processed matrix data block to the basic processing circuit according to a setting rule, for example, sequentially transmitting in a row order or sequentially in a column order, etc., the present application Not limited. Correspondingly, after receiving the target data and the associated data block corresponding to the target data, the basic processing circuit restores the target data block to the processed matrix data block according to a setting rule (for example, a row order). For example, in this example, the underlying processing circuit is based on the received data (1, 0.06, and 0.5) and the identified data block.

It can be known that the matrix data block corresponding to the data (that is, the matrix data block processed by the first mapping circuit in the main processing circuit) is

In the embodiment of the present invention, the first input data may be a distribution data block and/or a broadcast data block.

Correspondingly, the second mapping circuit can process the second input data by using the identification data associated with the first input data, thereby obtaining the processed second input data; wherein the first input data is different from the second input data. For example, when the first input data is at least one weight, the second input data may be at least one input neuron; or, when the first input data is at least one input neuron, then The second input data can be at least one weight.

In the embodiment of the present invention, the second input data is different from the first input data, and the second input data may be any one of: a distribution data block, a basic data block, a broadcast data block, and a partial broadcast data. Piece.

相应地利用上例中mask矩阵

处理后，获得处理后的部分广播数据块为

由于在实际应用中，输入数据涉及的矩阵数据块维数较大，本申请这里仅为示意，本不构成限定。 For example, when the first input data is a distribution data block, the second input data is a partial broadcast data block. Assume that the second input data is a matrix data block

Correspondingly use the mask matrix in the above example

After processing, the processed partial broadcast data block is obtained as

Since the dimension of the matrix data block involved in the input data is large in practical applications, the present application is merely illustrative and is not intended to be limiting.

In the second embodiment:

The first mapping circuit is configured to process the first input data and the second input data to obtain the processed first input data and the first identifier mask data associated with the first input data, and the processed second Input data and second identification mask data associated with the second input data. The first mask data or the second mask data is used to indicate whether the absolute value of the first or second input data is greater than a second threshold, and the second threshold is customized by the user side or the device side, for example, 0.05. 0 and so on.

经过第一映射电路处理后可获得处理后的分发数据块，这里处理后的分发数据块可为原矩阵数据块

也可为压缩处理后的矩阵数据块

应理解的，本申请为减少数据量的传输以及基础处理电路中数据处理效率，优选地所述处理后的输入数据(如处理后的基本数据块或部分广播数据块等)应为压缩处理后的数据。优选地，主处理电路向基础处理电路中发送的数据，具体可为所述处理后的输入数据中的目标数据，该目标数据具体可为绝对值大于预设阈值的数据，也可为非0数据等等。 The processed first input data or second input data may be processed input data or may be input data before processing. For example, the first input data is a distribution data block, such as the matrix data block in the above example.

After processing by the first mapping circuit, the processed distribution data block can be obtained, and the processed distribution data block can be the original matrix data block.

Can also be a compressed matrix data block

It should be understood that, in order to reduce the transmission of data amount and the data processing efficiency in the basic processing circuit, preferably, the processed input data (such as processed basic data block or partial broadcast data block, etc.) should be after compression processing. The data. Preferably, the data sent by the main processing circuit to the basic processing circuit may be the target data in the processed input data, and the target data may be data with an absolute value greater than a preset threshold, or may be non-zero. Data and more.

Correspondingly, in the basic processing circuit, the second mapping circuit may obtain connection identification data according to the first identification data associated with the first input data and the second identification data associated with the second input data; the connection identification data is used for And indicating that the absolute value of the first input data and the second input data are greater than a third threshold, wherein the third threshold is customized by the user side or the device side, such as 0.05, 0, and the like. Further, the second mapping circuit may process the received first input data and the second input data according to the connection identification data, thereby obtaining the processed first input data and the processed second input data.

第二输入数据块同样也为矩阵数据块

经过第一映射电路处理后可获得该第一输入数据关联的第一标识数据块

以及处理后的第一输入数据块

相应地获得该第二输入数据关联的第二标识数据块

处理后的第二输入数据块为

相应地，为提高数据传输速率，主处理电路中仅可将处理后的第一输入数据块中的目标数据1，0.06和0.5、以及该第一输入数据块关联的第一标识数据块发送给基础处理电路；同时，将处理后的第二输入数据块中的目标数据1，1.1，0.6，0.3和0.5，以及该第二输入数据块关联的第二标识数据块发送给基础处理电路。 For example, the first input data is a matrix data block

The second input data block is also a matrix data block

Obtaining, by the first mapping circuit, the first identification data block associated with the first input data

And the processed first input data block

Correspondingly obtaining the second identification data block associated with the second input data

The processed second input data block is

Correspondingly, in order to increase the data transmission rate, only the target data 1, 0.06 and 0.5 in the processed first input data block and the first identification data block associated with the first input data block may be sent to the main processing circuit. The basic processing circuit; at the same time, the target data 1, 1.1, 0.6, 0.3 and 0.5 in the processed second input data block and the second identification data block associated with the second input data block are sent to the basic processing circuit.

相应地，第二映射电路利用该连接标识数据块分别对所述处理后的第一输入数据块和处理后的第二输入数据块进行处理，从而获得处理后的第一输入数据块为

处理后的第二输入数据块为

其中，在基础处理电路中可根据第一标识数据块以及接收的第一数据块中的目标数据，确定出该目标数据对应所在的第一数据块(即经过第一映射电路处理后的第一数据块)；相应地，根据第二标识数据块以及接收的第二数据块中的目标数据，确定出该目标数据对应所在的第二数据块(即经过第一映射电路处理后的第二数据块)；然后，在第二映射电路获知连接标识数据块后，利用该连接标识数据块分别与确定的第一数据块和确定的第二数据块进行逐元素与操作，以获得经由第二映射电路处理后的第一数据块和处理后的第二数据块。 Correspondingly, after receiving the foregoing data, the basic processing circuit may perform the element-by-element operation on the first identification data block and the second identification data block by using the second mapping circuit to obtain the connection identification data block.

Correspondingly, the second mapping circuit processes the processed first input data block and the processed second input data block respectively by using the connection identification data block, so as to obtain the processed first input data block as

The processed second input data block is

The first processing block is configured to determine, according to the first identification data block and the target data in the received first data block, the first data block corresponding to the target data (ie, the first processed by the first mapping circuit) Correspondingly, determining, according to the second identification data block and the target data in the received second data block, the second data block corresponding to the target data (ie, the second data processed by the first mapping circuit) Blocking; then, after the second mapping circuit learns the connection identification data block, performing the element-by-element AND operation with the determined first data block and the determined second data block by using the connection identification data block, respectively, to obtain the second mapping The first data block after the circuit processing and the second data block after processing.

In the third embodiment:

The first mapping circuit is not disposed in the main processing circuit, but the main processing circuit may send the third input data and the pre-stored third identification data associated with the third input data to the basic processing circuit connected thereto in. A second mapping circuit is provided in the basic processing circuit. A specific embodiment of the data compression process involved in the second mapping circuit is explained below.

It should be understood that the third input data includes, but is not limited to, an elementary data block, a partial broadcast data block, a broadcast data block, and the like. Similarly, in the neural network processor, the third input data may also be at least one weight, and/or at least one input nerve, which is not limited herein.

In the second mapping circuit, the second mapping circuit may process the third input data according to the third identification data associated with the received third input data, thereby obtaining the processed third input data, so as to be subsequently The processed third input data performs related arithmetic operations, such as inner product operations.

相应地预存的该第三输入数据关联的第三标识数据块(也成mask矩阵数据块)为

进一步地，第二映射电路根据第三标识数据块对第三输入数据块进行处理得到处理后的第三输入数据块具体为

For example, the third input data received by the second mapping circuit is a matrix data block.

Correspondingly storing the third identification data block associated with the third input data (also into a mask matrix data block) is

Further, the third mapping circuit processes the third input data block according to the third identification data block, and the processed third input data block is specifically

Furthermore, the input neurons and output neurons mentioned in the embodiments of the present invention do not refer to neurons in the input layer of the entire neural network and neurons in the output layer, but to any adjacent two layers in the neural network. Neurons, the neurons in the lower layer of the network feedforward operation are the input neurons, and the neurons in the upper layer of the network feedforward operation are the output neurons. Taking a convolutional neural network as an example, suppose a convolutional neural network has an L layer, K=1, 2, 3...L-1. For the Kth and K+1th layers, the Kth layer is called an input. The layer, the neurons in the layer are the above input neurons, the K+1 layer is called the output layer, and the neurons in the layer are the above-mentioned output neurons, that is, each layer can be used as an input except the top layer. Layer, the next layer is the corresponding output layer.

In the fourth implementation:

A mapping circuit is not provided in the main processing circuit, and a first mapping circuit and a second mapping circuit are disposed in the basic processing circuit. For details about the data processing of the first mapping circuit and the second mapping circuit, refer to the foregoing first to third embodiments, and details are not described herein again.

Alternatively, there is also a fifth embodiment. In the fifth embodiment, the mapping circuit is not disposed in the basic processing circuit, and the first mapping circuit and the second mapping circuit are both disposed in the main processing circuit, and the first mapping circuit and the second mapping circuit are For details of the data processing, refer to the foregoing first to third embodiments, and details are not described herein again. That is, the main processing circuit completes the compression processing of the data, and sends the processed input data to the basic processing circuit, so that the basic processing circuit utilizes the processed input data (specifically, the processed neurons and the processed weights) Perform the corresponding arithmetic operations.

The following is a detailed structural diagram of the mapping circuit involved in the present application. Two possible mapping circuits are shown in Figures 5a and 5b. Among them, the mapping circuit shown in FIG. 5a includes a comparator and a selector. Regarding the number of the comparators and selectors, the application is not limited. A comparator and two selectors are shown in Fig. 5a, wherein the comparator is used to determine whether the input data satisfies a preset condition. The preset condition may be customized for the user side or the device side. For example, the absolute value of the input data described above in the application is greater than or equal to a preset threshold. If the preset condition is met, the comparator may determine to allow output of the input data, the input data corresponding to the associated identification data being 1; otherwise, it may be determined that the input data is not output, or the input data is 0 by default. Correspondingly, the identification data corresponding to the input data at this time is 0. That is, after the comparator is passed, the identification data associated with the input data can be known.

Further, after the comparator determines the preset condition of the input data, the obtained identification data may be input to the selector, so that the selector uses the identification data to determine whether to output the corresponding input data, that is, after the processing is obtained. Input data.

As shown in FIG. 5a, taking the input data as a matrix data block as an example, a comparator can determine a predetermined condition for each data in the matrix data block, so that an identifier data block associated with the matrix data block can be obtained ( Mask matrix). Further, the matrix data block may be filtered by using the identifier data block in the first selector, and the data in the matrix data block whose absolute value is greater than or equal to a preset threshold (ie, a preset condition is satisfied) is reserved. The remaining data is deleted to output the processed matrix data block. Optionally, the identifier data block may also be used in the second selector to process other input data (for example, the second matrix data block), for example, performing an element-by-element AND operation to obtain an absolute value in the second matrix data block. Data greater than or equal to the preset threshold is reserved to output the processed second matrix data block.

It should be understood that, corresponding to the above first and second embodiments, the specific structure of the first mapping circuit may include at least one comparator and at least one selector, such as the comparator and the first in FIG. 5a in the above example. a selector; the specific result of the second mapping circuit may include one or more selectors, such as the second selector of Figure 5a in the above example.

As shown in FIG. 5b, a schematic structural diagram of another mapping circuit is shown. As shown in FIG. 5b, the mapping circuit includes a selector, and the number of the selectors is not limited, and may be one or multiple. Specifically, the selector is configured to select the input data according to the identification data associated with the input data, so as to output data in which the absolute value of the input data is greater than or equal to a preset threshold, and the remaining data is deleted/ Does not output, thus obtaining processed input data.

Taking the input data as a matrix data block as an example, the matrix data block and the identification data block associated with the matrix data block are input to the mapping circuit, and the selector may select the matrix data block according to the identification data block. The data whose absolute value is greater than or equal to 0 is output, and the remaining data is not output, thereby outputting the processed matrix data block.

It should be understood that the structure shown in FIG. 5b can be applied to the second mapping circuit in the third embodiment described above, that is, the specific result of the second mapping circuit in the third embodiment described above may include at least one selector. For the same reason, the first mapping circuit and the second mapping circuit designed in the main processing circuit and the basic processing circuit may be cross-combined or split according to the functional components shown in FIG. 5a and FIG. 5b, which is not limited in this application.

The following provides a method for implementing calculation using a neural network forward operation as shown in FIG. 1a, which may be a neural network calculation method, such as a neural network training, in practical applications, The forward operation can perform matrix multiplication matrix, convolution operation, activation operation, transformation operation, and the like according to different input data, and the above operations can be implemented by using the apparatus shown in FIG. 1a.

The first mapping circuit of the main processing circuit first compresses the data and then transmits the data to the basic processing circuit for calculation. For example, the first mapping circuit of the main processing circuit can compress the data and then transmit the data to the basic processing circuit. The advantage is that the amount of data transmitted data can be reduced, the total number of bits transmitted is reduced, and the basic processing circuit performs data operations more efficiently and consumes less power.

The main processing circuit transmits the data to be calculated to all or a part of the basic processing circuit; taking the matrix multiplication by the vector calculation as an example, the control circuit of the main processing circuit can split the matrix data into each column as a basic data, for example, m*n The matrix can be split into n m rows of vectors, and the control circuit of the main processing circuit distributes the split n m rows of vectors to a plurality of basic processing circuits. For vectors, the control circuitry of the main processing circuit can broadcast the vector as a whole to each of the underlying processing circuits. If the value of m is relatively large, the control circuit may first split the m*n matrix into x*n vectors, taking x=2 as an example, and the specific one may be split into 2n vectors, each vector containing m/ 2 rows, that is, each vector of n m rows of vectors is divided into 2 vectors. In the first behavior example, if the first vector of n m rows of vectors is 1000 rows, then the two vectors can be divided into 2 vectors. The first 500 lines are composed of the first vector, and the last 500 lines are composed of the second vector. The control circuit broadcasts 2 vectors to the plurality of basic processing circuits through 2 broadcasts.

The manner of data transmission may be broadcast or distribution, or any other possible transmission method;

After receiving the data, the basic processing circuit first processes the data through the second mapping circuit, and then performs an operation to obtain an operation result;

The basic processing circuit transmits the operation result back to the main processing circuit;

The operation result may be an intermediate operation result or a final operation result.

The tensor multiplication tensor operation can be accomplished using the apparatus shown in Figure 1a, which is the same as the data block described above, which can be a matrix, a vector, a three-dimensional data block, a four-bit data block, and a high-dimensional data block. A combination of any one or more of the following; a specific implementation of the matrix multiplication vector and the matrix multiplication matrix operation is shown in Figures 2 and 2b, respectively.

The operation of the matrix multiplication vector is performed using the apparatus as shown in Fig. 1a; (the matrix multiplication vector can be an inner product of each row in the matrix with the vector, and the results are placed into a vector in the order of the corresponding rows.)

The following describes the operation of calculating the multiplication of the matrix S of the M rows and L columns and the vector P of the length L, as shown in Fig. 2a, (each row in the matrix S has the same length as the vector P, and the data in them is in position one Correspondingly, the neural network computing device has K basic processing circuits:

Referring to FIG. 2, FIG. 2 provides a method for implementing a matrix multiplication vector, which may specifically include:

Step S201, the control circuit of the main processing circuit distributes each row of data in the matrix S to one of the K basic processing circuits, and the basic processing circuit saves the received distribution data in an on-chip buffer of the basic processing circuit and/or In the register;

In an optional solution, the data of the matrix S is processed data. Specifically, the main processing circuit enables the first mapping circuit to process the matrix S, thereby obtaining the processed matrix S and a first mask matrix associated with the matrix S. Alternatively, the first mapping circuit of the main processing circuit processes the matrix S according to the first mask matrix associated with the pre-stored matrix S to obtain the processed matrix S. Further, each row data in the processed matrix S and the row data corresponding to the identification data associated in the first mask matrix are sent to one or more of the K basic processing circuits by the control circuit. When the main processing circuit sends data to the basic processing circuit, the data in the processed matrix S whose absolute value is greater than the preset threshold or the non-zero data may be sent to the basic processing circuit to reduce the data transmission amount. For example, the set of rows in the matrix S processed in the i-th basic processing circuit is Ai, and the Mi rows are shared; correspondingly, the identification matrix Bi corresponding to Ai is also distributed, and Bi is the first mask matrix. Part of a total of greater than or equal to the Mi line.

In an alternative, if the number of rows of the matrix S is M<=K, the control circuit of the main processing circuit separately distributes one row of the S matrix to the K basic processing circuits; optionally, the corresponding row is also sent by the row. Identification data of rows in the first identification matrix;

In an alternative, if the number of rows of the matrix S is M > K, the control circuitry of the main processing circuit distributes one or more rows of data in the S matrix to each of the underlying processing circuits. Optionally, the identifier data corresponding to the row in the first identifier matrix by the one or several rows is also sent;

The set of rows in S distributed to the i-th base processing circuit is Ai, sharing a total of Mi rows, as shown in Figure 2c, which is to be performed on the i-th base processing circuit.

In an alternative, in each of the basic processing circuits, such as the i-th basic processing circuit, the received distribution data, such as the matrix Ai, may be stored in a register and/or an on-chip buffer of the i-th base processing circuit. The advantage is to reduce the amount of data transmission of the distributed data afterwards, improve the calculation efficiency, and reduce the power consumption.

Step S202, the control circuit of the main processing circuit transmits the parts in the vector P to the K basic processing circuits in a broadcast manner;

In an alternative, the data (parts) of the vector P may be processed data. Specifically, the main processing circuit enables the first mapping circuit to process the vector P, thereby obtaining the processed vector P and a second mask matrix associated with the vector P. Alternatively, the first mapping circuit of the main processing circuit processes the vector P according to the second mask matrix associated with the pre-stored vector P to obtain the processed vector P. Further, the data in the processed vector P (ie, each part) and the corresponding data corresponding to the data in the second mask matrix are sent to one or more of the K basic processing circuits by the control circuit. in. When the main processing circuit sends data to the basic processing circuit, the data of the processed vector P having an absolute value greater than a preset threshold or non-zero data may be specifically sent to the basic processing circuit to reduce the data transmission amount.

In an alternative, the control circuit of the main processing circuit can broadcast each part of the vector P only once to the register or the on-chip buffer of each basic processing circuit, and the i-th basic processing circuit obtains the vector P of this time. The data is fully multiplexed to complete the inner product operation corresponding to each row in the matrix Ai. The advantage is that the data transmission amount of the repeated transmission of the vector P from the main processing circuit to the basic processing circuit is reduced, the execution efficiency is improved, and the transmission power consumption is reduced.

In an alternative, the control circuit of the main processing circuit can broadcast each part of the vector P to the register or the on-chip buffer of each basic processing circuit multiple times, and the i-th basic processing circuit obtains the vector P for each time. The data is not multiplexed, and the inner product operation corresponding to each row in the matrix Ai is completed in stages; the advantage is that the data transmission amount of the vector P of the single transmission inside the basic processing circuit is reduced, and the basic processing circuit buffer and the lower limit can be reduced. / or the capacity of the register to improve execution efficiency, reduce transmission power consumption, and reduce costs.

In an alternative, the control circuit of the main processing circuit can broadcast each part of the vector P to the register or the on-chip buffer of each basic processing circuit multiple times, and the i-th basic processing circuit obtains the vector P for each time. The data is partially multiplexed to complete the inner product operation corresponding to each row in the matrix Ai; the advantage is that the data transmission amount from the main processing circuit to the basic processing circuit is reduced, and the data transmission amount inside the basic processing circuit is also reduced, and the execution is improved. Efficiency, reducing transmission power consumption.

Step S203, the inner product operator circuit of the K basic processing circuits calculates an inner product of the data of the matrix S and the vector P, for example, the i-th basic processing circuit, and calculates an inner product of the data of the matrix Ai and the data of the vector P;

In a specific implementation, the basic processing circuit receives the data in the processed matrix S and the data corresponding to the identification data associated in the first mask matrix; and also receives the data in the processed vector P. . Correspondingly, the basic processing circuit enables the second mapping circuit to process the data of the received vector P according to the identification data in the received first mask matrix to obtain the data of the processed vector P. Further, the basic processing circuit enables the inner product operator circuit to perform an inner product operation on the received data in the processed matrix S and the processed vector P data to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the matrix Ai, the identification matrix Bi associated with the Ai, and the vector P; at this time, the second mapping circuit can be enabled to process the vector P by using the Bi to obtain the processed vector P; The product operator circuit performs an inner product operation on the matrix Ai and the processed vector P.

In a specific implementation, the basic processing circuit receives the data in the processed vector P and the data corresponding to the identification data associated in the second mask matrix; and also receives the data in the processed matrix S. . Correspondingly, the basic processing circuit enables the second mapping circuit to process the data of the received matrix S according to the identification data in the received second mask matrix to obtain the data of the processed matrix S. Further, the basic processing circuit enables the inner product operator circuit to perform an inner product operation on the received data of the processed vector P and the processed data in the matrix S to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the matrix Ai, the processed vector P, and the second identification matrix associated with the vector P; at this time, the second mapping circuit can be enabled to process the Ai by using the second identification matrix. The matrix Ai; the inner product operator circuit is further enabled to perform an inner product operation on the processed matrix Ai and the processed vector P.

In a specific implementation, the basic processing circuit receives the data in the processed matrix S and the data corresponding to the identification data associated in the first mask matrix; and also receives the data in the processed vector P. And the data corresponds to the identification data associated in the second mask matrix. Correspondingly, the basic processing circuit enables the second mapping circuit to obtain a relationship identification matrix (also referred to as a relational connection matrix) according to the identification data in the received first mask matrix and the identification data in the second mask matrix; and then use the relationship identification matrix The identification data in the data is processed on the data in the received matrix S and the data in the vector P, respectively, to obtain the data of the processed matrix S and the data of the processed vector P. Further, the inner product operator circuit is enabled to perform an inner product operation on the data in the processed matrix S and the processed vector P data to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the matrix Ai, the identification matrix Bi associated with the Ai, the vector P, and the second identification matrix associated with the vector P; at this time, the second mapping circuit can be enabled to obtain the second identification matrix by using the Bi and the second identification matrix. The relationship identification matrix is used to process the matrix Ai and the vector P simultaneously or separately to obtain the processed matrix Ai and the processed vector P. Next, the inner product operator circuit is enabled to perform an inner product operation on the processed matrix Ai and the processed vector P.

Step S204: The accumulator circuit of the K basic processing circuits accumulates the result of the inner product operation to obtain an accumulated result, and transmits the accumulated result to the main processing circuit in a fixed point type.

In an alternative, each part of the basic processing circuit may perform an inner product operation to obtain a part and (partial and part of the accumulated result, for example, the accumulated result is: F1*G1+F2*G2+F3*G3+F4 *G4+F5*G5, then the part and can be: the value of F1*G1+F2*G2+F3*G3) is transferred back to the main processing circuit for accumulation; the advantage is that the calculation amount inside the basic processing circuit is reduced, and the basis is improved. The computational efficiency of the processing circuit.

In an alternative, the part obtained by the inner product operation performed by each basic processing circuit and the register and/or the on-chip buffer stored in the basic processing circuit may be transferred to the main processing circuit after the accumulation is completed; The data transmission between the basic processing circuit and the main processing circuit is reduced, the operation efficiency is improved, and the data transmission power consumption is reduced.

In an alternative, it is also possible to accumulate the portion obtained by the inner product operation performed by each basic processing circuit and, in some cases, the register and/or the on-chip buffer of the basic processing circuit, and in some cases transmit to The main processing circuit performs accumulation, and is transferred back to the main processing circuit after the accumulation is completed; the advantage is that the data transmission amount between the basic processing circuit and the main processing circuit is reduced, the operation efficiency is improved, the data transmission power consumption is reduced, and the basic processing is reduced. The amount of calculation inside the circuit improves the computational efficiency of the basic processing circuit.

Referring to Figure 2b, the operation of the matrix multiplication matrix is performed using the apparatus as shown in Figure 1a;

The operation of calculating the multiplication of the matrix S of the M rows and L columns and the matrix P of the size of the L rows and N columns is described below (each row in the matrix S is the same length as each column of the matrix P, as shown in Fig. 2d). The neural network computing device has K basic processing circuits:

Step S201b, the control circuit of the main processing circuit distributes each row of data in the matrix S to one of the K basic processing circuits, and the basic processing circuit saves the received data in an on-chip buffer and/or a register;

In an optional solution, the data of the matrix S is processed data. Specifically, the main processing circuit enables the first mapping circuit to process the matrix S, thereby obtaining the processed matrix S and a first mask matrix associated with the matrix S. Alternatively, the first mapping circuit of the main processing circuit processes the matrix S according to the first mask matrix associated with the pre-stored matrix S to obtain the processed matrix S. Further, each row data in the processed matrix S and the row data corresponding to the corresponding associated identification data in the first mask matrix are sent to one or more of the K basic processing circuits by the control circuit. When the main processing circuit sends data to the basic processing circuit, the data in the processed matrix S whose absolute value is greater than the preset threshold or the non-zero data may be sent to the basic processing circuit to reduce the data transmission amount.

In an alternative, if the number of rows of S is M<=K, the control circuit of the main processing circuit respectively distributes one row of the S matrix to the M basic processing circuits; optionally, the corresponding row is also sent by the row. Identification data of a row in the first identification matrix;

In an alternative, if the number of rows of S is M > K, the control circuitry of the main processing circuit distributes one or more rows of data in the S matrix to each of the underlying processing circuits. Optionally, the identifier data corresponding to the row in the first identifier matrix by the one or several rows is also sent;

The Mi line in S is distributed to the i-th base processing circuit, and the set of Mi lines is called Ai, and the calculation to be performed on the i-th basic processing circuit is shown in Fig. 2e.

In an alternative, in each of the basic processing circuits, such as the i-th basic processing circuit:

The received matrix Ai distributed by the main processing circuit stores the matrix Ai in the i-th basic processing circuit register and/or the on-chip buffer; the advantage is that the amount of data transmission afterwards is reduced, the calculation efficiency is improved, and the power consumption is reduced.

Step S202b, the control circuit of the main processing circuit transmits each part of the matrix P to each basic processing circuit in a broadcast manner;

In an alternative, the data (parts) of the matrix P can be processed data. Specifically, the main processing circuit enables the first mapping circuit to process the matrix P, thereby obtaining the processed matrix P and a second matrix of the matrix associated with the matrix P. Alternatively, the first mapping circuit of the main processing circuit processes the matrix P according to the second mask matrix associated with the pre-stored matrix P to obtain the processed matrix P. Further, the data in the processed matrix P (ie, each part) and the corresponding data corresponding to the data in the second mask matrix are sent to one or more of the K basic processing circuits by the control circuit. in. When the main processing circuit sends data to the basic processing circuit, specifically, the data in the processed matrix P whose absolute value is greater than a preset threshold or non-zero data may be sent to the basic processing circuit to reduce the data transmission amount.

In an alternative, each part of the matrix P can be broadcast only once to the registers of the respective basic processing circuits or the on-chip buffer, and the i-th basic processing circuit fully multiplexes the data of the matrix P obtained this time. The inner product operation corresponding to each row in the matrix Ai is completed; the multiplexing in this embodiment may be repeatedly used in the calculation by the basic processing circuit, for example, the multiplexing of the data of the matrix P, and may be the data of the matrix P. use many times.

In an alternative, the control circuit of the main processing circuit can broadcast the parts of the matrix P to the registers of the respective basic processing circuits or the on-chip buffer multiple times, and the i-th basic processing circuit pairs the matrix P obtained each time. The data is not multiplexed, and the inner product operations corresponding to each row in the matrix Ai are completed in stages;

In an alternative, the control circuit of the main processing circuit can broadcast the parts of the matrix P to the registers of the respective basic processing circuits or the on-chip buffer multiple times, and the i-th basic processing circuit pairs the matrix P obtained each time. The data is partially multiplexed to complete an inner product operation corresponding to each row in the matrix Ai;

In an alternative, each of the basic processing circuits, such as the i-th basic processing circuit, calculates an inner product of the data of the matrix Ai and the data of the matrix P;

Step S203b, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit.

Optionally, before step S203b, the inner product operator of the basic processing circuit needs to calculate the inner product of the data of the matrix S and the matrix P. Specifically, the following implementation manners exist.

In a specific implementation, the basic processing circuit receives the data in the processed matrix S and the data corresponding to the identification data associated in the first mask matrix; and also receives the data in the processed matrix P. . Correspondingly, the basic processing circuit enables the second mapping circuit to process the data of the received matrix P according to the identification data in the received first mask matrix to obtain the data of the processed matrix P. Further, the basic processing circuit enables the inner product operator circuit to perform an inner product operation on the received data in the processed matrix S and the processed matrix P data to obtain a result of the inner product operation.

In a specific implementation, the basic processing circuit receives the data in the processed matrix P and the data corresponding to the identification data associated in the second mask matrix; and also receives the data in the processed matrix S. . Correspondingly, the basic processing circuit enables the second mapping circuit to process the data of the received matrix S according to the identification data in the received second mask matrix to obtain the data of the processed matrix S. Further, the basic processing circuit enables the inner product operator circuit to perform an inner product operation on the received data of the processed matrix P and the processed data in the matrix S to obtain a result of the inner product operation.

In a specific implementation, the basic processing circuit receives the data in the processed matrix S and the data corresponding to the identification data associated in the first mask matrix; and also receives the data in the processed matrix P. And the data corresponds to the identification data associated in the second mask matrix. Correspondingly, the basic processing circuit enables the second mapping circuit to obtain a relationship identification matrix (also referred to as a relational connection matrix) according to the identification data in the received first mask matrix and the identification data in the second mask matrix; and then use the relationship identification matrix The identification data respectively processes the data in the received matrix S and the data in the matrix P to obtain the processed matrix S data and the processed matrix P data. Further, the inner product operator circuit is enabled to perform an inner product operation on the data in the processed matrix S and the processed matrix P data to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the matrix Ai, the identification matrix Bi associated with the Ai, the matrix P, and the second identification matrix associated with the matrix P; at this time, the second mapping circuit can be enabled to obtain the second identification matrix by using the Bi and the second identification matrix. The relationship identification matrix is used to process the matrix Ai and the matrix P simultaneously or separately to obtain the processed matrix Ai and the processed matrix P. Next, the inner product operator circuit is enabled to perform an inner product operation on the processed matrix Ai and the processed matrix P.

In an alternative, the basic processing circuit may accumulate the portion obtained by performing the inner product operation each time and transmit back to the main processing circuit;

In an optional solution, the portion obtained by the inner product operation performed by each basic processing circuit may be stored in a register and/or an on-chip buffer of the basic processing circuit, and then accumulated and returned to the main processing circuit;

In an alternative, it is also possible to accumulate the portion obtained by the inner product operation performed by each basic processing circuit and, in some cases, the register and/or the on-chip buffer of the basic processing circuit, and in some cases transmit to The main processing circuit performs accumulation, and after the accumulation is completed, it is transmitted back to the main processing circuit.

Referring to Figure 3a, the full join operation is done using the apparatus shown in Figure 1a:

If the input data of the fully connected layer is a vector (that is, the input of the neural network is a single sample), the weight matrix of the fully connected layer is used as the matrix S, and the input vector is used as the vector P, according to the usage method of the device. Perform the operation of the matrix multiplication vector as shown in FIG. 2;

If the input data of the fully connected layer is a matrix (that is, the case where the input of the neural network is a plurality of samples as a batch), the weight matrix of the fully connected layer is used as the matrix S, the input vector is used as the matrix P, or the fully connected layer is used. The weight matrix as the matrix P, the input vector as the matrix S, using the device to complete the operation of the matrix multiplication matrix as shown in Figure 2c;

Referring to Figure 3b, the convolution operation is performed using the apparatus as shown in Figure 1a:

For a convolutional layer, remember that the number of convolution kernels is M;

Step S301: The control circuit of the main processing circuit distributes the weight of each convolution kernel in the convolution layer weight to one of the K basic processing circuits, and stores the on-chip buffer and/or the register in the basic processing circuit. in;

In an alternative, the weight of the convolution kernel is processed weight data. Specifically, the main processing circuit enables the first mapping circuit to process the weight of the convolution kernel, thereby obtaining the processed convolution kernel and a first mask matrix associated with the convolution kernel. Alternatively, the first mapping circuit of the main processing circuit processes the convolution kernel according to the first mask matrix associated with the pre-stored convolution kernel, and obtains the processed convolution kernel (ie, the weight of the processed convolution kernel). Further, the control circuit outputs the weight in the processed convolution kernel and the weight corresponding to the corresponding associated identification data in the first mask matrix to one or more of the K basic processing circuits. When the main processing circuit sends data to the basic processing circuit, the data in the processed convolution kernel whose absolute value is greater than a preset threshold or non-zero data may be specifically sent to the basic processing circuit to reduce the data transmission amount. For example, the set of Mi processed convolution kernels distributed to the i-th basic processing circuit is Ai; correspondingly, the identification matrix Bi corresponding to Ai is also distributed.

In an alternative, if the number of convolution kernels is M<=K, the control circuit of the main processing circuit separately distributes the weights of one convolution kernel to the M basic processing circuits; optionally, also sends The convolution kernel corresponds to an associated identification matrix (ie, identification data in the identification matrix).

In an alternative, if the number of convolution kernels is M>K, the control circuit of the main processing circuit separately distributes the weights of one or more convolution kernels to each of the basic processing circuits; optionally, Sending the one or more convolution kernels respectively corresponding to the associated identification matrix (ie, the identification data in the identification matrix).

A total of Mi convolution kernels are distributed to the i-th base processing circuit, and the set of these convolution kernel weights is called Ai.

In an alternative, in each of the basic processing circuits, such as the i-th basic processing circuit:

The received convolution kernel weight Ai distributed by the main processing circuit is stored in its register and/or on-chip buffer;

Step S302, the control circuit of the main processing circuit transmits each part of the input data P to each basic processing circuit in a broadcast manner;

In an alternative, the input data P is processed data. Specifically, the main processing circuit enables the first mapping circuit to process the input data P, thereby obtaining the processed input data P and a second mask matrix (ie, identification data) associated with the input data P. Alternatively, the first mapping circuit of the main processing circuit processes the input data P according to the second mask matrix associated with the pre-stored input data P to obtain the processed input data P. Further, the data in the processed input data P (ie, each part) and the corresponding data corresponding to the data in the second mask matrix are sent to one or more of the K basic processing circuits by the control circuit. In the middle. When the main processing circuit sends data to the basic processing circuit, the data of the processed input data P whose absolute value is greater than a preset threshold or non-zero data may be specifically sent to the basic processing circuit to reduce the data transmission amount.

In an alternative, the control circuit of the main processing circuit can broadcast each part of the input data P only once to the register or on-chip buffer of each basic processing circuit, and the i-th basic processing circuit inputs the input data for this time. The data of P is fully multiplexed, and the inner product operation corresponding to each convolution kernel in Ai is completed;

In an alternative, the control circuit of the main processing circuit can broadcast each part of the input data P to the register or the on-chip buffer of each basic processing circuit multiple times, and the input data obtained by the i-th basic processing circuit for each time. The data of P is not multiplexed, and the inner product operation corresponding to each convolution kernel in Ai is completed in stages;

In an alternative, the control circuit of the main processing circuit can broadcast each part of the input data P to the register or the on-chip buffer of each basic processing circuit multiple times, and the input data obtained by the i-th basic processing circuit for each time. The data of P is partially multiplexed to complete an inner product operation corresponding to each convolution kernel in Ai;

Step S303, each basic processing circuit calculates a data inner product of the convolution kernel and the input data P, for example, an i-th basic processing circuit, and calculates an inner product of each convolution kernel of the Ai and the data of the input data P;

In a specific embodiment, the base processing circuit receives the data in the processed convolution kernel and the data corresponds to the identification data associated in the first mask matrix; and also receives the processed input data P. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received input data P according to the identification data in the received first mask matrix to obtain the processed input data P. Further, the basic processing circuit enables the inner product operator circuit to perform an inner product operation on the data in the received processed convolution kernel and the processed input data P to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the convolution kernel weight Ai, the identification matrix Bi associated with the Ai, and the input data P; at this time, the second mapping circuit can be enabled to process the input data P by using Bi. The data P is input; the inner product operator circuit is further enabled to perform an inner product operation on the convolution kernel weight Ai and the processed input data P.

In a specific implementation, the basic processing circuit receives the data in the processed input data P and the data corresponding to the identification data associated in the second mask matrix; and also receives the processed convolution kernel The data. Correspondingly, the basic processing circuit enables the second mapping circuit to process the data of the received convolution kernel according to the identification data in the received second mask matrix to obtain the data of the processed convolution kernel. Further, the basic processing circuit enables the inner product operator circuit to perform an inner product operation on the received processed input data P and the processed convolution kernel to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the convolution kernel weight Ai, the processed input data P, and the second identification matrix associated with the input data P; at this time, the second mapping circuit can be enabled to utilize the second identification matrix pair Ai performs processing to obtain the processed convolution kernel weight Ai; and then enables the inner product operator circuit to perform inner product operation on the processed convolution kernel weight Ai and the processed input data P.

In a specific implementation, the basic processing circuit receives the data in the processed convolution kernel and the data corresponding to the identification data associated in the first mask matrix; and also receives the processed input data P. The data and the data correspond to the identification data associated in the second mask matrix. Correspondingly, the basic processing circuit enables the second mapping circuit to obtain a relationship identification matrix (also referred to as a relational connection matrix) according to the identification data in the received first mask matrix and the identification data in the second mask matrix; and then use the relationship identification matrix The identification data in the process respectively processes the data in the received convolution kernel and the data in the input data P to obtain the processed convolution kernel data and the processed input data P. Further, the inner product calculator circuit is enabled to perform an inner product operation on the data in the processed convolution kernel and the processed input data P to obtain a result of the inner product operation. For example, the i-th basic processing circuit receives the convolution kernel weight Ai, the Ai-associated identification matrix Bi, the input data P, and the second identification matrix associated with the input data P; at this time, the second mapping circuit can be enabled to utilize Bi. And obtaining a relationship identifier matrix with the second identifier matrix, and then processing the convolution kernel weight Ai and the input data P by using the relationship identifier matrix, respectively, to obtain the processed convolution kernel weight Ai and the processed input data P . Next, the inner product operator circuit is enabled to perform an inner product operation on the processed convolution kernel weight Ai and the processed input data P.

Step S304, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit:

In an alternative, the basic processing circuit may accumulate the portion obtained by performing the inner product operation each time and transmit back to the main processing circuit;

In an alternative, the basic processing circuit may also save the portion obtained by the inner product operation performed each time and the register and/or the on-chip buffer stored in the basic processing circuit, and then transfer back to the main processing circuit after the accumulation is completed;

In an alternative, the basic processing circuit may also accumulate the portion obtained by each inner product operation and, in some cases, the register and/or the on-chip buffer of the basic processing circuit, and in some cases transmit to The main processing circuit performs accumulation, and is transferred back to the main processing circuit after the accumulation is completed;

Use the device shown in Figure 1a to update the weights:

The vector updater circuit of the main processing circuit is used to implement the weight update function in the neural network training process. Specifically, the weight update refers to a method of updating the weight using the gradient of the weight.

In an alternative, the vector operator circuit of the main processing circuit is used to add and subtract the two vectors of the weight and the weight gradient to obtain an operation result, and the operation result is an update weight.

In an alternative, the vector operator circuit of the main processing circuit multiplies or divides the weight and the weight gradient by a number to obtain an intermediate weight and an intermediate weight gradient value, and the vector operator circuit pairs the intermediate weight And the intermediate weight gradient value is added and subtracted to obtain the operation result, and the operation result is the update weight.

In an alternative, a set of momentum can be calculated by using the gradient of the weights, and then the updated weights are obtained by adding and subtracting the momentum and the weights.

Method for implementing BLAS (Basic Linear Algebra Subprograms) function using the apparatus as shown in FIG. 1a

GEMM calculation refers to the operation of matrix-matrix multiplication in the BLAS library. The usual representation of this operation is: C = alpha * op (S) * op (P) + beta * C, where S and P are the two matrices of the input, C is the output matrix, alpha and beta are scalar, op Represents some operation on the matrix S or P. In addition, there are some auxiliary integers as parameters to describe the width and height of the S and P of the matrix;

The steps to implement GEMM calculations using the apparatus of Figure 1a include:

The data type conversion operation circuit of the main processing circuit can perform data type conversion on the matrix S and the matrix P;

The conversion circuit of the main processing circuit performs respective op operations on the input matrix S and the matrix P;

In an alternative, op can be a transposition operation of the matrix; the matrix transposition operation can be implemented by using a matrix transposition circuit of the main processing circuit;

In an alternative, after performing the OP operation of the matrix S and the matrix P, the data conversion operation circuit of the main processing circuit may also perform a data type conversion operation, that is, the data conversion operation circuit will op(S) and op The data type of (P) is converted from floating point type data to fixed point type data, and then matrix multiplication operation as shown in Fig. 2b is performed.

In an alternative, the op of a certain matrix may be empty, and the op operation is not performed;

Completing the matrix multiplication calculation between op(S) and op(P) using the calculation method of the matrix multiplication matrix as shown in Fig. 2b using the apparatus as shown in Fig. 1a;

Multiplying each value in the result of op(S)*op(P) by an arithmetic logic unit of the main processing circuit by an alpha;

In an alternative, the operation of multiplying alpha by alpha is not performed;

Implementing the operation of beta*C using the arithmetic logic unit of the main processing circuit;

In an alternative, where beta is 1, no multiplication by beta is performed;

The vector operator circuit of the main processing circuit is used to implement the steps of adding the corresponding positions between the matrix alpha*op(s)*op(P) and beta*C to obtain the result of the GEMM calculation.

In an alternative, if the beta is 0, this step is not performed;

GEMV calculation refers to the operation of matrix-vector multiplication in the BLAS library. The usual representation of this operation is: C = alpha * op (S) * P + beta * C, where S is the input matrix, P is the input vector, C is the output vector, alpha and beta are scalars, and op is the pair Some operation of the matrix S;

The steps to implement GEMV calculations using the apparatus of Figure 1a are:

The data type conversion operation circuit of the main processing circuit can perform data type conversion on the input matrix S and the matrix P;

The conversion circuit of the main processing circuit performs a corresponding op operation on the input matrix S;

In an alternative, op can be a transposition operation of the matrix; the matrix transposition operation is implemented by using a conversion circuit of the main processing circuit;

In an alternative, the op of a certain matrix may be empty, and the transposition operation is not performed;

Completing the matrix-vector multiplication calculation between the matrix op(S) and the vector P using the calculation method of the matrix multiplication vector as described in Fig. 2a as shown in Fig. 1a;

Multiplying each value of the result of op(S)*P by an arithmetic operation unit of the main processing circuit by an alpha;

In an alternative, the operation of multiplying alpha by alpha is not performed;

Implementing the operation of beta*C using the arithmetic logic unit of the main processing circuit;

In an alternative, where beta is 1, no multiplication by beta is performed;

The result of GEMV is obtained by the step of adding the corresponding positions between the matrix alpha*op(S)*P and beta*C by using the vector operator circuit of the main processing circuit.

In an alternative, if the beta is 0, the step of adding is not performed;

Method of implementing an activation function using the apparatus of Figure 1a

An activation circuit of the main processing circuit inputs a vector to calculate an activation vector of the vector;

In an alternative, the main processing circuit activation circuit passes each value in the input vector through an activation function (the input of the activation function is a value and the output is also a value), and a correspondence of the output of the value to the output vector is calculated. position;

In an alternative, the activation function can be: y = max(m, x), where x is the input value, y is the output value, and m is a constant;

In an alternative, the activation function can be: y = tanh(x), where x is the input value and y is the output value;

In an alternative, the activation function can be: y = sigmoid (x), where x is the input value and y is the output value;

In an alternative, the activation function can be a piecewise linear function;

In an alternative, the activation function can be any function that inputs a number and outputs a number.

In an alternative, the source of the input vector is (including but not limited to):

External data source of the device;

In an alternative, the input data is from the device for performing a matrix multiplication vector;

In an alternative, the input data is from the device for performing a matrix multiplication matrix;

The main processing circuit of the device calculates a result;

In an alternative, the input data is derived from the calculations after the device main processing circuit implements the offset.

It should be noted that the above activation operation may be implemented by an arithmetic logic circuit and an accumulator circuit in the main processing circuit, or an activation circuit may be separately added to the main processing circuit to implement an activation operation.

The biasing operation is implemented using the device of Figure 1a:

The vector operator circuit of the main processing circuit can realize the function of adding two vectors or two matrices;

The vector operator circuit of the main processing circuit can be used to add a vector to each row of a matrix, or to the function on each column.

In an alternative, the matrix may be derived from the result of the matrix multiplication matrix operation performed by the apparatus;

In an alternative, the matrix may be derived from the result of the device performing a matrix multiplication vector operation;

In an alternative, the matrix may be derived from externally accepted data from the main processing circuitry of the device.

In an alternative, the vector may be from externally accepted data from the main processing circuitry of the device.

This includes, but is not limited to, the above data sources.

With reference to the foregoing embodiments, the specific embodiments involved in the operation instructions being forward operation instructions are set forth below. In the apparatus provided by the first aspect, the main processing circuit is configured to receive a forward operation instruction, and parse the forward operation instruction to obtain an ith layer of the forward operation instruction in a forward operation of the neural network The first operation instruction included, and the input data block and the weight data block required by the first operation instruction, wherein the range of i is an integer greater than or equal to 1, and less than or equal to n, as the i is greater than or equal to 2. The input data block is an output data block of the i-1th layer;

The main processing circuit is configured to divide the input data block into a broadcast data block according to the first operation instruction, and divide the convolution kernel data block into a data block; according to the first forward instruction The operation control determines to start the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast data block; The product instruction sends the processed first data block to at least one of the basic processing circuits connected to the main processing circuit;

The plurality of basic processing circuits are configured to determine, according to the operation control of the first operation instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. The operation obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing determined by the basic processing circuit a data block sent by the circuit, the second data block being associated with the processed first data block;

The main processing circuit is configured to process the operation result to obtain an instruction result of the first operation instruction, and complete an operation of the first operation instruction included in the ith layer.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block and a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit, the distribution data block, and the Processing the broadcast data block to obtain the processed distribution data block and the identification data block associated with the distribution data block, the processed broadcast data block, and the identification data block associated with the broadcast data block; and the processed distribution data block And the identification data block associated with the distribution data block is subjected to split processing to obtain a plurality of basic data blocks and identification data blocks respectively associated with the basic data blocks, and the plurality of basic data blocks and the plurality of basic data blocks are respectively And the associated identification data block is distributed to the basic processing circuit connected thereto, and the processed broadcast data block and the identification data block associated with the broadcast data block are broadcasted to the basic processing circuit connected thereto;

Correspondingly, the basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the broadcast data block and the identification data associated with the basic data block, and according to the The connection identification data block processes the broadcast data block and the basic data block to obtain a processed broadcast data block and a basic data block; performing a forward operation on the processed broadcast data block and the basic data block to obtain an operation result And transmitting the operation result to the main processing circuit; wherein the forward operation includes but is not limited to a combination of any one or more of the following: a convolution operation (ie, an inner product operation), a product operation, One or any combination of paranoid operation, full connection operation, GEMM operation, GEMV operation, activation operation;

The main processing circuit is configured to process the operation result to obtain the instruction result.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block, the main processing circuit is specifically configured to start processing, by the first mapping circuit, the processing data block to be processed. Distributing the data block and the identification data block associated with the distribution data block, or initiating the first mapping circuit to process the distribution data block according to the pre-stored identification data block associated with the distribution data block to obtain the processed distribution data block And performing the split processing on the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks respectively associated with the basic data blocks, and the plurality of basic data And the identification data block associated with each of the plurality of basic data blocks is distributed to a basic processing circuit connected thereto, and the broadcast data block is broadcasted to a basic processing circuit connected thereto;

The basic processing circuit is specifically configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, to obtain a processed broadcast data block; The broadcast data block and the processed basic data block perform a forward operation to obtain an operation result, and the operation result is transmitted to the main processing circuit.

In an optional embodiment, the main processing circuit is further configured to split the broadcast data block or the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data. a block and an identification data block associated with each of the plurality of partial broadcast data blocks; broadcasting the plurality of partial broadcast data blocks and the identification data blocks associated with each of the plurality of partial broadcast data blocks by the one or more broadcasts to the a base processing circuit; wherein the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.

Correspondingly, the basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the partial broadcast data block and the identification data block associated with the basic data block; The connection identification data is processed to process the partial broadcast data block and the basic data block to obtain a processed partial broadcast data block and the processed basic data block; and the processed partial broadcast data block and the processed The basic data block performs a forward operation.

In the apparatus provided by the first aspect, when the first data block includes a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit to process the broadcast data block, and obtain the processed Broadcast data block and the identification data block associated with the broadcast data block, or the first mapping circuit is configured to process the broadcast data block according to the pre-stored identification data block associated with the broadcast data block to obtain the processed broadcast data. Blocking the distributed data block to obtain a plurality of basic data blocks; distributing the plurality of basic data blocks to a basic processing circuit connected thereto, and the processed broadcast data block and the broadcast data block The associated identification data block is broadcast to the basic processing circuit connected thereto;

The basic processing circuit is configured to start, by the second mapping circuit, the basic data block to be processed according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and the processed broadcast The data block and the processed basic data block perform a forward operation to obtain an operation result, and the operation result is transmitted to the main processing circuit.

In an optional embodiment, the main processing circuit is further configured to split the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and the And an identifier data block associated with the plurality of partial broadcast data blocks; and the identification data blocks associated with the plurality of partial broadcast data blocks and the plurality of partial broadcast data blocks are broadcast to the basic processing circuit by one or more times; And the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.

Correspondingly, the basic processing circuit is configured to process the basic data block according to the identifier data block associated with the partial broadcast data block to obtain a processed basic data block; and the processed basic data block and the The partial broadcast data block performs a forward operation.

The device involved in the present application can be specifically referred to the foregoing FIG. 1a, the integrated circuit chip device is configured to perform a neural network forward operation, the neural network includes n layers, and the main processing circuit includes a first mapping circuit, the plurality of At least one of the basic processing circuits includes a second mapping circuit, and the first mapping circuit and the second mapping circuit are both configured to perform compression processing of respective data in the neural network operation;

The main processing circuit is configured to receive a forward operation instruction, and parse the forward operation instruction to obtain a first operation instruction included in an ith layer of the forward operation instruction in the forward operation of the neural network An input data block and a weight data block required by the first operation instruction, wherein the range of i is an integer greater than or equal to 1, and less than or equal to n, and if the i is greater than or equal to 2, the input data block is Output data block of layer i-1;

The main processing circuit is configured to divide the input data block into a broadcast data block according to the first operation instruction, and divide the convolution kernel data block into a data block; according to the first forward instruction The operation control determines to start the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast data block; The product instruction sends the processed first data block to at least one of the basic processing circuits connected to the main processing circuit;

The plurality of basic processing circuits are configured to determine, according to the operation control of the first operation instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. The operation obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing determined by the basic processing circuit a data block sent by the circuit, the second data block being associated with the processed first data block;

The main processing circuit is configured to process the operation result to obtain an instruction result of the first operation instruction, and complete an operation of the first operation instruction included in the ith layer.

Optionally, the integrated circuit chip device may include a branch processing circuit connected between the main processing circuit and the basic processing circuit for converting data between the main processing circuit and the basic processing circuit. Setting the branch processing circuit can increase the number of basic processing circuits connected to the main processing circuit, and also reduce the overhead of data transmission of the main processing circuit.

As shown in FIG. 6a, a forward operation of a neural network according to an embodiment of the present disclosure, each layer uses its own input data and weights to calculate corresponding output data according to an operation rule specified by a layer type;

The forward operation process (also called inference) of the neural network is a process of processing the input data of each layer layer by layer, and obtaining the output data after a certain calculation, which has the following characteristics:

Input to a layer:

The input to a layer can be the input data of the neural network;

The input of one layer can be the output of other layers;

The input of a layer can be the output of the layer at a moment (corresponding to the case of a cyclic neural network);

A layer can simultaneously acquire input from a plurality of the above input sources;

The output of a layer:

The output of a layer can be used as the output of the neural network;

The output of one layer can be the input of other layers;

The output of a layer can be the input of this layer at the next moment (in the case of a cyclic neural network);

The output of a layer may output a result to the plurality of output directions described above;

Specifically, the types of operations of the layers in the neural network include, but are not limited to, the following:

Convolution layer (ie performing convolution operations);

Fully connected layer (ie performing a full join operation);

Normalized (regularized) layer: includes LRN (Local Response Normalization) layer, BN (Batch Normalization) layer, and the like;

Pooling layer

Activation layer: including but not limited to the following types of Sigmoid layer, ReLU layer, PReLu layer, LeakyReLu layer, Tanh layer;

The inverse operation of the layer, the inverse operation of each layer needs to perform a two-part operation: one part is to calculate the weight gradient using the output data gradient which may be sparse representation and the input data which may be sparse representation (for the weight The value update step updates the weight of this layer, and the other part uses the output data gradient, which may be sparse representation, and the weight of the sparse representation, to calculate the input data gradient (used as the next layer in the inverse operation). Output a data gradient for its inverse operation);

The inverse operation reverses the gradient from the last layer in the reverse order of the forward operation.

In an alternative, the inverse of the output data gradient calculated by a layer can be derived from:

The gradient of the return function of the last loss function or cost function of the neural network;

Input data gradient for other layers;

Input data gradient at a moment on this layer (corresponding to the case of a cyclic neural network);

A layer can simultaneously obtain an output data gradient from a plurality of the above sources;

After performing the inverse operation of the neural network, the gradient of the weights of the layers is calculated. In this step, the first input buffer and the second input buffer of the device are respectively used to store the weights of the layer. And the gradient of the weights, and then use the weight gradient to update the weights in the arithmetic unit;

The operations mentioned above are all operations of a layer in a neural network. For a multi-layer neural network, the implementation process is that, in the forward operation, when the previous artificial neural network is executed, the next layer of operations is performed. The instruction will operate the output data calculated in the operation unit as the input data of the next layer (or perform some operations on the output data as the input data of the next layer), and at the same time, replace the weight with the next one. The weight of one layer; in the reverse operation, when the reverse operation of the upper artificial neural network is completed, the next layer of operation instructions will use the input data gradient calculated in the operation unit as the output data of the next layer. Gradient operation (or some operation on the input data gradient and then as the output data gradient of the next layer), and replace the weight with the weight of the next layer; as shown in Figure 6b, the dotted line in the figure The arrows indicate the reverse operation, the solid arrows indicate the forward operation, and the labels below the respective figures indicate the meaning of the figure.

The data involved in the present application (that is, the data in the data block) is the compressed data, and can be implemented in the first mapping circuit and the second mapping circuit. For details, refer to the related description in the foregoing embodiment.

The specific implementation of several neural network forward operations is exemplarily given below. When the first operation instruction is a convolution instruction, the input data block (ie, the data in the input data block) is convolution input data, and the weight data (block) is a convolution kernel. The convolution operation is described below. One block in the following figure represents a data, and the input data is represented by Figure 7a (N samples, each sample has C channels, and the feature map of each channel has a height H and a width W). The weight, that is, the convolution kernel, is represented by Figure 7b (there are M convolution kernels, each convolution kernel has C channels, and the height and width are KH and KW, respectively). The rules for convolution operations are the same for N samples of input data. The following is a process of convolution on a sample. On one sample, each of the M convolution kernels must perform the same. Operation, each convolution kernel operation obtains a plane feature map, and M convolution kernels finally calculate M plane feature maps (for one sample, the convolution output is M feature maps), for a convolution kernel To perform an inner product operation at each plane position of a sample, and then slide in the H and W directions. For example, Figure 7c shows a convolution kernel performing inner product operations at the position of the lower right corner of a sample of the input data. Corresponding to the figure; Fig. 7d shows that the position of the convolution slides one space to the left and Fig. 7e shows that the position of the convolution slides up one space.

Specifically, the method of the convolution processing may be performed by using a chip structure as shown in FIG. 1a, and the first mapping circuit of the main processing circuit may process data in part or all of the convolution kernel of the weight to obtain a corresponding mask. Data and processed weight data (that is, data in some or all of the convolution kernels of the processed weights).

The control circuit of the main processing circuit sends the data in part or all of the convolution kernel of the weight (the data can be the original weight data or the processed weight data) to be directly connected to the main processing circuit through the horizontal data input interface. The basic processing circuit (also referred to as a base unit); at the same time, the control circuit sends the mask data associated with the data to the basic processing circuit connected to the main processing circuit;

In an alternative, the control circuit of the main processing circuit sends a certain number or part of the data of a convolution kernel to a basic processing circuit each time (for example, for a basic processing circuit, The first time, the first number of the third line is transmitted, the second number of the third line of data is transmitted for the second time, the third number of the third line is transmitted for the third time, ..., or the third line is transmitted for the first time. The first two numbers, the second time sends the third and third numbers of the third line, the third time sends the fifth and fifth numbers of the third line...;) At the same time, the control circuit takes a weight in one of the volumes The mask data corresponding to the product core also uses the above-mentioned one or a part of data to be sent to the basic processing circuit each time;

In another alternative, the control circuit of the main processing circuit sends the data of some convolution kernels of the weights to each of the basic processing circuits each time (for example, For a certain basic processing circuit, the first number of lines in the 3rd, 4th, and 5th lines is transmitted for the first time, and the second number of the 3rd, 4th, and 5th lines is transmitted for the second time, and the third time is transmitted. 3, 4, 5 lines, the 3rd number of each line..., or the first 2, 4, 5 lines, the first two lines, the second time, the 3rd, 4th, 5th line, the third line And the fourth number, the third time, the third, fourth, fifth line, the fifth and sixth numbers of each line...;) correspondingly, the control circuit will be associated with some of the convolution kernels in the weight The mask data also sends a number or a portion of data to the basic processing circuit each time in the same manner as described above;

The control circuit of the main processing circuit divides the input data according to the position of the convolution, and the control circuit of the main processing circuit sends the data in part or all of the convolution position in the input data to the main processing circuit through the vertical data input interface. Corresponding at least one basic processing circuit; correspondingly, the control circuit also divides the mask data associated with the input data according to the position of the convolution, and accordingly the control circuit also also partially or completely the input data The mask data corresponding to the data in the convolution position is also sent together to the basic processing circuit electrically connected to the main processing circuit;

In an alternative, the control circuit of the main processing circuit sends a certain number or part of the data of a certain convolution position in the input data and the mask data associated with the data to a certain basic processing circuit; For example, for a certain basic processing circuit, the first number of the third column is transmitted for the first time, the second number of the third column of data is transmitted for the second time, and the third number of the third column is transmitted for the third time... , or send the first two numbers in the third column for the first time, the third and fourth numbers in the third column for the second time, and the fifth and sixth numbers in the third column for the third time...;)

In another alternative, the control circuit of the main processing circuit sends data of a certain convolution position in the input data and mask data associated with the data each time by a number or a part of the data. a basic processing circuit; (for example, for a basic processing circuit, the first number of the third, fourth, and fifth columns is sent for the first time, and the second, third, fourth, and fifth columns are for the second 2 digits, the 3rd number of the 3rd, 4th, and 5th columns, the 3rd number of each column..., or the 1st transmission of the 3rd, 4th, and 5th columns, the first two digits, the second transmission of the third digit , 4, 5 columns, 3rd and 4th, and 3rd, 3rd, 4th, 5th, 5th and 6th, each column...;)

The basic processing circuit receives the data of the weight (specifically, the data of the convolution kernel in the weight (referred to as weight data) or the mask data associated with the weight data), and then outputs the data through the horizontal data thereof. The interface transmits to the next basic processing circuit connected thereto; after receiving the data of the input data (the data can be input data sent by the main processing circuit and the identification mask data associated with the input data), the basic processing circuit passes the data through the interface The vertical data output interface is transmitted to the next basic processing circuit connected thereto;

Specifically, the control circuit of the main processing circuit may send the input data and the mask data associated with the input data to the basic processing circuit, and the basic processing circuit receives the input data and the mask data associated with the input data;

Each of the basic processing circuits operates on the received data; specifically, the basic processing circuit can enable the second mapping circuit to associate the mask data associated with the input data with the mask data associated with the weight data (ie, the convolution kernel associated with the weights) The mask data is obtained by the connection identification data; and the connection identification data is used to select the input data and the data in which the absolute value of the weight data is greater than a preset threshold is multiplied;

In an optional solution, if the data is received in each basic processing circuit (specifically, the data block to be calculated, such as the data in the convolution kernel in the weight and the mask data, the input data, or the data associated with the data When the amount of data of the mask data of the input data exceeds a preset threshold, the basic processing circuit will no longer receive new input data, such as data in some convolution kernels of the weights subsequently sent by the main processing circuit and The data corresponds to the associated mask data and the like until the base processing circuit has sufficient buffer/storage space, and then receives the newly transmitted data of the main processing circuit.

In an alternative, the base processing circuit calculates a multiplication of one or more sets of two data at a time, and then accumulates the result on a register and/or an on-chip buffer;

In an alternative, the base processing circuit calculates the inner product of one or more sets of two vectors at a time, and then accumulates the result on the register and/or the on-chip buffer;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an alternative, the result of the calculation may be the final result or an intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface, and if not, the result is output in the direction of the basic processing circuit that can be directly output to the main processing circuit.

After receiving the calculation result from other basic processing circuits, the basic processing circuit transmits the data to other basic processing circuits or main processing circuits connected thereto;

Output results in a direction that can be directly output to the main processing circuit (for example, the bottom row of the basic processing circuit outputs its output directly to the main processing circuit, and the other basic processing circuits transfer the operation result from the vertical output interface);

The main processing circuit receives the result of the inner product operation of each basic processing circuit, and the output result is obtained.

The tensor multiplying tensor operation can be accomplished using the apparatus shown in Figure 1a, which is the same as the data block described above, which can be a matrix, a vector, a three-dimensional data block, a four-bit data block, and a high-dimensional data block. Combination of any one or more of the above; see Figure 2a and Figure 2b above for a specific implementation of matrix multiplication vector and matrix multiplication matrix operation, respectively. Specifically, when the forward operation indicated by the first operation instruction is a matrix multiplication matrix operation, the input data is the first matrix of the matrix multiplication matrix operation, and the weight is the second of the matrix multiplication matrix operation For the matrix, refer to the flowchart of the operation method of the matrix multiplication matrix in the foregoing example of FIG. 2b. When the forward operation indicated by the first operation instruction is a matrix multiplication vector operation, the input data is a first matrix of the matrix multiplication vector operation, and the weight is a vector of the matrix multiplication vector operation, specifically See the flow chart of the operation method of the matrix multiplication matrix in the foregoing example of Fig. 2a.

The present disclosure also provides an integrated circuit chip device for performing a forward operation of a neural network, the neural network comprising a plurality of layers, the device comprising: a processing circuit and an external interface;

The external interface is configured to receive a forward operation instruction;

The processing circuit is configured to parse the forward operation instruction to obtain a first operation instruction included in the i-th layer of the forward operation instruction and the input required by the first operation instruction in the forward operation of the neural network a data block and a weight data block, wherein the value range of i is an integer greater than or equal to 1, and less than or equal to n, as the i is greater than or equal to 2, and the input data block is an output data block of the i-1th layer. ;

The processing circuit is further configured to perform the operation of the first operation instruction included in the ith layer of the forward operation by the input data and the weight data.

The present disclosure also discloses a neural network computing device including one or more chips as shown in FIG. 9e for acquiring data to be processed and control information from other processing devices, performing specified neural network operations, and executing results. Passed to the peripheral device through the I/O interface. Peripherals such as cameras, monitors, mice, keyboards, network cards, wifi interfaces, servers. When more than one chip as shown in FIG. 9e is included, the chips shown in FIG. 9e can be linked and transmitted through a specific structure, for example, by interconnecting and transmitting data through the PCIE bus to support larger-scale nerves. Network computing. At this point, you can share the same control system, or you can have separate control systems; you can share memory, or each accelerator has its own memory. In addition, the interconnection method can be any interconnection topology. The neural network computing device has high compatibility and can be connected to various types of servers through a PCIE interface.

The specific embodiments involved in the operation instructions as training instructions are set forth below. In the apparatus provided by the first aspect, the integrated circuit chip device is configured to receive a training instruction, determine first layer input data and first layer weight group data according to the training instruction, input data to the first layer, and first The layer weight group data performs the n-th layer forward operation of the neural network to obtain the nth output result of the forward operation;

The main processing circuit is further configured to obtain an nth output result gradient according to the nth output result, and obtain an nth reverse operation nth inverse operation instruction and the nth inverse operation according to the training instruction The nth layer input data and the nth layer weight group data required by the instruction; dividing the nth output result gradient, the nth layer input data, and the nth layer weight group data according to the nth reverse operation instruction a broadcast data block and a distribution data block; determining, according to the operation control of the nth reverse operation instruction, starting the first mapping circuit to process the first data block, and obtaining the processed first data block; the first data block Include the distribution data block and/or the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the nth reverse operation instruction Processing circuit

The plurality of basic processing circuits are configured to determine, according to the operation control of the nth reverse operation instruction, whether to start the second mapping circuit to process the second data block, and execute the nerve in parallel according to the processed second data block An operation in the network obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is determined by the basic processing circuit to receive the a data block sent by the main processing circuit, the second data block being associated with the processed first data block;

The main processing circuit is further configured to process the operation result to obtain an nth layer weight group gradient and an nth layer input data gradient, and apply the nth layer weight group gradient to the nth layer weight group data. Update

The integrated circuit chip device is further configured to perform an n-1 layer inverse operation on the nth layer input data gradient as an n-1th output gradient of the n-1th layer to obtain an n-1 layer weight group gradient, and apply The n-1 layer weight group gradient updates the weight group data of the corresponding layer, and the weight group data includes at least two weights.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block and a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit, the distribution data block, and the Processing the broadcast data block to obtain the processed distribution data block and the identification data block associated with the distribution data block, the processed broadcast data block, and the identification data block associated with the broadcast data block; and the processed distribution data block And the identification data block associated with the distribution data block is subjected to split processing to obtain a plurality of basic data blocks and identification data blocks respectively associated with the basic data blocks, and the plurality of basic data blocks and the plurality of basic data blocks are respectively And the associated identification data block is distributed to the basic processing circuit connected thereto, and the processed broadcast data block and the identification data block associated with the broadcast data block are broadcasted to the basic processing circuit connected thereto;

Correspondingly, the basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the broadcast data block and the identification data associated with the basic data block, and according to the The connection identification data block processes the broadcast data block and the basic data block to obtain a processed broadcast data block and a basic data block; performing inverse operation on the processed broadcast data block and the basic data block to obtain an operation result Transmitting the operation result to the main processing circuit; wherein. The reverse operation includes, but is not limited to, a combination of any one or more of the following: a convolution operation (ie, an inner product operation), a product operation, a paranoid operation, a full connection operation, a GEMM operation, a GEMV operation, and an activation operation. One or any combination of

The main processing circuit is configured to process the operation result to obtain the instruction result.

In the apparatus provided by the first aspect, when the first data block includes a distribution data block, the main processing circuit is specifically configured to start processing, by the first mapping circuit, the processing data block to be processed. Distributing the data block and the identification data block associated with the distribution data block, or initiating the first mapping circuit to process the distribution data block according to the pre-stored identification data block associated with the distribution data block to obtain the processed distribution data block And performing the split processing on the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks respectively associated with the basic data blocks, and the plurality of basic data And the identification data block associated with each of the plurality of basic data blocks is distributed to a basic processing circuit connected thereto, and the broadcast data block is broadcasted to a basic processing circuit connected thereto;

The basic processing circuit is specifically configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, to obtain a processed broadcast data block; The broadcast data block and the processed basic data block perform a reverse operation to obtain an operation result, and the operation result is transmitted to the main processing circuit.

In an optional embodiment, the basic processing circuit is specifically configured to start, by the second mapping circuit, the connection identifier data block according to the identifier data block associated with the partial broadcast data block and the identifier data block associated with the basic data block. Processing the partial broadcast data block and the basic data block according to the connection identification data to obtain a processed partial broadcast data block and the processed basic data block; and the processed partial broadcast data block and the The processed basic data block performs a reverse operation.

In the apparatus provided by the first aspect, when the first data block includes a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit to process the broadcast data block, and obtain the processed Broadcast data block and the identification data block associated with the broadcast data block, or the first mapping circuit is configured to process the broadcast data block according to the pre-stored identification data block associated with the broadcast data block to obtain the processed broadcast data. Blocking the distributed data block to obtain a plurality of basic data blocks; distributing the plurality of basic data blocks to a basic processing circuit connected thereto, and the processed broadcast data block and the broadcast data block The associated identification data block is broadcast to the basic processing circuit connected thereto;

The basic processing circuit is configured to start, by the second mapping circuit, the basic data block to be processed according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and the processed broadcast The data block and the processed basic data block perform a reverse operation to obtain an operation result, and the operation result is transmitted to the main processing circuit.

In an optional embodiment, the main processing circuit is further configured to split the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and the And an identifier data block associated with the plurality of partial broadcast data blocks; and the identification data blocks associated with the plurality of partial broadcast data blocks and the plurality of partial broadcast data blocks are broadcast to the basic processing circuit by one or more times; And the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.

Correspondingly, the basic processing circuit is configured to process the basic data block according to the identifier data block associated with the partial broadcast data block to obtain a processed basic data block; and the processed basic data block and the The partial broadcast data block performs a reverse operation.

In the apparatus provided by the first aspect, the nth output result gradient is: one or any combination of a vector, a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block;

The nth layer input data may be represented by a tensor, which may specifically be: one or any combination of a vector, a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block;

The n-layer weight group data may be represented by a tensor, which may specifically be: one or any combination of a vector, a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block.

As shown in Figure 8, the steps of neural network training include:

Each layer in a (multi-layer) neural network performs a forward operation in sequence;

Performing a reverse operation in the order of the opposite layers to obtain a weight gradient;

Using the calculated gradient of weights to update the weight of the forward operation;

This is the sequential iteration of the training of the neural network. The entire training process requires repeated execution (ie multiple iterations) to process this process multiple times.

Referring to FIG. 1a, FIG. 1a is an integrated circuit chip device for performing training of a neural network, where the neural network includes n layers, and the n value ranges from an integer of 2 or more, characterized in that The integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, and at least one of the plurality of basic processing circuits includes a second mapping circuit, the first mapping The circuit and the second mapping circuit are both used to perform compression processing of each data in the neural network operation;

The integrated circuit chip device is configured to receive a training instruction, determine first layer input data and first layer weight group data according to the training instruction, and perform a neural network on the first layer input data and the first layer weight group data. The n-th layer forward operation obtains the nth output result of the forward operation;

The main processing circuit is further configured to obtain an nth output result gradient according to the nth output result, and obtain an nth reverse operation nth inverse operation instruction and the nth inverse operation according to the training instruction The nth layer input data and the nth layer weight group data required by the instruction; dividing the nth output result gradient, the nth layer input data, and the nth layer weight group data according to the nth reverse operation instruction a broadcast data block and a distribution data block; determining, according to the operation control of the nth reverse operation instruction, starting the first mapping circuit to process the first data block, and obtaining the processed first data block; the first data block Include the distribution data block and/or the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the nth reverse operation instruction Processing circuit

The plurality of basic processing circuits are configured to determine, according to the operation control of the nth reverse operation instruction, whether to start the second mapping circuit to process the second data block, and execute the nerve in parallel according to the processed second data block An operation in the network obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is determined by the basic processing circuit to receive the a data block sent by the main processing circuit, the second data block being associated with the processed first data block;

The main processing circuit is further configured to process the operation result to obtain an nth layer weight group gradient and an nth layer input data gradient, and apply the nth layer weight group gradient to the nth layer weight group data. Update

The integrated circuit chip device is further configured to perform an n-1 layer inverse operation on the nth layer input data gradient as an n-1th output gradient of the n-1th layer to obtain an n-1 layer weight group gradient, and apply The n-1 layer weight group gradient updates the weight group data of the corresponding layer, and the weight group data includes at least two weights.

Neural network training method

All or part of the data involved in the neural network training process may be processed data. For details, refer to the foregoing embodiment, which is obtained by the first mapping circuit and/or the second mapping circuit, and is not described here.

It should be noted that different times of the training process (specifically, different iterations or initialization time), different stages of the training process (ie, forward or reverse operations), different layers, and different layers A data block (ie, multiple input data blocks, an output data block), or a sub-block of a different portion divided in the same data block, may refer to a processed data block.

The following is a practical example to illustrate the specific implementation of neural network training. Figure 6b shows the specific calculation of neural network training for single-layer operation. Figure 6b shows the forward operation of single-layer neural network. The reverse operation of the single layer neural network is shown by the dashed line in Fig. 6b. Specifically, the forward operation of the layer is performed according to the input data and the weight or the parameter, and the output data is obtained, and then the preset rule is calculated according to the output data (the preset rule can be set by the manufacturer according to its own needs, here) The specific operation step of the preset rule operation is not limited to obtain the output data gradient of the layer. Then, the inverse operation of the layer neural network can be performed according to the input data, the weight or the parameter of the layer, and the output data gradient, and the gradient of the input data and the gradient of the weight or the parameter of the layer can be obtained, and the weight obtained by the calculation can be used. Or the gradient of the parameter corresponds to updating the weight or parameter of the layer, that is, the neural network training of the layer is completed.

In the specific implementation process, the data involved in the forward operation or the reverse operation may be the processed data. Taking the forward operation as an example, the technical solution provided by the embodiment of the present application may be an inverse operation according to the layer. The instruction determines whether to initiate a correlation mapping circuit (specifically, the first mapping circuit and/or the second mapping circuit) to process the input data and/or the weight, and then perform the layer operation using the processed input data and/or weights . For the principle of the above data processing, refer to the related description in the foregoing embodiment, and details are not described herein again. It should be understood that using the above processed data to perform neural network operations can greatly reduce the transmission overhead between the calculators. In addition, for the calculator, the data storage space of fewer bits is also small. That is, the storage overhead will be smaller, the amount of calculation will be reduced, that is, the calculation overhead will be reduced, so the calculation overhead and the storage overhead can be reduced.

The following is a structural diagram of neural network training for matrix multiplication and convolution, taking FIG. 9a and FIG. 9b as examples. The calculation mode of the layer shown in FIG. 9a is matrix multiplication, and the operation mode of the layer shown in FIG. 9b is a convolution operation, assuming that the input data and the weight of the layer are all matrix, for convenience, the input data is matrix I. For example, the weight is based on the matrix W, where the output data = matrix I * matrix W; if the matrix I and the matrix W are sparse matrices with larger dimensions, the sparse matrix here means that the matrix includes absolute values less than Equal to the preset threshold, or more data with 0. The larger the dimension can be understood as the larger the number of columns of the matrix I and the matrix W and the sum of the number of rows, that is, the matrix I and the matrix W can be considered to occupy a larger space in the memory and/or the register and the calculation amount is larger. At this time, according to the conventional matrix multiplication processing, the amount of data calculation is large; in order to improve the data processing efficiency, the matrix I and the matrix W need to be processed, and then the matrix multiplication operation is performed.

For example, the matrix I is a sparse matrix of 1000*1000, and the matrix W is also a sparse matrix of 1000*1000. Then, the sum of the number of columns and the number of rows is 2000, the number of which is large, and the corresponding calculation amount is larger, matrix multiplication The multiplication operation of the inner product of the matrix is 109 times. For this technical solution, since the number of the matrix I and the matrix W is large, it is impossible to transmit and calculate all the data at a time, so that the same data may be transmitted multiple times. Calculation, if the data processing method is used at this time, the data processing of the two sparse matrices can greatly reduce the dimension of the matrix I and the matrix W (ie, the amount of data), thereby greatly reducing the amount of data transmitted and Calculate the amount, which in turn reduces transmission overhead and computational overhead.

A schematic diagram of the specific structure of the multilayer neural network training is shown in Figures 9c and 9d. As shown in Figure 9c, the direction of the dashed arrow shows a reverse operation. For the inverse operation, the output of the inverse operation is the output data gradient; when the output data gradient is the last layer of the iterative calculation of the multilayer neural network, the output data gradient is specifically the last layer of the iterative calculation The output data is subjected to a preset operation (the preset operation can be set by the manufacturer according to his own needs, and the specific operation steps of the preset operation are not limited herein); if the output data gradient is not an iterative for the multilayer neural network The last layer of the calculation, for example, the output data gradient is the nth layer calculated for this iteration, then the output data gradient of the nth layer can be the input data gradient calculated by the n+1th layer inverse operation. Similarly, FIG. 4d can be understood. FIG. 4h can be a schematic diagram of multi-layer convolutional neural network training (including forward operation and reverse operation), and other operations in the figure can be expressed as layers other than the convolution layer. Or the operation between the layers, not limited.

The present disclosure also provides an integrated circuit chip device for performing training of a neural network, the neural network comprising a plurality of layers, the integrated circuit chip device comprising: a processing circuit and an external interface;

The external interface is configured to receive a training instruction;

The processing circuit is configured to determine first layer input data and first layer weight data according to the training instruction, and perform n-th layer forward operation of the neural network by using the first layer input data and the first layer weight data to obtain the nth Output result

The processing circuit is further configured to obtain an nth output result gradient according to the nth output result, and obtain an nth reverse operation instruction of the nth layer reverse operation and the nth reverse operation instruction according to the training instruction. The nth layer input data and the nth layer weight group data; performing n-layer reversal of the neural network according to the nth reverse operation instruction, the nth output result gradient, the nth layer input data, and the nth layer weight group data The operation obtains n weight gradients of the n-layer operation;

The processing circuit is further configured to update the n weights of the n-layer operation by applying the n weight gradients.

The present invention also provides a chip, the chip comprising a computing device, the computing device comprising:

Including a main processing circuit, the data involved in the main processing circuit may be compressed processed data. In an optional embodiment, the compressed processed data includes at least one input neuron or at least one weight. Each of the at least one neuron is greater than a first threshold or each of the at least one weight is greater than a second threshold. The first threshold and the second threshold are customized by the user side, and they may be the same or different.

In an alternative, the main processing circuit includes a first mapping circuit;

In an alternative, the main processing circuit includes an arithmetic unit that performs data compression processing, such as a vector operation unit;

Specifically, a data input interface including receiving input data;

In an alternative, the received data source may be: external to the neural network operation circuit device or part or all of the basic processing circuit of the neural network operation circuit device;

In an alternative, the data input interface may have multiple; specifically, may include a data output interface for outputting data;

In an alternative, the outgoing data may be: external to the neural network computing device or part or all of the basic processing circuit of the neural network computing circuit device;

In an alternative, the data output interface may have multiple;

In an alternative, the main processing circuit includes an on-chip buffer and/or a register;

In an optional implementation, the main processing circuit includes an operation unit, and can perform a data operation;

In an optional aspect, the main processing circuit includes an arithmetic operation unit;

In an optional implementation, the main processing circuit includes a vector operation unit, and the operation may be performed on a group of data at the same time; specifically, the arithmetic operation and/or the vector operation may be any type of operation, including but not Limited to: two numbers plus, minus, multiply and divide, one number plus constant plus, minus, multiply and divide, perform exponential operation, power operation, logarithm operation, and various nonlinear operations on one number, perform comparison operation on two numbers, logical operation Wait. Two vectors are added, subtracted, multiplied, and divided. Each element in a vector is added, subtracted, multiplied, and divided by a constant. Each element in the vector is subjected to exponential operations, power operations, logarithmic operations, and various nonlinear operations. Each of the two corresponding elements in the vector performs a comparison operation, a logical operation, and the like.

In an alternative, the main processing circuit includes a data rearranging unit for transmitting data to the basic processing circuit in a certain order, or rearranging the data in place in a certain order;

In an alternative, the order of the data arrangement comprises: performing a dimensional order transformation on a multi-dimensional data block; the order of the data arrangement may further comprise: segmenting a data block to send to a different basis. Processing circuit.

The computing device also includes a plurality of basic processing circuits: each of the basic processing circuits is configured to calculate an inner product of the two vectors, the calculation method is, the two sets of numbers received by the basic processing circuit, and the elements in the two sets of numbers are corresponding Multiply, and accumulate the results of the multiplication; the result of the inner product is transmitted, where it is transmitted according to the position of the basic processing circuit, and may be transmitted to other basic processing circuits, or directly to the main processing circuit.

The data involved in the basic processing circuit may be compressed processed data. In an optional embodiment, the compressed processed data includes at least one input neuron or at least one weight, the at least one neural Each of the neurons is greater than a first threshold or each of the at least one weight is greater than a second threshold. The first threshold and the second threshold are customized by the user side, and they may be the same or different.

In an alternative, the basic processing circuit includes a second mapping circuit;

In an alternative, the basic processing circuit includes a vector operation unit that performs data compression processing;

Specifically, a memory unit including an on-chip buffer and/or a register is included;

Specifically, a data input interface including one or more received data;

In an alternative, two data input interfaces are included, and one or more data can be respectively obtained from two data input interfaces at a time;

In an alternative, the basic processing circuit may receive the input data from the data input interface and save it in a register and/or an on-chip buffer;

The source of the data input interface receiving data may be: other basic processing circuits and/or main processing circuits.

a main processing circuit of the neural network operation circuit device;

Other basic processing circuits of the neural network operation circuit device (the neural network operation circuit device has a plurality of basic processing circuits);

Specifically, the data output interface includes one or more transmission output data;

In an alternative, one or more data can be transmitted from the data output interface;

Specifically, the data transmitted through the data output interface may be: data received from the data input interface, data stored in the on-chip buffer and/or register, multiplier operation result, accumulator operation result, or inner product operator operation One or any combination of the results.

In an alternative, three data output interfaces are included, two of which correspond to two data input interfaces, each layer outputs a layer of data received from the data input interface, and a third data output interface Responsible for outputting the operation result;

Specifically, the data output interface may transmit data in a direction where the data source and the data direction here determine the connection relationship of the basic processing circuit in the device.

a main processing circuit of the neural network operation circuit device;

Other basic processing circuits of the neural network operation circuit device, the neural network operation circuit device having a plurality of basic processing circuits;

Specifically, the arithmetic operation circuit includes: one or more multiplier circuits, one or more accumulator circuits, one or more of one or more circuits that perform two sets of inner product operations. combination.

In an alternative, two numbers of multiplication operations can be performed, the results of which can be stored in on-chip buffers and/or registers, or directly added to registers and/or on-chip buffers;

In an alternative, an inner product operation of the two sets of data can be performed, the result of which can be stored in an on-chip buffer and/or register, or directly added to the register and/or on-chip buffer;

In an alternative, the data accumulation operation can be performed to accumulate the data into the on-chip buffer and or the register;

Specifically, the accumulated data of the accumulator circuit may be: data received from the data input interface, data stored in the on-chip buffer and/or register, multiplier operation result, accumulator operation result, inner product operator operation One or any combination of the results.

It should be noted that the "data input interface" and "data output interface" used in the above description of the basic processing circuit refer to the data input and output interface of each basic processing circuit, rather than the data input and output of the entire device. interface.

The present invention also provides a neural network processor board, including the integrated circuit chip device described in the foregoing embodiments, as shown in FIG. 1a.

In the card provided by the present invention, the neural network chip package structure further includes: a heat sink.

In the card provided by the present invention, the neural network chip package structure is any one of the following packages: a flip chip ball grid array package, a thin quad flat package, a quad flat package with a heat sink, and no lead Quad flat package, small pitch quad flat package.

The present disclosure also discloses a combined processing apparatus including the above-described neural network computing device, a universal interconnect interface, and other processing devices (i.e., general processing devices). The neural network computing device interacts with other processing devices to perform user-specified operations. As shown in Figure 10a below is a schematic diagram of a combined processing device.

Other processing devices include processor types of one or more of general purpose/dedicated processors such as a central processing unit CPU, a graphics processing unit GPU, a neural network processor, and the like. The number of processors included in other processing devices is not limited. The other processing device serves as an interface between the neural network computing device and external data and control, including data handling, and completes basic control such as opening and stopping of the neural network computing device; other processing devices may also cooperate with the neural network computing device to complete the computing task.

A universal interconnect interface for transmitting data and control commands between the neural network computing device and other processing devices. The neural network computing device acquires the required input data from other processing devices and writes the storage device on the slice of the neural network computing device; the control command can be obtained from the other processing device and written into the control cache on the slice of the neural network computing device; The data in the storage module of the neural network computing device can be read and transmitted to other processing devices.

As shown in FIG. 10b, optionally, the structure further includes a storage device for storing data required by the operation unit/operation device or other operation unit, and is particularly suitable for the data required for calculation in the neural network operation device. Data that cannot be saved in the internal storage of other processing devices.

The combined processing device can be used as a SOC on-chip system for mobile phones, robots, drones, video monitoring devices, etc., effectively reducing the core area of the control part, increasing the processing speed, and reducing the overall power consumption. In this case, the universal interconnect interface of the combined processing device is coupled to certain components of the device. Some components such as camera, monitor, mouse, keyboard, network card, wifi interface.

Embodiments of the present disclosure provide a neural network processor board that can be used in numerous general purpose or dedicated computing system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, smart homes, home appliances, multiprocessor systems, microprocessor-based systems, robots, programmable consumer electronics devices, personal computers (personal computers) , PC), small computer, mainframe computer, distributed computing environment including any of the above systems or devices, and so on.

Please refer to FIG. 11a. FIG. 11a is a schematic structural diagram of a neural network processor card according to an embodiment of the present disclosure. As shown in FIG. 11a, the neural network processor board 10 includes a neural network chip package structure 11, a first electrical and non-electrical connection device 12, and a first substrate 13.

The disclosure is not limited to the specific structure of the neural network chip package structure 11. Alternatively, as shown in FIG. 11b, the neural network chip package structure 11 includes: a neural network chip 111, a second electrical and non-electrical connection device 112, and a Two substrates 113.

The specific form of the neural network chip 111 involved in the disclosure is not limited. The above neural network chip 111 includes, but is not limited to, a neural network chip integrated with a neural network processor, and the above silicon wafer may be made of silicon material, germanium material, quantum material or molecule. Made of materials, etc. The neural network wafer can be packaged according to actual conditions (for example, a more severe environment) and different application requirements, so that most of the neural network wafer is wrapped, and the pins on the neural network wafer are passed through the gold wire. The conductors are connected to the outside of the package structure for electrical connection to the outer layer.

The specific structure of the neural network chip 111 is not limited in this disclosure. Alternatively, please refer to the device shown in FIG. 1a or 1b.

The present disclosure is not limited to the types of the first substrate 13 and the second substrate 113, and may be a printed circuit board (PCB) or a printed wiring board (PWB), and may be other circuit boards. There are no restrictions on the materials used to make the PCB.

The second substrate 113 of the present disclosure is used to carry the neural network chip 111, and the neural network chip package structure 11 is obtained by connecting the above-mentioned neural network chip 111 and the second substrate 113 through the second electrical and non-electrical connection device 112. The neural network chip 111 is protected to further encapsulate the neural network chip package structure 11 and the first substrate 13.

The structure of the specific second electrical and non-electrical connection device 112 and the corresponding configuration of the packaging method are not limited, and a suitable packaging method can be selected according to actual conditions and different application requirements and simply improved, for example, flip chip Flip Chip Ball Grid Array Package (FCBGAP), Low-profile Quad Flat Package (LQFP), Quad Flat Package with Heat Sink (HQFP), None Packaged in a Quad Flat Non-lead Package (QFN) or a Fine-Pitch Ball Grid Package (FBGA).

Flip Chip is suitable for the case where the area after packaging is required to be high or sensitive to the inductance of the wire and the transmission time of the signal. In addition, wire bonding can be used to reduce the cost and increase the flexibility of the package structure.

Ball Grid Array, which can provide more pins, and the average lead length of the pins is short, which has the function of transmitting signals at high speed. The package can be packaged in Pin Grid Array (PGA). Zero Insertion Force (ZIF), Single Edge Contact Connection (SECC), Land Grid Array (LGA), etc.

Optionally, the neural network chip 111 and the second substrate 113 are encapsulated by using a flip chip ball grid array (Flip Chip Ball Grid Array). For a schematic diagram of a specific neural network chip package structure, reference may be made to FIG. 12a. As shown in FIG. 12a, the neural network chip package structure includes a neural network chip 21, a pad 22, a solder ball 23, a second substrate 24, a connection point 25 on the second substrate 24, and a lead 26.

Wherein, the pad 22 is connected to the neural network chip 21, and the solder ball 23 is soldered between the pad 22 and the connection point 25 on the second substrate 24 to connect the neural network chip 21 and the second substrate 24, thereby realizing The package of the neural network chip 21.

The pin 26 is used for connecting with an external circuit of the package structure (for example, the first substrate 13 on the neural network processor board 10), and can realize transmission of external data and internal data, facilitating the neural network chip 21 or the neural network chip 21. The corresponding neural network processor processes the data. The type and number of pins are not limited in this disclosure. Different pin types may be selected according to different packaging technologies, and are arranged according to certain rules.

Optionally, the neural network chip package structure further includes an insulating filler disposed in the gap between the pad 22, the solder ball 23 and the connection point 25 for preventing interference between the solder ball and the solder ball.

The material of the insulating filler may be silicon nitride, silicon oxide or silicon oxynitride; the interference includes electromagnetic interference, inductance interference and the like.

Optionally, the neural network chip package structure further includes a heat dissipation device for dissipating heat of the neural network chip 21 during operation. The heat sink may be a piece of metal with good thermal conductivity, a heat sink or a heat sink, for example, a fan.

For example, as shown in FIG. 12b, the neural network chip package structure 11 includes: a neural network chip 21, a pad 22, a solder ball 23, a second substrate 24, a connection point 25 on the second substrate 24, and a pin 26, Insulating filler 27, thermal grease 28 and metal housing fins 29. Among them, the heat dissipation paste 28 and the metal case heat sink 29 are used to dissipate the heat of the neural network chip 21 during operation.

Optionally, the neural network chip package structure 11 further includes a reinforcing structure, is connected to the pad 22, and is embedded in the solder ball 23 to enhance the connection strength between the solder ball 23 and the pad 22.

The reinforcing structure may be a metal wire structure or a columnar structure, which is not limited herein.

The present disclosure is not limited to the specific form of the first electrical and non-electrical device 12, and the description of the second electrical and non-electrical device 112 may be referred to, that is, the neural network chip package structure 11 may be packaged by soldering, or may be connected. The manner of connecting the second substrate 113 and the first substrate 13 by wire connection or plugging and unplugging facilitates subsequent replacement of the first substrate 13 or the neural network chip package structure 11.

Optionally, the first substrate 13 includes an interface of a memory unit for expanding a storage capacity, for example, a Synchronous Dynamic Random Access Memory (SDRAM), and a Double Rate Rate SDRAM (Double Rate Rate SDRAM). DDR), etc., improve the processing power of the neural network processor by expanding the memory.

The first substrate 13 may further include a Peripheral Component Interconnect-Express (PCI-E or PCIe) interface, a Small Form-factor Pluggable (SFP) interface, and an Ethernet interface. Controller Area Network (CAN) interface, etc., used for data transmission between the package structure and external circuits, which can improve the operation speed and convenience of operation.

The neural network processor is packaged as a neural network chip 111, the neural network chip 111 is packaged as a neural network chip package structure 11, and the neural network chip package structure 11 is packaged as a neural network processor board 10, through an interface on the board ( The slot or ferrule) performs data interaction with an external circuit (for example, a computer motherboard), that is, directly implements the function of the neural network processor by using the neural network processor board 10, and protects the neural network chip 111. Moreover, other modules can be added to the neural network processor board 10, which improves the application range and computational efficiency of the neural network processor.

In one embodiment, the present invention discloses a neural network computing device that includes functional units for performing all or part of the embodiments provided in the method embodiments described above.

In one embodiment, the present invention discloses a chip for performing all or part of the embodiments provided in the method embodiments described above.

In one embodiment, the present invention discloses an electronic device comprising functional units for performing all or part of the embodiments of the method as described above.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cameras, camcorders, projectors, watches, headphones, mobile storage , wearables, vehicles, household appliances, and/or medical equipment.

The vehicle includes an airplane, a ship, and/or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, a rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, a range hood; the medical device includes a nuclear magnetic resonance instrument, B-ultrasound and / or electrocardiograph.

The specific embodiments of the present invention have been described in detail with reference to the specific embodiments of the present disclosure. It is understood that the above description is only the specific embodiment of the disclosure, and is not intended to limit the disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure are intended to be included within the scope of the disclosure.

Claims (22)

An integrated circuit chip device, comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit comprising a first mapping circuit, at least one of the plurality of basic processing circuits The circuit includes a second mapping circuit, each of the first mapping circuit and the second mapping circuit for performing compression processing of each data in the neural network operation; The main processing circuit is configured to perform each successive operation in the neural network operation and transmit data with the basic processing circuit; The plurality of basic processing circuits are configured to perform operations in the neural network in parallel according to data transmitted by the main processing circuit, and transmit the operation results to the main processing circuit. The integrated circuit chip device according to claim 1, wherein the integrated circuit chip device further comprises: a branch processing circuit, the branch processing circuit being disposed between the main processing circuit and the at least one basic processing circuit; The branch processing circuit is configured to forward data between the main processing circuit and the at least one basic processing circuit; The branch processing circuit includes a plurality of branch processing circuits, the main processing circuit and the plurality of branch processing circuits are respectively connected, and each branch processing circuit is connected to at least one basic processing circuit. The integrated circuit chip device according to claim 1, wherein The main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; and start the first mapping circuit Processing the data block and the broadcast data block to process the processed distribution data block, the identification data block associated with the distribution data block, the processed broadcast data block, and the identification data block associated with the broadcast data block; The processed distribution data block and the identification data block associated with the distribution data block are split to obtain a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; the plurality of basic data blocks and the An identification data block associated with each of the plurality of basic data blocks is distributed to a basic processing circuit connected thereto, and the broadcast data block and the identification data block associated with the broadcast data block are broadcasted to a basic processing circuit connected thereto; The basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the basic data block and the identification data block associated with the broadcast data block; and identify the data block according to the connection Processing the basic data block and the broadcast data block, performing an inner product operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and transmitting the operation result to the main processing circuit; The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result corresponding to the operation instruction. The integrated circuit chip device according to claim 1, wherein The main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; and start the first mapping circuit The distribution data block is processed to obtain the processed distribution data block and the identification data block associated with the distribution data block, or the first mapping circuit is activated to store the distribution data block according to the identifier data block associated with the pre-stored distribution data block. Processing the processed distribution data block; and separating the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identifiers associated with each of the plurality of basic data blocks a data block; distributing the plurality of basic data blocks and the identification data blocks respectively associated with the plurality of basic data blocks to a base processing circuit connected thereto; broadcasting the broadcast data block to a base processing circuit connected thereto; The basic processing circuit is configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, and execute the processed broadcast data block and the basic data block The product operation obtains an operation result, and the operation result is sent to the main processing circuit; The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction. The integrated circuit chip device according to claim 1, wherein The main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; and start the first mapping circuit Processing the broadcast data block to obtain the processed broadcast data block and the identification data block associated with the broadcast data block, or initiating the first mapping circuit to perform the broadcast data according to the pre-stored identification data block associated with the broadcast data block. Processing the block to obtain a processed broadcast data block; splitting the distributed data block to obtain a plurality of basic data blocks; distributing the plurality of basic data to a base processing circuit connected thereto; and transmitting the processed broadcast The data block and the identification data block associated with the broadcast data block are broadcast to a basic processing circuit connected thereto; The basic processing circuit is configured to start, by the second mapping circuit, processing the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and processing the processed basic data. Performing an inner product operation on the block and the processed broadcast data block to obtain an operation result, and transmitting the operation result to the main processing circuit; The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result corresponding to the operation instruction. The integrated circuit chip arrangement according to any of claims 3-5, wherein the data block to be calculated comprises at least one weight, and/or at least one input neuron. The integrated circuit chip device according to claim 6, wherein when said identification data block is represented by a direct index, said identification data block is a matrix data block composed of 0 and 1, wherein 0 represents said The weight or the absolute value of the input neuron is less than or equal to the first threshold, and 1 represents the weight or the absolute value of the input neuron is greater than the first threshold. The integrated circuit chip device according to any one of claims 3 to 5, wherein The basic processing circuit is configured to perform inner product processing on the basic data block and the broadcast data block to obtain an inner product result, accumulate the inner product processing result to obtain an operation result, and send the operation result to the main Processing circuit The main processing circuit is configured to, when the result of the operation is the result of the inner product processing, accumulate the operation result to obtain an accumulation result, and arrange the accumulation result to obtain the data block to be calculated and the operation instruction unit. Corresponding instruction result. The integrated circuit chip device according to any one of claims 3 to 5, wherein The main processing circuit is configured to split the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and the plurality of partial broadcast data blocks respectively associated with each other. Identifying a data block; broadcasting the plurality of partial broadcast data blocks and the identification data blocks respectively associated with the plurality of partial broadcast data blocks to the basic processing circuit by one or more times; the plurality of partial broadcast data block combinations Forming the processed broadcast data block; The basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the partial broadcast data block and the identification data block associated with the basic data block; according to the connection identifier Processing, by the data block, the partial broadcast data block and the basic data block to obtain the processed broadcast data block and the processed basic data block; and executing the processed broadcast data block and the processed basic data block Product operation Or the basic processing circuit is configured to start, by the second mapping circuit, the basic data block to be processed according to the identifier data block associated with the partial broadcast data block, and the processed basic data block is processed. The subsequent basic data and the partial broadcast data block perform an inner product operation. The integrated circuit chip device according to any one of claims 3 to 5, wherein The basic processing circuit is specifically configured to perform inner product processing on the partial broadcast data block and the basic data block to obtain an inner product processing result, and accumulate the inner product processing result to obtain a partial operation result, and the partial operation result The result is sent to the main processing circuit; or The basic processing circuit is specifically configured to multiplex the partial broadcast data block n times to perform an inner product operation of the partial broadcast data block and the n basic data blocks to obtain n partial processing results, and accumulate the n partial processing results respectively. Then, n partial operation results are obtained, and the n partial operation results are sent to the main processing circuit, and the n is an integer greater than or equal to 2. The integrated circuit chip device according to claim 3, wherein said operation instruction is a convolution instruction. The main processing circuit is configured to acquire an input data block, a convolution kernel data block, and a convolution instruction, and divide the input data block into a broadcast data block according to the convolution instruction, and divide the convolution kernel data block into Generating a data block; determining, according to the operation control of the convolution instruction, starting the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and And the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the convolution instruction; The plurality of basic processing circuits are configured to determine, according to the operation control of the convolution instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. Calculating an operation result, and transmitting the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing circuit determined by the basic processing circuit a transmitted data block, the second data block being associated with the processed first data block; The main processing circuit is configured to process the operation result to obtain an instruction result of the convolution instruction. The integrated circuit chip device according to claim 3, wherein said operation instruction is a multiplication instruction. The main processing circuit is configured to acquire an input data block, a weight data block, and a multiplication instruction, and divide the input data block into a data block according to the multiplication instruction, and divide the weight data block into a broadcast data block. And determining, according to the operation control of the multiplication instruction, starting the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast a data block; transmitting, according to the multiplication instruction, the processed first data block to at least one of the basic processing circuits connected to the main processing circuit; The plurality of basic processing circuits are configured to determine, according to the operation control of the multiplication instruction, whether to start the second mapping circuit to process the second data block, and perform the operation in the neural network in parallel according to the processed second data block. Obtaining an operation result, and transmitting the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is sent by the main processing circuit determined by the basic processing circuit Data block, the second data block is associated with the processed first data block; The main processing circuit is configured to process the operation result to obtain an instruction result of the multiplication instruction. The integrated circuit chip device according to claim 3, wherein said operation instruction is a forward operation instruction. The main processing circuit is configured to receive a forward operation instruction, and parse the forward operation instruction to obtain a first operation instruction included in an ith layer of the forward operation instruction in the forward operation of the neural network An input data block and a weight data block required by the first operation instruction, wherein the range of i is an integer greater than or equal to 1, and less than or equal to n, and if the i is greater than or equal to 2, the input data block is Output data block of layer i-1; The main processing circuit is configured to divide the input data block into a broadcast data block according to the first operation instruction, and divide the convolution kernel data block into a data block; according to the first forward instruction The operation control determines to start the first mapping circuit to process the first data block to obtain the processed first data block; the first data block includes the distribution data block and/or the broadcast data block; The product instruction sends the processed first data block to at least one of the basic processing circuits connected to the main processing circuit; The plurality of basic processing circuits are configured to determine, according to the operation control of the first operation instruction, whether to start the second mapping circuit to process the second data block, and execute the neural network in a parallel manner according to the processed second data block. The operation obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is the receiving the main processing determined by the basic processing circuit a data block sent by the circuit, the second data block being associated with the processed first data block; The main processing circuit is configured to process the operation result to obtain an instruction result of the first operation instruction, and complete an operation of the first operation instruction included in the ith layer. The integrated circuit chip device according to claim 3, wherein said device is configured to perform training of a neural network, said neural network comprising n layers, said n value range being an integer greater than or equal to 2, said The operation instruction is a training instruction, The integrated circuit chip device is configured to receive a training instruction, determine first layer input data and first layer weight group data according to the training instruction, and perform a neural network on the first layer input data and the first layer weight group data. The n-th layer forward operation obtains the nth output result of the forward operation; The main processing circuit is further configured to obtain an nth output result gradient according to the nth output result, and obtain an nth reverse operation nth inverse operation instruction and the nth inverse operation according to the training instruction The nth layer input data and the nth layer weight group data required by the instruction; dividing the nth output result gradient, the nth layer input data, and the nth layer weight group data according to the nth reverse operation instruction a broadcast data block and a distribution data block; determining, according to the operation control of the nth reverse operation instruction, starting the first mapping circuit to process the first data block, and obtaining the processed first data block; the first data block Include the distribution data block and/or the broadcast data block; transmitting the processed first data block to at least one of the basic processing circuits connected to the main processing circuit according to the nth reverse operation instruction Processing circuit The plurality of basic processing circuits are configured to determine, according to the operation control of the nth reverse operation instruction, whether to start the second mapping circuit to process the second data block, and execute the nerve in parallel according to the processed second data block An operation in the network obtains an operation result, and transmits the operation result to the main processing circuit through a basic processing circuit connected to the main processing circuit; the second data block is determined by the basic processing circuit to receive the a data block sent by the main processing circuit, the second data block being associated with the processed first data block; The main processing circuit is further configured to process the operation result to obtain an nth layer weight group gradient and an nth layer input data gradient, and apply the nth layer weight group gradient to the nth layer weight group data. Update The integrated circuit chip device is further configured to perform n-1 layer inverse operation on the nth layer input data gradient as the n-1th output gradient of the n-1th layer to obtain an n-1 layer weight group gradient, and apply The n-1 layer weight group gradient updates the weight group data of the corresponding layer, and the weight group data includes at least two weights. The integrated circuit chip device according to any one of claims 11 to 14, wherein when the first data block includes a distribution data block and a broadcast data block, The main processing circuit is specifically configured to start, by the first mapping circuit, the distributed data block processed by the distribution data block and the broadcast data block, and the identification data block associated with the distribution data block, and after processing a broadcast data block and an identification data block associated with the broadcast data block; splitting the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and the plurality of basic data blocks Corresponding identification data blocks; distributing the plurality of basic data blocks and the identification data blocks respectively associated with the plurality of basic data blocks to a base processing circuit connected thereto, associating the broadcast data block and the broadcast data block The identification data block is broadcast to the basic processing circuit connected thereto; The basic processing circuit is configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the basic data block and the identification data block associated with the broadcast data block; and identify the data block according to the connection Processing the basic data block and the broadcast data block, performing an operation indicated by the operation instruction on the processed basic data block and the processed broadcast data block to obtain an operation result, and transmitting the operation result to the The main processing circuit. The integrated circuit chip device according to any one of claims 11 to 14, wherein when the first data block includes a distribution data block, The main processing circuit is specifically configured to start, by the first mapping circuit, processing the distributed data block to obtain a processed distribution data block and an identification data block associated with the distribution data block, or start the first mapping circuit. And processing the distributed data block according to the pre-stored identifier data block associated with the distribution data block to obtain the processed distribution data block; and splitting the processed distribution data block and the identification data block associated with the distribution data block Obtaining a plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks; distributing the plurality of basic data blocks and the identification data blocks associated with each of the plurality of basic data blocks to a basic processing connected thereto a circuit; broadcasting the broadcast data block to a base processing circuit connected thereto; The basic processing circuit is configured to start the second mapping circuit to process the broadcast data block according to the identifier data block associated with the basic data block, and execute the processed broadcast data block and the basic data block The operation indicated by the operation command obtains the operation result, and the operation result is transmitted to the main processing circuit. The integrated circuit chip device according to any one of claims 11 to 14, wherein when the first data block includes a broadcast data block, The main processing circuit is configured to: start, by the first mapping circuit, processing the broadcast data block to obtain a processed broadcast data block and an identifier data block associated with the broadcast data block, or start the first mapping circuit Processing the broadcast data block according to the pre-stored identifier data block associated with the broadcast data block to obtain a processed broadcast data block; splitting the distribution data block to obtain a plurality of basic data blocks; The basic data is distributed to the basic processing circuit connected thereto; the processed broadcast data block and the identification data block associated with the broadcast data block are broadcasted to the basic processing circuit connected thereto; The basic processing circuit is configured to start, by the second mapping circuit, processing the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and processing the processed basic data. The block and the processed broadcast data block perform an operation indicated by the operation instruction to obtain an operation result, and the operation result is transmitted to the main processing circuit. A neural network processor board, characterized in that: the neural network processor board comprises: a neural network chip package structure, a first electrical and non-electrical connection device and a first substrate; the neural network chip package structure comprises a neural network chip, a second electrical and non-electrical connection device and a second substrate, the second substrate carrying the neural network chip, the second substrate passing through the second electrical and non-electrical connection device and the nerve Network chip connection; The neural network chip comprises the integrated circuit chip device of any of claims 1-17. The neural network processor board according to claim 18, wherein the neural network chip package structure further comprises: a heat sink. The neural network processor card according to claim 18, wherein the package structure of the neural network chip package structure is any one of the following packages: Flip-chip ball grid array package, thin quad flat package, quad flat package with heat sink, leadless quad flat package, small pitch quad flat package. A chip, characterized in that the chip integrates the device according to any one of claims 1-17. A smart device, characterized in that the smart device comprises the chip of claim 22.

Images