CN109961136A - Integrated circuit chip devices and related products - Google Patents

Abstract

The present disclosure provides an integrated circuit chip device and related products, the integrated circuit chip device comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit or at least one of the plurality of basic processing circuits comprises: a data type arithmetic circuit; the data type arithmetic circuit is used for executing conversion between floating point type data and fixed point type data; the plurality of base processing circuits are distributed in an array; each basic processing circuit is connected with other adjacent basic processing circuits, and the main processing circuit is connected with the n basic processing circuits of the 1 st row, the n basic processing circuits of the m th row and the m basic processing circuits of the 1 st column. The technical scheme provided by the disclosure has the advantages of small calculation amount and low power consumption.

Description

Integrated circuit chip devices and related products

technical field

The present disclosure relates to the field of neural networks, and in particular, to an integrated circuit chip device and related products.

Background technique

Artificial Neural Network (ANN) has been a research hotspot in the field of artificial intelligence since the 1980s. It abstracts the human brain neuron network from the perspective of information processing, establishes a certain simple model, and forms different networks according to different connection methods. In engineering and academia, it is often simply referred to as neural network or neural-like network. Neural network is an operation model, which is composed of a large number of nodes (or neurons) connected with each other. The operation of the existing neural network is based on a CPU (Central Processing Unit, central processing unit) or a GPU (English: Graphics Processing Unit, graphics processing unit) to realize the operation of the neural network, which requires a large amount of calculation and high power consumption.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide an integrated circuit chip device and related products, which can improve the processing speed and efficiency of the computing device.

In a first aspect, an integrated circuit chip device is provided, the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; at least one of the main processing circuit or the plurality of basic processing circuits includes: a data type operation a circuit; the data type operation circuit is used to perform conversion between floating point type data and fixed point type data;

The plurality of basic processing circuits are distributed in an array; each basic processing circuit is connected to other adjacent basic processing circuits, and the main processing circuit is connected to the n basic processing circuits in the first row and the n basic processing circuits in the mth row. circuit and m basic processing circuits in column 1;

The main processing circuit is used to perform each successive operation in the neural network operation and transmit data with the basic processing circuit connected to it;

The plurality of basic processing circuits are used to perform operations in the neural network in parallel according to the transmitted data, and transmit the operation results to the main processing circuit through the basic processing circuits connected to the main processing circuit.

A second aspect provides a neural network computing device, the neural network computing device comprising one or more integrated circuit chip devices provided in the first aspect.

A third aspect provides a combined processing device, the combined processing device comprising: the neural network computing device, the universal interconnection interface, and the universal processing device provided in the second aspect;

The neural network computing device is connected with the general processing device through the general interconnection interface.

A fourth aspect provides a chip that integrates the device of the first aspect, the device of the second aspect, or the device of the third aspect.

In a fifth aspect, an electronic device is provided, and the electronic device includes the chip of the fourth aspect.

In a sixth aspect, a method for computing a neural network is provided, the method is applied in an integrated circuit chip device, and the integrated circuit chip device includes: the integrated circuit chip device described in the first aspect, and the integrated circuit chip device uses for performing neural network operations.

It can be seen that, through the embodiments of the present disclosure, a data conversion operation circuit is provided to perform operations after converting the types of data blocks, which saves transmission resources and computing resources, so it has the advantages of low power consumption and small calculation amount.

Description of drawings

FIG. 1a is a schematic structural diagram of an integrated circuit chip device.

FIG. 1b is a schematic structural diagram of another integrated circuit chip device.

FIG. 1c is a schematic structural diagram of a basic processing circuit.

FIG. 1d is a schematic structural diagram of a main processing circuit.

FIG. 1e is a schematic structural diagram of a fixed-point data type.

FIG. 2a is a schematic diagram of a method of using a basic processing circuit.

Figure 2b is a schematic diagram of a main processing circuit transmitting data.

Figure 2c is a schematic diagram of a matrix multiplied by a vector.

FIG. 2d is a schematic structural diagram of an integrated circuit chip device.

FIG. 2e is a schematic structural diagram of another integrated circuit chip device.

Figure 2f is a schematic diagram of matrix-by-matrix.

Figure 3a is a schematic diagram of convolution input data.

Figure 3b is a schematic diagram of the convolution kernel.

FIG. 3c is a schematic diagram of an operation window of a three-dimensional data block of input data.

FIG. 3d is a schematic diagram of another operation window of a three-dimensional data block of input data.

Figure 3e is a schematic diagram of another operation window of a three-dimensional data block of input data.

Figure 4a is a schematic diagram of the forward operation of the neural network.

Figure 4b is a schematic diagram of the reverse operation of the neural network.

FIG. 4c also discloses a schematic structural diagram of a combined processing device for the present disclosure.

FIG. 4d also discloses another structural schematic diagram of a combined processing device for the present disclosure.

5a is a schematic structural diagram of a neural network processor board according to an embodiment of the disclosure;

5b is a schematic structural diagram of a neural network chip packaging structure provided by an embodiment of the disclosure;

FIG. 5c is a schematic structural diagram of a neural network chip provided by an embodiment of the disclosure;

6 is a schematic diagram of a neural network chip packaging structure provided by an embodiment of the present disclosure;

FIG. 6a is a schematic diagram of another neural network chip packaging structure provided by an embodiment of the present disclosure.

Detailed ways

In order to make those skilled in the art better understand the disclosed solutions, the technical solutions in the disclosed embodiments will be clearly and completely described below with reference to the accompanying drawings in the disclosed embodiments. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present disclosure.

In the device provided in the first aspect, the main processing circuit is configured to acquire a data block to be calculated and an operation instruction, and convert the data block to be calculated into a fixed-point type data block through the data type operation circuit, According to the operation instruction, the fixed-point type data block to be calculated is divided into a distribution data block and a broadcast data block; the distribution data block is split to obtain a plurality of basic data blocks, and the plurality of basic data blocks are divided into distribute to the basic processing circuit connected to it, broadcast the broadcast data block to the basic processing circuit connected to it;

the basic processing circuit, configured to perform an inner product operation on the basic data block and the broadcast data block in a fixed-point data type to obtain an operation result, and send the operation result to the main processing circuit;

Or forward the basic data block and the broadcast data block to other basic processing circuits to perform an inner product operation with a fixed-point data type to obtain an operation result, and send the operation result to the main processing circuit;

The main processing circuit is configured to convert the operation result into floating point type data through the data type operation circuit, and process the floating point type data to obtain the data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided in the first aspect, the main processing circuit is specifically configured to send the broadcast data block to the basic processing circuit connected thereto through one broadcast.

In the apparatus provided in the first aspect, the basic processing circuit is specifically configured to perform inner product processing on the basic data block and the broadcast data block in a fixed-point data type to obtain an inner product processing result, and process the inner product The results are accumulated to obtain an operation result, and the operation result is sent to the main processing circuit.

In the device provided in the first aspect, the main processing circuit is configured to, if the operation result is a result of inner product processing, accumulate the operation results to obtain an accumulation result, and arrange the accumulation results to obtain the accumulation result. The data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided in the first aspect, the main processing circuit is specifically configured to divide the broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the basic processing by multiple times. a circuit; the plurality of partial broadcast data blocks are combined to form the broadcast data block.

In the device provided in the first aspect, the basic processing circuit is specifically configured to perform an inner product processing on the part of the broadcast data block and the basic data block in a fixed-point data type to obtain an inner product processing result, The inner product processing results are accumulated to obtain partial operation results, and the partial operation results are sent to the main processing circuit.

In the device provided in the first aspect, the basic processing circuit is specifically configured to multiplex the part of the broadcast data block n times to perform an inner product operation of the part of the broadcast data block and the n basic data blocks to obtain n partial processing results, After accumulating the n partial processing results respectively, n partial operation results are obtained, and the n partial operation results are sent to the main processing circuit, where n is an integer greater than or equal to 2.

In the device provided in the first aspect, the main processing circuit includes: a main register or a main on-chip cache circuit;

The basic processing circuit includes: a basic register or a basic on-chip cache circuit.

In the device provided in the first aspect, the main processing circuit includes: a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transpose circuit, a direct memory access circuit, a data type operation circuit or a data rearrangement circuit one or any combination.

In the device provided in the first aspect, the main processing circuit is configured to obtain a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; The distribution data block is split to obtain a plurality of basic data blocks, the plurality of basic data blocks are distributed to the basic processing circuit connected to it, and the broadcast data block is broadcast to the basic processing circuit connected to it;

The basic processing circuit is configured to convert the basic data block and the broadcast data block into a fixed-point type data block, perform an inner product operation according to the fixed-point type data block to obtain an operation result, and convert the operation result into a floating-point type. After the point data is sent to the main processing circuit;

Or convert the basic data block and the broadcast data block into fixed-point data blocks, forward the fixed-point data blocks to other basic processing circuits to perform an inner product operation to obtain an operation result, and convert the operation result into a floating-point number The data is then sent to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided in the first aspect, the data is one or any combination of vectors, matrices, three-dimensional data blocks, four-dimensional data blocks, and n-dimensional data blocks.

In the device provided in the first aspect, if the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is a broadcast data block, and the multiplicand data block is a distribution data block;

Or if the operation instruction is a convolution instruction, the main processing circuit determines that the convolution input data block is a broadcast data block, and the convolution kernel is a distribution data block.

In the method provided in the sixth aspect, the operation of the neural network includes: one of convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, paranoid operation, fully connected operation, GEMM operation, GEMV operation, and activation operation or any combination.

Referring to FIG. 1a, FIG. 1a provides an integrated circuit chip device for the present disclosure, the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits, and the plurality of basic processing circuits are arranged in an array (m*n array), where the value range of m and n is an integer greater than or equal to 1, and at least one of m and n has a value greater than or equal to 2. For a plurality of basic processing circuits distributed in an m*n array, each basic processing circuit is connected to an adjacent basic processing circuit, and the main processing circuit is connected to k basic processing circuits of the plurality of basic processing circuits, the k basic processing circuits The processing circuits may be: n basic processing circuits in the first row, n basic processing circuits in the mth row, and m basic processing circuits in the first column. In the integrated circuit chip device shown in FIG. 1a, the main processing circuit and/or a plurality of basic processing circuits may include a data type conversion operation circuit, and some of the specific basic processing circuits may include a data type conversion circuit, For example, in an optional technical solution, the k basic processing circuits may be configured with data type conversion circuits, so that the n basic processing circuits may be respectively responsible for performing the data type conversion step on the data of the m basic processing circuits in this column. This setting can improve computing efficiency and reduce power consumption, because for the n basic processing circuits in the first row, since they first receive the data sent by the main processing circuit, the received data is converted into fixed-point type. The data can reduce the amount of calculation of the subsequent basic processing circuits and the amount of data transmission with the subsequent basic processing circuits. Similarly, for the m basic processing circuits in the first column, the configuration data type conversion circuit also has the advantages of small calculation amount and low power consumption. advantage. In addition, according to this structure, the main processing circuit can adopt a dynamic data transmission strategy. For example, the main processing circuit broadcasts data to the m basic processing circuits in the first column, and the main processing circuit sends the distribution to the n basic processing circuits in the first row. The advantage of this is that different data are transmitted to the basic processing circuit through different data input ports, so that the basic processing circuit can not distinguish what kind of data the received data is, and only needs to determine which receiving port the data is received from. You can know what kind of data it belongs to.

The main processing circuit is used to perform each continuous operation in the neural network operation and transmit data with the basic processing circuit connected to it; the above-mentioned continuous operation is not limited to: accumulation operation, ALU operation, activation operation, etc. .

The plurality of basic processing circuits are used to perform operations in the neural network in parallel according to the transmitted data, and transmit the operation results to the main processing circuit through the basic processing circuits connected to the main processing circuit. The operations in the neural network performed in the above-mentioned parallel manner include, but are not limited to, inner product operations, matrix or vector multiplication operations, and the like.

The main processing circuit may include: a data transmission circuit, a data reception circuit or an interface. The data transmission circuit may integrate a data distribution circuit and a data broadcast circuit. Of course, in practical applications, the data distribution circuit and the data broadcast circuit can also be provided separately. For broadcast data, that is the data that needs to be sent to each underlying processing circuit. For distribution data, that is, data that needs to be selectively sent to some basic processing circuits, specifically, such as convolution operation, the convolution input data of convolution operation needs to be sent to all basic processing circuits, all of which are broadcast data, volume The product kernel needs to be selectively sent to some basic data blocks, so the convolution kernel is to distribute data. The manner in which the distribution data is specifically selected and sent to the basic processing circuit may be specifically determined by the main processing circuit according to the load and other distribution manners. For the broadcast transmission method, the broadcast data is transmitted to each basic processing circuit in a broadcast form. (In practical applications, broadcast data is sent to each basic processing circuit by one broadcast, and broadcast data can also be sent to each basic processing circuit by multiple broadcasts. The specific embodiments of the present disclosure do not limit the above The number of broadcasts), for the distribution transmission mode, the distribution data is selectively sent to some basic processing circuits.

The main processing circuit (as shown in Figure 1d) may include a register and/or an on-chip buffer circuit, and the main processing circuit may also include: a control circuit, a vector operator circuit, an ALU (arithmetic and logic unit, arithmetic logic unit) circuit, an accumulation circuit, DMA (Direct Memory Access, Direct Memory Access) circuit, of course, in practical applications, the above-mentioned main processing circuit can also be added, conversion circuit (such as matrix transposition circuit), data rearrangement circuit or activation circuit, etc. and other circuits.

Each basic processing circuit may include a basic register and/or a basic on-chip buffer circuit; each basic processing circuit may further include one or any combination of an inner product operator circuit, a vector operator circuit, an accumulator circuit, and the like. The above inner product operator circuit, vector operator circuit and accumulator circuit may all be integrated circuits, and the above inner product operator circuit, vector operator circuit and accumulator circuit may also be separately provided circuits.

Optionally, the accumulator circuit of the n basic processing circuits in the mth row can perform the accumulation operation of the inner product operation, because for the basic processing circuit in the mth row, it can receive the product of all the basic processing circuits in this column. As a result, the accumulation operation of the inner product operation is performed by the n basic processing circuits in the mth row to perform the accumulation operation of the inner product operation, which can effectively allocate computing resources and has the advantage of saving power consumption. This technical solution is especially applicable when the number of m is large.

For data type conversion, the main processing circuit can allocate the execution circuit. Specifically, the execution circuit can be allocated in an explicit or implicit way. For the display mode, the main processing circuit can be configured with a special instruction or instruction. When the basic processing When the circuit receives the special instruction or instruction, it determines to perform data type conversion. For example, when the basic processing circuit does not receive the special instruction or instruction, it determines not to perform data type conversion. For another example, it can be performed in an implicit manner, for example, when the basic processing circuit receives data of a floating-point type and determines that an inner product operation needs to be performed, it converts the data type to data of a fixed-point type. For the display configuration method, a special instruction or instruction can configure a decrementing sequence. Each time the decrementing sequence passes through a basic processing circuit, the value decreases by 1, and the basic processing circuit reads the value of the decrementing sequence. If the value is greater than zero, execute Data type conversion, if the value is equal to or less than zero, no data type conversion is performed. This setting is configured according to the basic processing circuits allocated by the array. For example, for the m basic processing circuits in the i-th column, the main processing circuit needs the first 5 basic processing circuits to perform data type conversion, and the main processing circuit sends a Special instruction, the special instruction contains a decrement sequence, the initial value of the decrement sequence can be 5, then each time it passes through a basic processing circuit, the value of the decrement sequence is decremented by 1, and when it reaches the fifth basic processing circuit, the value of the decrement sequence is reduced by 1. The value is 1. When the 6th basic processing circuit is reached, the decrement sequence is 0. At this time, the 6th basic processing circuit will not perform the data type conversion. This method allows the main processing circuit to dynamically configure the data type conversion. Execution subject and execution times.

An embodiment of the present disclosure provides an integrated circuit chip device, including a main processing circuit (also referred to as a main unit) and a plurality of basic processing circuits (also referred to as basic units); the structure of the embodiment is shown in FIG. 1b ; Wherein, the dashed box is the internal structure of the neural network computing device; the arrows filled in gray represent the data transmission paths between the main processing circuit and the basic processing circuit array, and the hollow arrows represent the basic processing circuits in the basic processing circuit array ( A data transmission path between adjacent basic processing circuits). The length, width and length of the basic processing circuit array may be different, that is, the values of m and n may be different, and of course they may be the same. The present disclosure does not limit the specific values of the above values.

The circuit structure of the basic processing circuit is shown in Figure 1c; the dashed box in the figure represents the boundary of the basic processing circuit, and the thick arrows crossing the dashed box represent the data input and output channels (pointing to the dashed box is the input channel, and pointing out that the dashed box is the output channel. ); the rectangular box in the dashed box represents the storage unit circuit (register and/or on-chip cache), including input data 1, input data 2, multiplication or inner product result, and accumulated data; the diamond box represents the operator circuit, including multiplication or inner Product operator, adder.

In this embodiment, the neural network computing device includes a main processing circuit and 16 basic processing circuits (the 16 basic processing circuits are only for illustration, and other values may be used in practical applications);

In this embodiment, the basic processing circuit has two data input interfaces and two data output interfaces; in the subsequent description of this example, the horizontal input interface (the horizontal arrow pointing to the unit in FIG. The vertical input interface (the vertical arrow pointing to this unit in Figure 1b) is called input 1; each horizontal data output interface (the horizontal arrow pointing to this unit in Figure 1b) is called output 0, the vertical The data output interface (the vertical arrow pointing from this unit in Figure 1b) is called output 1.

The data input interface and data output interface of each basic processing circuit can be connected to different units, including the main processing circuit and other basic processing circuits;

In this example, the input 0 of the four basic processing circuits 0, 4, 8, and 12 (numbered as shown in Figure 1b) is connected to the data output interface of the main processing circuit;

In this example, the input 1 of the basic processing circuits 0, 1, 2, and 3 is connected to the data output interface of the main processing circuit;

In this example, the outputs 1 of the four basic processing circuits 12, 13, 14, and 15 are connected to the data input interface of the main processing circuit;

In this example, the connection of the output interface of the basic processing circuit with the input interface of other basic processing circuits is shown in Figure 1b, and will not be listed one by one;

Specifically, the output interface S1 of the S unit is connected to the input interface P1 of the P unit, indicating that the P unit can receive the data sent by the S unit to its S1 interface from its P1 interface.

This embodiment includes a main processing circuit, the main processing circuit is connected with an external device (that is, an input interface also has an output interface), and a part of the data output interface of the main processing circuit is connected with a part of the data input interface of the basic processing circuit; the main processing circuit A part of the data input interface is connected with a part of the data output interface of the basic processing circuit.

Method of using integrated circuit chip device

The data involved in the use method provided by the present disclosure may be data of any data type, for example, data represented by floating-point numbers of any bit width or data represented by fixed-point numbers of any bit width.

A schematic diagram of the structure of the fixed-point type data is shown in Figure 1e, which is an expression method of fixed-point type data. For a computing system, the storage bits of one floating-point data are 32 bits, while Data, especially the floating-point data as shown in Figure 1e is used for data representation, and the storage bits of one fixed-point data can be less than 16Bit, so for this conversion, it can greatly reduce the calculator In addition, for the calculator, the data storage space with fewer bits is also smaller, that is, the storage overhead will be smaller, and the amount of calculation will be reduced, that is, the calculation overhead will be reduced, so the calculation can be reduced. Overhead and storage overhead, but the conversion of data types also requires some overhead, hereinafter referred to as conversion overhead. For data with a large amount of calculation and a large amount of data storage, the conversion overhead is relative to the subsequent calculation overhead, storage overhead and transmission overhead. The overhead is almost negligible, so for the data with a large amount of calculation and a large amount of data storage, the present disclosure adopts the technical solution of converting the data type into fixed-point type data. On the contrary, for the data with a small amount of calculation and a small amount of data storage At this time, the calculation overhead, storage overhead and transmission overhead are relatively small. If fixed-point data is used at this time, since the precision of fixed-point data will be slightly lower than that of floating-point data, it is necessary to ensure that the calculation amount is small under the premise of small amount of calculation. Therefore, the data of fixed-point type is converted into floating-point data here, that is, the purpose of improving the accuracy of calculation by adding less overhead.

Operations that need to be done in the basic processing circuit can be performed using the following methods:

The main processing circuit first converts the type of data and then transmits it to the basic processing circuit for operation (for example, the main processing circuit can convert floating-point numbers into fixed-point numbers with a lower bit width and then transmit them to the basic processing circuit, which has the advantage of reducing transmission. The bit width of the data reduces the total number of bits transmitted, and the basic processing circuit performs the position-wide fixed-point operation with higher efficiency and lower power consumption)

The basic processing circuit can convert the data type and then perform the calculation after receiving the data (for example, the basic processing circuit receives the floating-point number transmitted from the main processing circuit, and then converts it into a fixed-point number for operation, which improves the operation efficiency and reduces the power consumption) .

After the basic processing circuit calculates the result, it can first perform data type conversion and then transmit it to the main processing circuit (for example, the floating-point operation result calculated by the basic processing circuit can be converted into a fixed-point number with low bit width first and then transmitted to the main processing circuit, The advantage is that the data bit width of the transmission process is reduced, the efficiency is higher, and the power consumption is saved).

How to use the basic processing circuit (as shown in Figure 2a);

The main processing circuit receives input data to be calculated from outside the device;

Optionally, the main processing circuit uses various operation circuits, vector operation circuits, inner product operator circuits, accumulator circuits, etc. of this unit to perform operation processing on the data;

The main processing circuit sends data to the basic processing circuit array (the set of all basic processing circuits is called the basic processing circuit array) through the data output interface (as shown in Figure 2b);

The method of sending data here may be to directly send data to a part of the basic processing circuit, that is, the method of broadcasting multiple times;

The method of sending data here can send different data to different basic processing circuits respectively, that is, the distribution method;

The basic processing circuit array calculates the data;

The basic processing circuit performs operations after receiving the input data;

Optionally, after receiving the data, the basic processing circuit transmits the data from the data output interface of the unit; (transmits it to other basic processing circuits that do not directly receive data from the main processing circuit.)

Optionally, the basic processing circuit transmits the operation result from the data output interface; (intermediate calculation result or final calculation result)

The main processing circuit receives the output data returned from the basic processing circuit array;

Optionally, the main processing circuit continues to process (eg accumulate or activate) data received from the base processing circuit array;

After the main processing circuit finishes processing, it transmits the processing result from the data output interface to the outside of the device.

using the circuit arrangement to perform a matrix multiplication vector operation;

(A matrix multiplication by a vector can be performed by performing an inner product operation on each row in the matrix and a vector, and arranging the results into a vector in the order of the corresponding rows.)

The following describes an operation for calculating the multiplication of a matrix S of size M rows and L columns and a vector P of length L, as shown in Figure 2c below.

This method uses all or a part of the basic processing circuits of the neural network computing device, assuming that K basic processing circuits are used;

the main processing circuit sends data in some or all of the rows of the matrix S to each of the k basic processing circuits;

In an optional solution, the control circuit of the main processing circuit sends the data of a certain row in the matrix S a number or a part of the number to a basic processing circuit at a time; (for example, for sending a number at a time, it can be For a certain basic processing circuit, the first number of the third line is sent for the first time, the second number of the third line of data is sent for the second time, the third number of the third line is sent for the third time..., or For each time a part of the number is sent, the first two numbers (ie, the 1st and 2nd numbers) of the third line are sent for the first time, the third and fourth numbers of the third line are sent for the second time, and the third line is sent for the third time. 5th and 6th numbers... ;)

In an optional solution, the control circuit of the main processing circuit sends the data of certain rows in the matrix S each time a number or a part of the number to a certain basic processing circuit; (for example, for a certain basic processing circuit, The 1st time sends the 1st number of each line of 3,4,5 lines, the 2nd time sends the 2nd number of each line of 3,4,5 lines, the 3rd time sends the 3rd,4,5th line of each line The 3rd number of the line..., or the first two numbers of the 3rd, 4th, and 5th lines of each line are sent, and the 3rd and 4th numbers of each line of the 3rd, 4th, and 5th lines are sent for the second time. Send the 3rd, 4th, 5th row 5th and 6th numbers each three times....)

The control circuit of the main processing circuit sends the data in the vector P to the 0th basic processing circuit successively;

After the 0th basic processing circuit receives the data of the vector P, it sends the data to the next basic processing circuit connected to it, that is, basic processing circuit 1;

Specifically, some basic processing circuits cannot directly obtain all the data required for calculation from the main processing circuit. For example, the basic processing circuit 1 in Fig. 2d has only one data input interface connected to the main processing circuit, so it can only directly obtain data from the main processing circuit. The main processing circuit obtains the data of the matrix S, and the data of the vector P needs to rely on the basic processing circuit 0 to output to the basic processing circuit 1. Similarly, the basic processing circuit 1 must continue to output the data of the vector P after receiving the data. to the base processing circuit 2.

Each basic processing circuit performs operations on the received data, including but not limited to: inner product operations, multiplication operations, addition operations, etc.;

In an alternative, the base processing circuit computes the multiplication of one or more sets of two data at a time, and then accumulates the results into registers and/or on-chip caches;

In an alternative, the underlying processing circuit computes the inner product of one or more sets of two vectors at a time, and then accumulates the results into registers and or on-chip caches;

After the basic processing circuit calculates the result, it transmits the result from the data output interface (that is, to other basic processing circuits connected to it);

In an optional solution, the calculation result may be the final result or the intermediate result of the inner product operation;

After the basic processing circuit receives the calculation results from other basic processing circuits, it transmits the data to other basic processing circuits or main processing circuits connected to it;

The main processing circuit receives the result of the inner product operation of each basic processing circuit, and processes the result to obtain a final result (the processing may be an accumulation operation or an activation operation, etc.).

Adopt the above-mentioned computing device to realize the embodiment of matrix multiplication vector method:

In an optional solution, the plurality of basic processing circuits used in the method are arranged in the manner shown in Figure 2d or Figure 2e below;

As shown in Figure 2c, the data conversion operation circuit of the main processing circuit converts the matrix S and the matrix P into fixed-point data; the control circuit of the main processing unit divides the data of the M rows of the matrix S into K groups, which are divided by the i-th basic The processing circuit is responsible for the operation of the i-th group (the set of rows in this group of data is denoted as Ai);

The method for grouping M rows of data here is an arbitrary grouping method that will not be repeatedly allocated;

In an optional solution, the following allocation method is adopted: the jth row is allocated to the j%Kth (% is the remainder operation) basic processing circuit;

In an optional solution, in the case of not being able to be grouped equally, a part of the rows can be evenly allocated first, and the remaining rows can be allocated in an arbitrary manner.

The control circuit of the main processing circuit sends the data in part or all of the rows in the matrix S to the corresponding basic processing circuit in turn;

In an optional solution, the control circuit of the main processing circuit sends one or more data in a row of data in the i-th group of data Mi that it is responsible for to the i-th basic processing circuit each time;

In an optional solution, the control circuit of the main processing circuit sends one or more data of each row in part or all of the rows in the i-th group of data Mi that it is responsible for to the i-th basic processing circuit each time;

The control circuit of the main processing circuit sends the data in the vector P to the first basic processing circuit in turn;

In an optional solution, the control circuit of the main processing circuit can send one or more data in the vector P each time;

After receiving the data of the vector P, the i-th basic processing circuit sends it to the i+1-th basic processing circuit connected to it;

Each basic processing circuit performs operations (including but not limited to multiplication or addition) after receiving one or more data from a certain row or several rows in the matrix S and one or more data from the vector P;

In an alternative, the base processing circuit computes the multiplication of one or more sets of two data at a time, and then accumulates the results into registers and/or on-chip caches;

In an alternative, the underlying processing circuit computes the inner product of one or more sets of two vectors at a time, and then accumulates the results into registers and or on-chip caches;

In an optional solution, the data received by the basic processing circuit can also be an intermediate result, which is stored in a register and or on-chip cache;

The basic processing circuit transmits the local calculation result to the next basic processing circuit or main processing circuit connected to it;

In an optional solution, corresponding to the structure of Fig. 2d, only the output interface of the last basic processing circuit of each column is connected to the main processing circuit. In this case, only the last basic processing circuit can directly connect the local The calculation results of the basic processing circuit are transmitted to the main processing circuit, and the calculation results of other basic processing circuits are passed to the next basic processing circuit of their own, and the next basic processing circuit is passed to the next basic processing circuit until all are transmitted to the last basic processing circuit. , the last basic processing circuit accumulates the local calculation results and the received results of other basic processing circuits in this column to obtain intermediate results, and sends the intermediate results to the main processing circuit; of course, it can also be the last basic processing circuit. The results of the other base circuits in this column and the local processing results are sent directly to the main processing circuit.

In an optional solution, corresponding to the structure of FIG. 2e, each basic processing circuit has an output interface connected to the main processing circuit. In this case, each basic processing circuit directly transmits the local calculation result to the main processing circuit;

After the basic processing circuit receives the calculation result passed by other basic processing circuits, it transmits it to the next basic processing circuit or main processing circuit connected to it.

The main processing circuit receives the results of the M inner product operations as the operation result of multiplying a matrix by a vector.

using the circuit arrangement to perform a matrix-by-matrix operation;

The following describes the operation to calculate the multiplication of a matrix S of size M rows and L columns and a matrix P of size L rows and N columns, (each row in matrix S is the same length as each column of matrix P, as shown in Figure 2f)

The method is described using the embodiment of the device as shown in FIG. 1b;

The data conversion operation circuit of the main processing circuit converts the matrix S and the matrix P into fixed-point type data;

The control circuit of the main processing circuit sends the data in some or all of the rows of matrix S to those underlying processing circuits that are directly connected to the main processing circuit through the horizontal data input interface (for example, the uppermost gray-filled vertical data in Figure 1b). path);

In an optional solution, the control circuit of the main processing circuit sends the data of a certain row in the matrix S one number or part of the number to a certain basic processing circuit at a time; (for example, for a certain basic processing circuit, the first time Send the 1st number in the 3rd line, send the 2nd number in the 3rd line for the 2nd time, send the 3rd number in the 3rd line for the 3rd time..., or send the first two in the 3rd line for the 1st time Numbers, the second time to send the 3rd and 4th numbers on the 3rd line, the third time to send the 5th and 6th numbers on the 3rd line... ;)

In an optional solution, the control circuit of the main processing circuit sends the data of certain rows in the matrix S each time a number or a part of the number to a certain basic processing circuit; (for example, for a certain basic processing circuit, the first 1 time to send the 1st number of each line 3,4,5, 2nd time to send the 2nd number of each line of 3,4,5 lines, 3rd time to send the 3rd,4,5 lines of each line The 3rd number..., or the first two numbers of the 3rd, 4th, and 5th lines of each line are sent, the second time the 3rd and 4th numbers of each line of the 3rd, 4th, and 5th lines are sent, and the third Send the 3rd, 4th, 5th line 5th and 6th numbers each time... ;)

The control circuit of the main processing circuit sends the data in some or all of the columns in matrix P to those basic processing circuits that are directly connected to the main processing circuit through the vertical data input interface (for example, the left side of the basic processing circuit array in Figure 1b). horizontal data paths filled in gray);

In an optional solution, the control circuit of the main processing circuit sends a number or a part of the data of a certain column of the matrix P to a certain basic processing circuit at a time; (for example, for a certain basic processing circuit, the first time Send the 1st number in the 3rd column, send the 2nd number in the 3rd column for the 2nd time, send the 3rd number in the 3rd column for the 3rd time..., or send the first two in the 3rd column for the 1st time Numbers, the second send the 3rd and 4th numbers in the 3rd column, the third time the 5th and 6th numbers in the 3rd column... ;)

In an optional solution, the control circuit of the main processing circuit sends the data of certain columns in the matrix P each time a number or a part of the number to a certain basic processing circuit; (for example, for a certain basic processing circuit, the first 1 time to send the 1st number of each column of the 3rd,4th,5th column, 2nd time to send the 2nd number of each column of the 3rd,4th,5th column, 3rd time to send the 3rd,4th,5th column of each column The 3rd number..., or the first two numbers in the 3rd, 4th, and 5th columns are sent, and the 3rd and 4th numbers in the 3rd, 4th, and 5th columns are sent in the second time, and the third Send the 3rd, 4th, 5th column 5th and 6th numbers each time... ;)

After the basic processing circuit receives the data of the matrix S, it transmits the data to the next basic processing circuit connected to it through its horizontal data output interface (for example, the white-filled horizontal data path in the middle of the basic processing circuit array in Figure 1b). ); after the basic processing circuit receives the data of the matrix P, it transmits the data to the next basic processing circuit connected to it through its vertical data output interface (for example, the white-filled ones in the middle of the basic processing circuit array in FIG. 1b vertical data path);

Each basic processing circuit operates on the received data;

In an alternative, the base processing circuit computes the multiplication of one or more sets of two data at a time, and then accumulates the results into registers and/or on-chip caches;

In an alternative, the underlying processing circuit computes the inner product of one or more sets of two vectors at a time, and then accumulates the results into registers and or on-chip caches;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an optional solution, the calculation result may be the final result or the intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface, if not, the result is output in the direction of the basic processing circuit that can directly output to the main processing circuit (for example, Fig. In 1b, the basic processing circuit in the bottom row directly outputs its output result to the main processing circuit, and other basic processing circuits transmit the operation result downward from the vertical output interface).

After the basic processing circuit receives the calculation results from other basic processing circuits, it transmits the data to other basic processing circuits or main processing circuits connected to it;

Output the results in the direction that can be directly output to the main processing circuit (for example, in Figure 1b, the basic processing circuit in the bottom row outputs its output results directly to the main processing circuit, and other basic processing circuits transmit operations downward from the vertical output interface. result);

The main processing circuit can obtain the output result after receiving the result of the inner product operation of each basic processing circuit.

Example of "matrix by matrix" method:

The method uses the basic processing circuit array arranged as shown in Figure 1b, assuming that there are h rows and w columns;

The data conversion operation circuit of the main processing circuit converts the matrix S and the matrix P into fixed-point type data;

The control circuit of the main processing circuit divides the h-row data of the matrix S into h groups, and the i-th basic processing circuit is responsible for the operation of the i-th group (the set of rows in this group of data is denoted as Hi);

The method of grouping the h rows of data here is any grouping method that will not be repeatedly allocated;

In an optional solution, the following allocation method is adopted: the control circuit of the main processing circuit allocates the jth row to the j%hth basic processing circuit;

In an optional solution, in the case of not being able to be grouped equally, a part of the rows can be evenly allocated first, and the remaining rows can be allocated in an arbitrary manner.

The control circuit of the main processing circuit divides the W column data of the matrix P into w groups, and the i-th basic processing circuit is responsible for the operation of the i-th group (the set of rows in this group of data is denoted as Wi);

The method of grouping the data in column W here is any grouping method that will not be repeatedly allocated;

In an optional solution, the following allocation method is adopted: the control circuit of the main processing circuit allocates the jth row to the j%wth basic processing circuit;

In an optional solution, in the case that the grouping cannot be evenly divided, a part of the columns can be evenly allocated first, and the remaining columns can be allocated in an arbitrary manner.

The control circuit of the main processing circuit sends the data in some or all of the rows of the matrix S to the first basic processing circuit of each row in the array of basic processing circuits;

In an optional solution, the control circuit of the main processing circuit sends one or more pieces of data in a row of the i-th group of data Hi that it is responsible for to the first basic processing circuit in the i-th row in the basic processing circuit array each time. data;

In an optional solution, the control circuit of the main processing circuit sends each time the first basic processing circuit of the i-th row in the basic processing circuit array is responsible for every part or all of the rows in the i-th group of data Hi. one or more data of the row;

The control circuit of the main processing circuit sends the data in some or all of the columns of the matrix P to the first basic processing circuit of each column in the basic processing circuit array;

In an optional solution, the control circuit of the main processing circuit sends one or more data of a column in the i-th group of data Wi that it is responsible for to the first basic processing circuit of the i-th column in the basic processing circuit array each time. data;

In an optional solution, the control circuit of the main processing circuit sends each time the first basic processing circuit of the i-th column in the basic processing circuit array is responsible for every part or all of the columns in the i-th group of data Ni. one or more data for a column;

After the basic processing circuit receives the data of the matrix S, it transmits the data to the next basic processing circuit connected to it through its horizontal data output interface (for example, the white-filled horizontal data path in the middle of the basic processing circuit array in Figure 1b). ); after the basic processing circuit receives the data of the matrix P, it transmits the data to the next basic processing circuit connected to it through its vertical data output interface (for example, the white-filled ones in the middle of the basic processing circuit array in FIG. 1b vertical data path);

Each basic processing circuit operates on the received data;

In an alternative, the base processing circuit computes the multiplication of one or more sets of two data at a time, and then accumulates the results into registers and/or on-chip caches;

In an alternative, the underlying processing circuit computes the inner product of one or more sets of two vectors at a time, and then accumulates the results into registers and or on-chip caches;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an optional solution, the calculation result may be the final result or the intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface, if not, the result is output in the direction of the basic processing circuit that can directly output to the main processing circuit (for example, the most The following line of basic processing circuits directly output their output results to the main processing circuit, and other basic processing circuits transmit the operation results downward from the vertical output interface).

After the basic processing circuit receives the calculation results from other basic processing circuits, it transmits the data to other basic processing circuits or main processing circuits connected to it;

Output results in a direction that can be directly output to the main processing circuit (for example, the basic processing circuit in the bottom row directly outputs its output results to the main processing circuit, and other basic processing circuits transmit the operation results downward from the vertical output interface);

The main processing circuit can obtain the output result after receiving the result of the inner product operation of each basic processing circuit.

The words "horizontal" and "vertical" used in the above description are only to describe the example shown in Figure 1b. In actual use, it is only necessary to distinguish the "horizontal" and "vertical" interfaces of each unit to represent two different interfaces, namely Can.

A fully connected operation is done using the described circuit arrangement:

If the input data of the fully connected layer is a vector (that is, the input of the neural network is a single sample), the weight matrix of the fully connected layer is used as the matrix S, the input vector is used as the vector P, and the matrix multiplication is performed according to the use of the device. perform operations in vector methods;

If the input data of the fully connected layer is a matrix (that is, when the input of the neural network is multiple samples), the weight matrix of the fully connected layer is used as the matrix S, the input vector is used as the matrix P, or the weight of the fully connected layer is used as the matrix S. The value matrix is used as the matrix P, the input vector is used as the matrix S, and the operation is performed according to the matrix multiplication of the device by the matrix;

The convolution operation is performed using the described circuit arrangement:

The convolution operation is described below. In the figure below, a block represents a data, and the input data is represented by Figure 3a (N samples, each sample has C channels, and the feature map of each channel has a height of H and a width of W) , and the weights, that is, the convolution kernels, are shown in Figure 3b (there are M convolution kernels, and each convolution kernel has C channels, and the height and width are KH and KW, respectively). For N samples of input data, the rules of the convolution operation are the same. The following explains the process of performing the convolution operation on a sample. On a sample, each of the M convolution kernels must perform the same Operation, each convolution kernel operation obtains a plane feature map, and M convolution kernels finally calculate M plane feature maps, (for a sample, the output of the convolution is M feature maps), for a convolution kernel , to perform the inner product operation at each plane position of a sample, and then slide along the H and W directions. For example, Figure 3c shows that a convolution kernel performs the inner product operation at the lower right corner of a sample of the input data. Corresponding figures; Figure 3d shows that the position of the convolution slides one cell to the left and Figure 3e shows that the position of the convolution slides one cell up.

The method is described using the embodiment of the device as shown in FIG. 1b;

The data conversion operation circuit of the main processing circuit can convert part or all of the data in the convolution kernel of the weight into fixed-point data, and the control circuit of the main processing circuit sends part or all of the data in the convolution kernel of the weight to Those basic processing circuits that are directly connected to the main processing circuit through the horizontal data input interface (eg, the uppermost gray-filled vertical data path in Figure 1b);

In an optional solution, the control circuit of the main processing circuit sends a number or a part of the data of a certain convolution kernel in the weight value to a certain basic processing circuit at a time; (for example, for a certain basic processing circuit, Send the 1st number of the 3rd line for the 1st time, send the 2nd number of the 3rd line data for the 2nd time, send the 3rd number of the 3rd line for the 3rd time..., or send the 3rd line for the 1st time The first two numbers, the 3rd and 4th numbers on the 3rd line are sent the second time, the 5th and 6th numbers on the 3rd line are sent the third time... ;)

In an alternative solution, another situation is that the control circuit of the main processing circuit sends the data of some convolution kernels in the weights each time a number or a part of the number to a basic processing circuit; (for example, For a certain basic processing circuit, the first number of lines 3, 4, and 5 is sent for the first time, the second number of lines 3, 4, and 5 is sent for the second time, and the third number is sent for the third time. The 3rd number of each line of 3,4,5 lines..., or the first two numbers of each line of 3,4,5 lines are sent for the first time, and the 3rd number of each line of 3,4,5 lines is sent for the second time and 4th number, 3rd time sending 3rd, 4th, 5th line 5th and 6th number per line... ;)

The control circuit of the main processing circuit divides the input data according to the convolution position, and the control circuit of the main processing circuit sends part or all of the data in the convolution position in the input data to the main processing circuit directly through the vertical data input interface. those underlying processing circuits that are connected (for example, the gray-filled lateral data paths to the left of the underlying processing circuit array in Figure 1b);

In an optional solution, the control circuit of the main processing circuit sends a number or a part of the data at a certain convolution position in the input data to a certain basic processing circuit at a time; (for example, for a certain basic processing circuit, Send the 1st number in the 3rd column for the 1st time, send the 2nd number in the 3rd column for the 2nd time, send the 3rd number in the 3rd column for the 3rd time..., or send the 3rd column for the 1st time The first two numbers, the 3rd and 4th numbers in the 3rd column are sent the second time, the 5th and 6th numbers in the 3rd column are sent the third time... ;)

In an alternative solution, another situation is that the control circuit of the main processing circuit sends the data of certain convolution positions in the input data each time a number or a part of the number to a certain basic processing circuit; (for example, For a certain basic processing circuit, the first number of columns 3, 4, and 5 is sent for the first time, the second number of columns 3, 4, and 5 is sent for the second time, and the second number of each column is sent for the third time. The 3rd number of each column of 3, 4, 5 columns..., or the first two numbers of columns 3, 4, and 5 are sent, and the second number of columns 3, 4, and 5 is sent to the 3rd number of each column. and 4th number, 3rd time sending 3rd, 4th, 5th column 5th and 6th number in each column... ;)

After the basic processing circuit receives the weight data, it transmits the data to the next basic processing circuit connected to it through its horizontal data output interface (for example, the white-filled horizontal data path in the middle of the basic processing circuit array in Figure 1b). ); after the basic processing circuit receives the data of the input data, it transmits the data to the next basic processing circuit connected to it through its vertical data output interface (for example, the white-filled ones in the middle of the basic processing circuit array in FIG. 1b vertical data path);

Each basic processing circuit operates on the received data;

In an alternative, the base processing circuit computes the multiplication of one or more sets of two data at a time, and then accumulates the results into registers and/or on-chip caches;

In an optional solution, the basic processing circuit calculates the inner product of one or more sets of two vectors at a time, and then accumulates the result into a register and/or on-chip cache;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an optional solution, the calculation result may be the final result or the intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface, if not, the result is output in the direction of the basic processing circuit that can directly output to the main processing circuit (for example, Fig. In 1b, the basic processing circuit in the bottom row directly outputs its output result to the main processing circuit, and other basic processing circuits transmit the operation result downward from the vertical output interface).

After the basic processing circuit receives the calculation results from other basic processing circuits, it transmits the data to other basic processing circuits or main processing circuits connected to it;

Output results in a direction that can be directly output to the main processing circuit (for example, the basic processing circuit in the bottom row directly outputs its output results to the main processing circuit, and other basic processing circuits transmit the operation results downward from the vertical output interface);

The main processing circuit can obtain the output result after receiving the result of the inner product operation of each basic processing circuit.

A method of performing a biasing operation using the circuit arrangement;

The vector operator circuit of the main processing circuit can realize the function of adding two vectors or two matrices;

The function of adding a vector to each row or each column of a matrix can be realized by using the vector operator circuit of the main processing circuit.

In an alternative, the matrix may come from the result of the apparatus performing a matrix-by-matrix operation;

In an optional solution, the vector may come from the result of the apparatus performing a matrix-multiply-vector operation;

In an alternative, the matrix may be from data received externally by the main processing circuit of the device.

In an alternative, the vector may come from data received externally by the main processing circuit of the device.

Including but not limited to the above data sources.

A method for completing the activation function operation using the circuit arrangement:

Using the activation circuit of the main processing circuit, input a vector, and calculate the activation vector of the vector;

In an optional solution, the activation circuit of the main processing circuit passes each value in the input vector through an activation function (the input of the activation function is a numerical value, and the output is also a numerical value), and calculates a numerical value and outputs it to the output vector. corresponding position;

In an optional solution, the activation function can be: y=max(m,x), where x is the input value, y is the output value, and m is a constant;

In an optional solution, the activation function may be: y=tanh(x), where x is the input value and y is the output value;

In an optional solution, the activation function may be: y=sigmoid(x), where x is the input value and y is the output value;

In an alternative, the activation function can be a piecewise linear function;

In an alternative solution, the activation function can be any function that inputs a number and outputs a number.

In an optional solution, the sources of input vectors are (including but not limited to):

external data sources for the device;

In an optional solution, the input data comes from the operation result of matrix multiplication vector by the device;

In an optional solution, the input data comes from the operation result of matrix multiplication performed by the device;

the calculation result of the main processing circuit of the device;

In an optional solution, the input data comes from the calculation result of the main processing circuit of the device after adding the bias.

A method for realizing BLAS (Basic Linear Algebra Subprograms) using the device;

GEMM calculation refers to: the operation of matrix-matrix multiplication in the BLAS library. The usual representation of this operation is: C=alpha*op(S)*op(P)+beta*C, where A and B are the two input matrices, C is the output matrix, alpha and beta are scalars, and op Represents a certain operation on the matrix S or P, in addition, there will be some auxiliary integers as parameters to explain the width and height of A and B of the matrix;

The steps of using the device to realize GEMM calculation are:

The main processing circuit can convert the data type of the input matrix S and the matrix P before performing the OP operation;

The conversion circuit of the main processing circuit performs respective op operations on the input matrix S and the matrix P;

In an optional solution, op can be the transpose operation of the matrix; the vector operation function or the data rearrangement function of the main processing circuit is used (it was mentioned above that the main processing circuit has a data rearrangement circuit) to realize the matrix transposition operation. , of course, in practical applications, the above OP can also be directly implemented by a conversion circuit. For example, when a matrix transpose operation is performed, the OP operation can be directly implemented by a matrix transpose circuit;

In an optional solution, the op of a certain matrix can be empty, and the OP operation is not performed;

The matrix multiplication calculation between op(S) and op(P) is completed by the calculation method of matrix multiplication matrix;

Use the arithmetic logic circuit of the main processing circuit to multiply each value in the result of op(S)*op(P) by alpha;

In an optional solution, the multiplication by alpha operation is not performed when alpha is 1;

Use the arithmetic logic circuit of the main processing circuit to realize the operation of beta*C;

In an optional solution, when beta is 1, the multiplication by beta operation is not performed;

Use the arithmetic logic circuit of the main processing circuit to realize the steps of adding the corresponding positions between the matrices alpha*op(S)*op(P) and beta*C;

In an alternative solution, when beta is 0, no addition operation is performed;

GEMV calculation refers to: the operation of matrix-vector multiplication in the BLAS library. The usual representation of this operation is: C=alpha*op(S)*P+beta*C, where S is the input matrix, P is the input vector, C is the output vector, alpha and beta are scalars, and op represents the pair some operation of matrix S;

The steps of using the device to realize GEMV calculation are:

The main processing circuit can convert the data type of the input matrix S and the matrix P before performing the OP operation;

The conversion circuit of the main processing circuit performs corresponding op operations on the input matrix S;

In an optional solution, op can be a matrix transposition operation; the matrix transposition operation is realized by using the matrix transposition circuit of the main processing circuit;

In an optional solution, the op of a certain matrix can be empty, and the op operation is not performed;

The matrix-vector multiplication calculation between the matrix op(S) and the vector P is completed by the calculation method of matrix multiplication vector;

Use the arithmetic logic circuit of the main processing circuit to multiply each value in the result of op(S)*P by alpha;

In an optional solution, the multiplication by alpha operation is not performed when alpha is 1;

Use the arithmetic logic circuit of the main processing circuit to realize the operation of beta*C;

In an optional solution, when beta is 1, the multiplication by beta operation is not performed;

Use the arithmetic logic circuit of the main processing circuit to realize the steps of adding the corresponding positions between the matrices alpha*op(S)*P and beta*C;

In an alternative solution, when beta is 0, no addition operation is performed;

Implement data type conversion

The data type conversion is realized by using the data type conversion operation circuit of the main processing circuit;

In an optional solution, the form of data type conversion includes but is not limited to: floating-point number to fixed-point number, fixed-point number to floating-point number, etc.;

How to update weights:

The vector operator circuit of the main processing circuit is used to implement the weight update function in the training process of the neural network. Specifically, the weight update refers to a method of using the gradient of the weight to update the weight.

In an optional solution, the vector operator circuit of the main processing circuit is used to perform addition and subtraction operations on the two vectors of the weight value and the weight value gradient to obtain an operation result, and the operation result is the updated weight value.

In an optional solution, the vector operator circuit of the main processing circuit is used to multiply or divide the weight and the weight gradient by a number to obtain the intermediate weight and the intermediate weight gradient value, and the vector operator circuit calculates the intermediate weight value. Add and subtract the gradient value of the intermediate weight to obtain the operation result, and the operation result is the updated weight.

In an optional solution, a set of momentums can be first calculated using the gradient of the weights, and then the updated weights can be obtained by adding and subtracting the momentums and the weights.

A method to implement the reverse operation of the fully connected layer

The reverse operation of the fully connected layer can be divided into two parts, as shown in the figure below, the solid line arrow represents the forward calculation process of the fully connected layer, and the dotted line part represents the reverse calculation process of the fully connected layer.

As can be seen from the above figure, the reverse operation of the fully connected layer can be completed by using the method of using the device to complete the matrix multiplication operation;

Implement the reverse operation of the convolutional layer;

The reverse operation of the convolutional layer can be divided into two parts, as shown in Figure 4a below, the solid arrow indicates the forward calculation process of the convolutional layer, as shown in Figure 4b, the reverse calculation process of the convolutional layer.

For the reverse operation of the convolutional layer shown in FIG. 4a and FIG. 4b, the reverse operation of the convolutional layer can be completed by using the device shown in FIG. 1a and the device shown in FIG. 1b. When performing forward operation or reverse operation, it is actually multiple operations of a neural network, and the multiple operations include, but are not limited to: one of matrix multiplication by matrix, matrix multiplication by vector, convolution operation, activation operation, etc. Or any combination, the above operation manners can be described in the present disclosure, which will not be repeated here.

Embodiments of the present disclosure provide a neural network processor board that can be used in numerous general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, smart homes, appliances, multiprocessor systems, microprocessor-based systems, robotics, programmable consumer electronics, network personal computers , PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

Please refer to FIG. 5a, which is a schematic structural diagram of a neural network processor board according to an embodiment of the present disclosure. As shown in FIG. 5 a , the neural network processor board 10 includes a neural network chip package structure 11 , a first electrical and non-electrical connection device 12 and a first substrate 13 .

The present disclosure does not limit the specific structure of the neural network chip packaging structure 11. Optionally, as shown in FIG. 5b, the neural network chip packaging structure 11 includes: a neural network chip 111, a second electrical and non-electrical connection device 112, a first Two substrates 113 .

The specific form of the neural network chip 111 involved in the present disclosure is not limited. The above-mentioned neural network chip 111 includes but is not limited to a neural network chip that integrates a neural network processor. The above-mentioned chip can be made of silicon material, germanium material, quantum material or molecular material. materials etc. According to the actual situation (for example: harsh environment) and different application requirements, the above neural network chip can be packaged, so that most of the neural network chip is wrapped, and the pins on the neural network chip are passed through gold wires. The other conductors are connected to the outside of the package structure for circuit connection with the outer layers.

The present disclosure does not limit the specific structure of the neural network chip 111. Optionally, please refer to the device shown in FIG. 1a or FIG. 1b.

The present disclosure does not limit the types of the first substrate 13 and the second substrate 113, which may be a printed circuit board (PCB) or a printed wiring board (PWB), and may also be other circuit boards. The material for making the PCB is also not limited.

The second substrate 113 involved in the present disclosure is used to carry the above-mentioned neural network chip 111 , and the neural network chip package structure 11 is obtained by connecting the above-mentioned neural network chip 111 and the second substrate 113 through the second electrical and non-electrical connection device 112 . , which is used to protect the neural network chip 111 and facilitate further packaging of the neural network chip packaging structure 11 and the first substrate 13 .

The packaging method and the structure corresponding to the packaging method of the above-mentioned specific second electrical and non-electrical connection device 112 are not limited, and an appropriate packaging method can be selected according to the actual situation and different application requirements and can be simply improved, for example: flip chip Ball Grid Array Package (Flip Chip Ball Grid Array Package, FCBGAP), Low-profile Quad Flat Package (LQFP), Quad Flat Package with Heat Sink (HQFP), No Lead Quad Flat Package (Quad Flat Non-lead Package, QFN) or Small Pitch Quad Flat Package (Fine-pitch Ball Grid Package, FBGA) and other packaging methods.

Flip Chip is suitable for situations where the area after the package is high or is sensitive to the inductance of the wire and the transmission time of the signal. In addition, a wire bonding (Wire Bonding) packaging method can be used to reduce the cost and improve the flexibility of the packaging structure.

Ball Grid Array (Ball Grid Array) can provide more pins, and the average lead length of the pins is short, which has the function of high-speed signal transmission. Among them, the package can be packaged with Pin Grid Array (PGA) , zero insertion force (Zero Insertion Force, ZIF), single edge contact connection (Single Edge Contact Connection, SECC), contact array (Land Grid Array, LGA), etc. instead.

Optionally, the neural network chip 111 and the second substrate 113 are packaged in a flip chip ball grid array (Flip Chip Ball Grid Array) packaging method. Refer to FIG. 6 for a schematic diagram of a specific neural network chip packaging structure. As shown in FIG. 6 , the above-mentioned neural network chip package structure includes: a neural network chip 21 , pads 22 , solder balls 23 , a second substrate 24 , connection points 25 and pins 26 on the second substrate 24 .

Wherein, the pad 22 is connected to the neural network chip 21, and solder balls 23 are formed by welding between the pad 22 and the connection point 25 on the second substrate 24, and the neural network chip 21 and the second substrate 24 are connected. Packaging of the neural network chip 21 .

The pin 26 is used to connect with the external circuit of the package structure (for example, the first substrate 13 on the neural network processor board 10 ), which can realize the transmission of external data and internal data, and is convenient for the neural network chip 21 or the neural network chip 21 The corresponding neural network processor processes the data. The present disclosure also does not limit the type and quantity of pins, and different pin forms can be selected according to different packaging technologies, and are arranged in accordance with certain rules.

Optionally, the above-mentioned neural network chip package structure further includes insulating fillers, which are placed in the gaps between the pads 22 , the solder balls 23 and the connection points 25 , to prevent interference between the solder balls and the solder balls.

Wherein, the material of the insulating filler may be silicon nitride, silicon oxide or silicon oxynitride; the interference includes electromagnetic interference, inductive interference, and the like.

Optionally, the above-mentioned neural network chip packaging structure further includes a heat dissipation device for dissipating heat during operation of the neural network chip 21 . Wherein, the heat dissipation device may be a metal sheet, a heat sink or a heat sink with good thermal conductivity, such as a fan.

For example, as shown in FIG. 6a, the neural network chip package structure 11 includes: a neural network chip 21, pads 22, solder balls 23, a second substrate 24, connection points 25 on the second substrate 24, pins 26, Insulation filler 27, thermal paste 28 and metal shell heat sink 29. Among them, the heat dissipation paste 28 and the metal shell heat dissipation fins 29 are used to dissipate the heat of the neural network chip 21 during operation.

Optionally, the above-mentioned neural network chip package structure 11 further includes a reinforcing structure, which is connected to the pads 22 and embedded in the solder balls 23 to enhance the connection strength between the solder balls 23 and the pads 22 .

Wherein, the reinforcing structure may be a metal wire structure or a columnar structure, which is not limited herein.

The present disclosure also does not limit the specific form of the first electrical and non-electrical device 12. Reference can be made to the description of the second electrical and non-electrical device 112, that is, the neural network chip package structure 11 is packaged by welding, or a connection can be used. The way of connecting the second substrate 113 and the first substrate 13 by wire connection or plugging is convenient for subsequent replacement of the first substrate 13 or the neural network chip package structure 11 .

Optionally, the first substrate 13 includes an interface and the like for a memory unit used to expand the storage capacity, for example: a synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), a double-rate synchronous dynamic random access memory (Double Date Rate SDRAM, DDR), etc., the processing power of the neural network processor is improved by expanding the memory.

The first substrate 13 may further include a Peripheral Component Interconnect-Express (PCI-E or PCIe) interface, a Small Form-factor Pluggable (SFP) interface, an Ethernet interface, and a controller. A local area network bus (Controller Area Network, CAN) interface, etc., is used for data transmission between the package structure and the external circuit, which can improve the operation speed and the convenience of operation.

The neural network processor is packaged as a neural network chip 111, the neural network chip 111 is packaged as a neural network chip package structure 11, and the neural network chip package structure 11 is packaged as a neural network processor board 10, through the interface ( socket or ferrule) for data interaction with an external circuit (eg, computer motherboard), that is, directly using the neural network processor board 10 to realize the function of the neural network processor and protect the neural network chip 111 . In addition, other modules can be added to the neural network processor board 10, which improves the application scope and operation efficiency of the neural network processor.

In one embodiment, the present disclosure discloses an electronic device including the above-mentioned neural network processor board 10 or neural network chip package structure 11 .

Electronic devices include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cameras, video cameras, projectors, watches, headphones, mobile storage , wearable devices, vehicles, home appliances, and/or medical devices.

The vehicles include airplanes, ships and/or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lamps, gas stoves, and range hoods; the medical equipment includes nuclear magnetic resonance instruments, B-ultrasound and/or electrocardiograph.

The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present disclosure in further detail. It should be understood that the above are only specific embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this disclosure shall be included within the protection scope of this disclosure.

Claims (19)

1. An integrated circuit chip device, characterized in that the integrated circuit chip device comprises: a main processing circuit and a plurality of basic processing circuits; at least one circuit in the main processing circuit or the plurality of basic processing circuits comprises: a data type an arithmetic circuit; the data type arithmetic circuit is used to perform conversion between floating-point type data and fixed-point type data; The plurality of basic processing circuits are distributed in an array; each basic processing circuit is connected to other adjacent basic processing circuits, and the main processing circuit is connected to the n basic processing circuits in the first row and the n basic processing circuits in the mth row. circuit and m basic processing circuits in column 1; The main processing circuit is used to perform each successive operation in the neural network operation and transmit data with the basic processing circuit connected to it; The plurality of basic processing circuits are used to perform operations in the neural network in parallel according to the transmitted data, and transmit the operation results to the main processing circuit through the basic processing circuits connected to the main processing circuit. 2. The integrated circuit chip device according to claim 1, wherein, The main processing circuit is used to obtain the data block to be calculated and an operation instruction, and convert the data block to be calculated into a fixed-point type data block through the data type operation circuit, and the fixed-point type is processed according to the operation instruction. The data block to be calculated is divided into a distribution data block and a broadcast data block; the distribution data block is split to obtain a plurality of basic data blocks, and the plurality of basic data blocks are distributed to the basic processing circuit connected to it, broadcasting the broadcast data block to underlying processing circuits connected thereto; the basic processing circuit, configured to perform an inner product operation on the basic data block and the broadcast data block in a fixed-point data type to obtain an operation result, and send the operation result to the main processing circuit; Or forward the basic data block and the broadcast data block to other basic processing circuits to perform an inner product operation with a fixed-point data type to obtain an operation result, and send the operation result to the main processing circuit; The main processing circuit is configured to convert the operation result into floating point type data through the data type operation circuit, and process the floating point type data to obtain the data block to be calculated and the instruction result of the operation instruction. 3. The integrated circuit chip device according to claim 2, wherein, The main processing circuit is specifically configured to send the broadcast data block to the basic processing circuit connected to it through a broadcast. 4. The integrated circuit chip device according to claim 2, wherein, The basic processing circuit is specifically configured to perform inner product processing on the basic data block and the broadcast data block in a fixed-point data type to obtain an inner product processing result, and accumulate the inner product processing results to obtain an operation result. The operation result is sent to the main processing circuit. 5. The integrated circuit chip device according to claim 4, wherein, The main processing circuit is configured to, if the operation result is the result of inner product processing, accumulate the operation result to obtain an accumulation result, and arrange the accumulation result to obtain the data block to be calculated and the value of the operation instruction. command result. 6. The integrated circuit chip device according to claim 2, wherein, The main processing circuit is specifically configured to divide the broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the basic processing circuit through multiple times; the plurality of partial broadcast data blocks The blocks are combined to form the broadcast data blocks. 7. The integrated circuit chip device according to claim 6, wherein, The basic processing circuit is specifically configured to perform an inner product processing on the part of the broadcast data block and the basic data block in a fixed-point data type to obtain an inner product processing result, and accumulate the inner product processing results to obtain a partial operation result , and send the partial operation result to the main processing circuit. 8. The integrated circuit chip device according to claim 7, wherein, The basic processing circuit is specifically used for multiplexing the part of the broadcast data block n times to perform the inner product operation of the part of the broadcast data block and the n basic data blocks to obtain n partial processing results, and after accumulating the n partial processing results respectively. Obtain n partial operation results, and send the n partial operation results to the main processing circuit, where n is an integer greater than or equal to 2. 9. The integrated circuit chip device according to claim 1, wherein, The main processing circuit includes: a main register or a main on-chip buffer circuit; The basic processing circuit includes: a basic register or a basic on-chip cache circuit. 10. The integrated circuit chip device according to claim 9, wherein, The main processing circuit includes one or any combination of a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, a data type operation circuit or a data rearrangement circuit. 11. The integrated circuit chip device according to claim 1, wherein, The main processing circuit is used to obtain the data block to be calculated and an operation instruction, divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; perform split processing on the distribution data block obtaining a plurality of basic data blocks, distributing the plurality of basic data blocks to the basic processing circuit connected thereto, and broadcasting the broadcast data blocks to the basic processing circuit connected thereto; The basic processing circuit is configured to convert the basic data block and the broadcast data block into a fixed-point type data block, perform an inner product operation according to the fixed-point type data block to obtain an operation result, and convert the operation result into a floating-point type. After the point data is sent to the main processing circuit; Or convert the basic data block and the broadcast data block into fixed-point data blocks, forward the fixed-point data blocks to other basic processing circuits to perform an inner product operation to obtain an operation result, and convert the operation result into a floating-point number The data is then sent to the main processing circuit; The main processing circuit is configured to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction. 12. The integrated circuit chip device according to claim 1, wherein, The data is one or any combination of vectors, matrices, three-dimensional data blocks, four-dimensional data blocks and n-dimensional data blocks. 13. The integrated circuit chip device according to claim 2, wherein, If the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is a broadcast data block, and the multiplicand data block is a distribution data block; Or if the operation instruction is a convolution instruction, the main processing circuit determines that the convolution input data block is a broadcast data block, and the convolution kernel is a distribution data block. 14. A neural network computing device, wherein the neural network computing device comprises one or more integrated circuit chip devices according to any one of claims 1-13. 15. A combined processing device, characterized in that, the combined processing device comprises: the neural network computing device according to claim 14, a general interconnection interface and a general processing device; The neural network computing device is connected with the general processing device through the general interconnection interface. 16. A chip, characterized in that, the chip integrates the device according to any one of claims 1-15. 17. An electronic device, wherein the electronic device comprises the chip of claim 16. 18. A method for computing a neural network, wherein the method is applied in an integrated circuit chip device, the integrated circuit chip device comprising: the integrated circuit chip device according to any one of claims 1-13, The integrated circuit chip device is used to perform the operation of the neural network. 19. The method according to claim 16, wherein the operation of the neural network comprises: convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, paranoid operation, fully connected operation, GEMM operation, GEMV operation, One or any combination of activation operations.