TWI768159B - Integrated circuit chip apparatus and related product - Google Patents

Abstract

The present disclosure provides an integrated circuit chip device and related products, the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; at least one basic processing circuit of the main processing circuit or the plurality of basic processing circuits includes: data A type operation circuit, the data type operation circuit is used to perform conversion between floating point type data and fixed point type data. The technical solution provided by the present disclosure has the advantages of small calculation amount and low power consumption.

Description

Integrated circuit chip devices and related products

The present disclosure relates to the field of neural networks, and in particular, to an integrated circuit chip device and related products.

Artificial Neural Network (ANN) has been a research hotspot in the field of artificial intelligence since the 1980s. It abstracts the human brain neuron network from the perspective of information processing, establishes a certain simple model, and forms different networks according to different connection methods. In engineering and academia, it is often simply referred to as neural network or neural-like network. Neural network is an operation model, which is composed of a large number of nodes (or neurons) connected with each other. The operation of the existing neural network is based on a CPU (Central Processing Unit, central processing unit) or a GPU (Graphics Processing Unit, graphics processing unit) to realize the operation of the neural network, which requires a large amount of calculation and high power consumption.

Embodiments of the present disclosure provide an integrated circuit chip device and related products, which can improve the processing speed and efficiency of the computing device.

A first aspect provides an integrated circuit chip device, the integrated circuit chip device includes: a main processing circuit, k branch circuits and k groups of basic processing circuits, the main processing circuit is connected to the k branch circuits respectively, k Each of the branch circuits corresponds to a group of basic processing circuits in the k groups of basic processing circuits, and the group of basic processing circuits includes at least one basic processing circuit; the branch circuits include: a data type operation circuit for executing Conversion between floating-point type data and fixed-point type data; the main processing circuit is used to perform each successive operation in the neural network operation and transmit data with the k branch circuits connected to it; the k branches a circuit for forwarding the transmission data between the main processing circuit and the k groups of basic processing circuits, and controlling whether to start the data type operation circuit to perform conversion on the type of the transmission data according to the operation of the transmission data; the The k groups of basic processing circuits are used to execute the operation in the neural network in parallel according to the transmission data or the converted transmission data, and transmit the operation result to the main processing circuit through the branch circuit connected to the main processing circuit circuit.

A second aspect provides a neural network computing device, the neural network computing device comprising one or more integrated circuit chip devices provided in the first aspect.

In a third aspect, a combined processing device is provided. The combined processing device includes: the neural network computing device, a general interconnection interface, and a general processing device provided in the second aspect; the neural network computing device communicates with the neural network computing device through the general interconnection interface. Universal processing unit connection.

A fourth aspect provides a chip that integrates the device of the first aspect, the device of the second aspect, or the device of the third aspect.

In a fifth aspect, an electronic device is provided, and the electronic device includes the chip of the fourth aspect.

In a sixth aspect, a method for computing a neural network is provided, the method is applied in an integrated circuit chip device, and the integrated circuit chip device includes: the integrated circuit chip device according to the first aspect, the integrated circuit chip device is used for Execute the operations of the neural network.

It can be seen that, through the embodiments of the present disclosure, a data conversion operation circuit is provided to perform operations after converting the types of data blocks, which saves transmission resources and computing resources, so it has the advantages of low power consumption and small calculation amount.

S201, S202, S203, S204, S201b, S202b, S203b, S301, S302, S303, S304: Steps

A, Ai, B, S: Matrix

P: vector

10: Neural network processor board

11: Neural network chip packaging structure

12: First electrical and non-electrical connection device

13: The first substrate

111: Neural Network Chip

112: Second electrical and non-electrical connection device

113: Second substrate

1111: storage unit

1112: Direct Memory Access Unit

1113: Instruction cache unit

1114: Weight cache unit

1115: Input neuron buffer unit

1116: output neuron buffer unit

1117: Control Unit

1118: arithmetic unit

21: Neural Network Chip

22: Pad

23: Solder Balls

24: Second substrate

25: Connection point on the second substrate 24

26: pin

27: Insulation filler

28: Thermal paste

29: Metal shell heat sink

FIG. 1a is a schematic structural diagram of an integrated circuit chip device.

FIG. 1b is a schematic structural diagram of another integrated circuit chip device.

FIG. 1c is a schematic structural diagram of a basic processing circuit.

FIG. 1d is a schematic structural diagram of a fixed-point data type.

FIG. 2 is a schematic diagram of a process flow of multiplying a matrix by a vector.

Figure 2a is a schematic diagram of a matrix multiplied by a vector.

FIG. 2b is a schematic flow chart of a matrix multiplication matrix.

Figure 2c is a schematic diagram of matrix Ai multiplied by a vector.

Figure 2d is a schematic diagram of matrix A multiplied by matrix B.

Figure 2e is a schematic diagram of matrix Ai multiplied by matrix B.

Figure 3a is a schematic diagram of neural network training.

Figure 3b is a schematic diagram of the convolution operation.

Figure 4a is a schematic diagram of the forward operation of the neural network.

Figure 4b is a schematic diagram of the reverse operation of the neural network.

FIG. 4c also discloses a schematic structural diagram of a combined processing device for the present disclosure.

FIG. 4d also discloses another structural schematic diagram of a combined processing device for the present disclosure.

Figure 5a is a schematic diagram of another forward operation of the neural network.

Figure 5b is a schematic diagram of another reverse operation of the neural network.

5c is a schematic structural diagram of a neural network processor board provided by an embodiment of the disclosure; FIG. 5d is a schematic structural diagram of a neural network chip packaging structure provided by an embodiment of the disclosure; Schematic diagram of the structure of a network chip; FIG. 6 is a schematic diagram of a packaging structure of a neural network chip provided by an embodiment of the disclosure; FIG. 6a is a schematic diagram of another packaging structure of a neural network chip provided by the embodiment of the disclosure.

In order for those skilled in the art to better understand the disclosed solutions, the technical solutions in the disclosed embodiments will be clearly and completely described below with reference to the drawings in the disclosed embodiments. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those with ordinary knowledge in the technical field without creative work shall fall within the protection scope of the present disclosure.

In the device provided in the first aspect, the main processing circuit is configured to obtain a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; Splitting the distribution data block to obtain a plurality of basic data blocks, distributing the plurality of basic data blocks to the k branch circuits connected to it, and broadcasting the broadcast data block to the k connected to it branch circuits; the k branch circuits are used to receive basic data blocks and broadcast data blocks, and start the data type operation circuit to convert the basic data blocks and broadcast data blocks into fixed-point data types; Forwarding to k groups of basic processing circuits in the fixed-point data type; the basic processing circuit is configured to perform an inner product operation on the basic data block and the broadcast data block in the fixed-point data type to obtain an operation result, and send the operation result to the k branch circuits; the k branch circuits are used to convert the operation result into a floating-point type operation result, and send the floating-point type operation result to the main processing circuit; the main processing circuit, It is used to process the operation result of the floating point type to obtain the data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided in the first aspect, the main processing circuit is specifically configured to broadcast the broadcast data block to the k branch circuits at one time.

In the apparatus provided in the first aspect, the main processing circuit is specifically configured to divide the broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the k through multiple broadcasts branch circuit.

In the apparatus provided in the first aspect, the basic processing circuit is specifically configured to perform an inner product processing on the part of the broadcast data block and the basic data block in a fixed-point type to obtain an inner product processing results, accumulating the inner product processing results to obtain partial operation results, sending the partial operation results to the k branch circuits, and the k branch circuits are used to convert the partial operation results into floating point Type data is sent to the main processing circuit.

In the device provided in the first aspect, the basic processing circuit is specifically configured to multiplex the part of the broadcast data block n times to perform an inner product operation of the part of the broadcast data block and the n basic data blocks in a fixed-point data type to obtain fixed-point data n partial processing results of the type, the n partial processing results of the fixed-point data type are respectively accumulated to obtain n partial operation results of the fixed-point type, and the n partial operation results of the fixed-point type are sent to the branch circuit; the branch A circuit for converting the n partial operation results of the fixed point type into n partial operation results of the floating point type, and sending the n partial operation structures of the floating point type to the main processing circuit, where n is greater than or equal to 2 the integer.

In the device provided in the first aspect, the main processing circuit includes: a main register or a main on-chip buffer circuit; or the branch circuit includes: a basic register or a basic on-chip buffer circuit; or the basic processing circuit includes: a basic register or Basic on-chip cache circuit.

In the device provided in the first aspect, the main processing circuit includes: a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transpose circuit, a direct memory access circuit, a data type operation circuit or a data rearrangement circuit one or any combination.

In the apparatus provided in the first aspect, the data is one or any combination of vectors, matrices, three-dimensional data blocks, four-dimensional data blocks, and n-dimensional data blocks.

In the device provided in the first aspect, if the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is a broadcast data block, and the multiplicand data block is a distribution data block; if the operation instruction is a volume The main processing circuit determines that the input data block is a broadcast data block, and the convolution kernel is a distribution data block.

In the method provided in the fourth aspect, the operation of the neural network includes: one of convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, bias operation, fully connected operation, GEMM operation, GEMV operation, and activation operation or any combination.

Referring to FIG. 1a, FIG. 1a is a schematic structural diagram of an integrated circuit chip device. As shown in FIG. 1a, the chip device includes: a main processing circuit, a basic processing circuit and a branch processing circuit. Specifically, the integrated circuit chip device includes: a main processing circuit, k branch circuits (as shown in FIG. 1a, k=4, of course, other values can also be used in practical applications, such as 8, 16, etc.) and K groups of basic processing circuits, the main processing circuit is respectively connected to the k branch circuits, each branch circuit in the k branch circuits corresponds to a group of basic processing circuits in the k groups of basic processing circuits, and the group of basic processing circuits The circuit includes at least one basic processing circuit; the branch circuit includes: a data type operation circuit for performing conversion between floating point type data and fixed point type data; the main processing circuit for performing each of the neural network operations Continuous operation and transmission of data with the k branch circuits connected to it; the k branch circuits are used to forward the transmission data between the main processing circuit and the k groups of basic processing circuits, according to the transmission data. Operation control whether to start the data type operation circuit to perform conversion on the type of the transmission data; the k groups of basic processing circuits are used to perform operations in the neural network in parallel according to the transmission data or the converted transmission data , and transmit the operation result to the main processing circuit through the branch circuit connected to the main processing circuit

The main processing circuit may include a register and/or an on-chip buffer circuit, and the main processing circuit may further include: a control circuit, a vector operator circuit, an ALU (arithmetic and logic unit, arithmetic logic unit) circuit, an accumulator circuit, a DMA (Direct Memory) circuit Access, direct memory access) circuits and other circuits, of course, in practical applications, the above-mentioned main processing circuits can also be added, conversion circuits (such as matrix transposition circuits), data rearrangement circuits or activation circuits and other circuits; optional; Yes, the main processing circuit may include: a data type conversion operation circuit, and the data type conversion operation circuit can be used to convert the received or transmitted data from floating-point type data to fixed-point type data. Of course, in practical applications, fixed-point type data can also be converted. Data is converted to floating point type data. The present invention does not limit the specific form of the above-mentioned data type conversion operation circuit.

The main processing circuit also includes a data transmission circuit, a data reception circuit or an interface. The data transmission circuit can integrate a data distribution circuit and a data broadcast circuit. Of course, in practical applications, the data distribution circuit and the data broadcast circuit can also be set separately; in practical applications The above-mentioned data transmission circuit and data reception circuit can also be integrated together to form a data transmission and reception circuit. For broadcast data, that is the data that needs to be sent to each underlying processing circuit. For the distribution data, that is, the data that needs to be selectively sent to some basic processing circuits, the specific selection method can be specifically determined by the main processing circuit according to the load and the calculation method. For the broadcast transmission method, the broadcast data is transmitted to each basic processing circuit in a broadcast form. (In practical applications, broadcast data is sent to each basic processing circuit by one broadcast, and broadcast data can also be sent to each basic processing circuit by multiple broadcasts. The specific implementation of this application does not limit the above The number of broadcasts), for the distribution transmission mode, the distribution data is selectively sent to some basic processing circuits.

When distributing data, the control circuit of the main processing circuit transmits data to some or all of the basic processing circuits (the data may be the same or different. Specifically, if the distribution method is adopted The data received by each basic processing circuit that receives data may be different, and of course some basic processing circuits may receive the same data; specifically, when broadcasting data, the control circuit of the main processing circuit sends data to some or all of The processing circuits transmit data, and each underlying processing circuit that receives the data can receive the same data.

Optionally, the vector operator circuit of the above-mentioned main processing circuit can perform vector operations, including but not limited to: addition, subtraction, multiplication and division of two vectors, addition, subtraction, multiplication, and division between vectors and constants, or performing operations on each element in the vector. Arbitrary operation. The continuous operation may specifically be, vector and constant addition, subtraction, multiplication, division operation, activation operation, accumulation operation, and the like.

Each basic processing circuit may include a basic register and/or a basic on-chip buffer circuit; each basic processing circuit may further include one or any combination of an inner product operator circuit, a vector operator circuit, an accumulator circuit, and the like. The above inner product operator circuit, vector operator circuit and accumulator circuit may all be integrated circuits, and the above inner product operator circuit, vector operator circuit and accumulator circuit may also be separately provided circuits.

Optionally, the chip device may further include one or more branch processing circuits. For example, when a branch processing circuit is provided, the main processing circuit is connected to the branch processing circuit, the branch processing circuit is connected to the basic processing circuit, and the internal processing circuit of the basic processing circuit is connected. The product operator circuit is used to perform the inner product operation between the data blocks, the control circuit of the main processing circuit controls the data receiving circuit or the data transmitting circuit to send and receive external data, and controls the data transmitting circuit to distribute the external data to the branch processing through the control circuit A circuit, the branch processing circuit is used to send and receive data from the main processing circuit or the basic processing circuit. The structure shown in Figure 1a is suitable for the calculation of complex data, because for the main processing circuit, the number of connected units is limited, so it is necessary to add branch processing circuits between the main processing circuit and the basic processing circuit to achieve more Access to basic processing circuits to enable computation of complex data blocks. Branch processing circuits and base processing The connection structure of the circuit can be arbitrary, and is not limited to the H-type structure in FIG. 1a. Optionally, the main processing circuit to the basic processing circuit is a broadcast or distribution structure, and the basic processing circuit to the main processing circuit is a gather structure. Broadcast, distribution and collection are defined as follows. For distribution or broadcast structure, the number of basic processing circuits at this time is greater than that of main processing circuits, that is, 1 main processing circuit corresponds to multiple basic processing circuits, that is, from the main processing circuit to multiple basic processing circuits. The processing circuit is a broadcast or distribution structure, and conversely, from a plurality of base processing circuits to the main processing circuit can be a collection structure.

The basic processing circuit receives the data distributed or broadcasted by the main processing circuit and saves it to the on-chip cache of the basic processing circuit, can perform operations to generate results, and can send data to the main processing circuit.

The data involved in the basic processing circuit can be data of any data type, data represented by floating-point numbers of any bit width, or data represented by fixed-point numbers of any bit width; all operation circuits and storage involved. The circuits can be arithmetic circuits and storage circuits of any data type that can be processed, arithmetic circuits and storage circuits of floating-point numbers of any bit width, or fixed-point arithmetic circuits and storage circuits of arbitrary bit widths.

Optionally, each basic processing circuit may include a data type conversion operation circuit, or a data type conversion operation circuit may be configured in some basic processing circuits; the data type conversion operation circuit may be used to convert the received or sent data from floating-point type. The data is converted into fixed-point type data, and the fixed-point type data can also be converted into floating-point type data. The present invention does not limit the specific form of the above-mentioned data type conversion operation circuit.

Optionally, the vector operator circuit of the basic processing circuit can perform vector operations on the two vectors after data type conversion. Of course, in practical applications, the inner product operator circuit of the basic processing circuit can perform the data type conversion. Two vectors perform an inner product operation, and the accumulator circuit can also accumulate the results of the inner product operation.

In an optional solution, the two vectors can be stored in the on-chip cache and/or register, and the basic processing circuit can extract the two vectors to perform operations according to actual computing needs. The operations include, but are not limited to, inner product operations, multiplication operations, addition operations, or other operations.

In an alternative solution, the result of the inner product operation can be accumulated in the on-chip cache and/or register; the advantages of the alternative solution are that the amount of data transmission between the basic processing circuit and the main processing circuit is reduced, and the operation is improved. efficiency, reducing data transmission power consumption.

In an optional solution, the result of the inner product operation is not accumulated, and is directly transmitted as the result; the advantage of this technical solution is that the amount of computation inside the basic processing circuit is reduced, and the computational efficiency of the basic processing circuit is improved.

In an optional solution, each basic processing circuit can perform multiple sets of inner product operations of two vectors, and can also accumulate the results of multiple sets of inner product operations respectively; in an alternative solution, two of the multiple sets of Vector data can be stored in on-chip caches and/or registers; in an optional solution, the results of multiple sets of inner product operations can be accumulated in on-chip caches and/or registers respectively; The result of the operation can be directly transmitted as the result without being accumulated; in an optional solution, each basic processing circuit can perform the inner product operation of the same vector and multiple vectors respectively ("one-to-many" inner product, That is, one of the two vectors in each group in the multi-group inner product is shared), and the inner product results corresponding to each vector are accumulated separately. This technical solution can realize the same set of weights to perform multiple calculations on different input data, increase data multiplexing, reduce the data transmission amount of data in the basic processing circuit, improve calculation efficiency, and reduce power consumption.

Specifically, in the data used to calculate the inner product, the data source of the vector shared by each group and the other vector of each group (that is, the vector that is different between each group) can be different: in an alternative solution, within the calculation When calculating the product, the vectors shared by each group come from the broadcast or distribution of the main processing circuit or the branch processing circuit; in an alternative solution, when calculating the inner product, the vectors shared by each group come from the on-chip cache; in an alternative solution, When calculating the inner product, the vectors shared by each group come from registers; in an alternative, when calculating the inner product, the other non-shared vector of each group is broadcast or distributed from the main processing circuit or the branch processing circuit; in a In the alternative, when calculating the inner product, the other non-shared vector of each group comes from the slave on-chip cache; in an alternative, when calculating the inner product, the other non-shared vector of each group comes from the register; in a In an optional solution, when performing inner product operations of multiple groups, the shared vectors of each group are retained in any number of copies in the on-chip cache and/or registers of the basic processing circuit; One copy of each product is reserved; in an optional solution, only one copy of the shared vector can be reserved; specifically, the results of multiple groups of inner product operations can be respectively accumulated in the on-chip cache and/or register; The result of the operation can be directly transmitted as the result without being accumulated; Referring to the structure shown in FIG. 1a, it includes a main processing circuit (which can perform vector operations) and a multi-base processing circuit (which can perform inner product operations). The advantage of such a combination is that the device can not only use the basic processing circuit to perform matrix and vector multiplication operations, but also use the main processing circuit to perform other arbitrary vector operations, so that the device can complete more rapid operations with limited hardware circuit configuration. More operations, reduce the number of data transmission with the outside of the device, improve computing efficiency, and reduce power consumption. In addition, this chip can be provided with a data type conversion operation circuit in both the basic processing circuit and/or the main processing circuit, so that when performing neural network calculations, floating-point type data can be converted into fixed-point type data, and fixed-point type data can also be converted into Floating-point type data, and the chip can convert the data type by that circuit dynamically according to the calculation amount (that is, the load) of each circuit (mainly the main processing circuit and the basic processing circuit), which can reduce the complexity of data calculation. Programs, reduce power consumption, and dynamically allocate data type conversions without affecting the computing efficiency of the chip. The distribution methods include, but are not limited to, load balancing, load minimum value distribution, and the like.

Referring to the device shown in FIG. 1b, the device shown in FIG. 1b is a computing device in which the branch processing circuit is independently connected to the basic processing circuit. The device shown in FIG. 1b includes: a main processing circuit and N basic processing circuits, wherein, The main processing circuit (the specific structure is shown in Figure 1c) and the N basic processing circuits can be directly or indirectly connected. If the connection is indirect, an optional solution can include N/4 branches as shown in Figure 1a. Processing circuit, each branch processing circuit is connected to 4 basic processing circuits respectively. For the circuits respectively included in the main processing circuit and the N basic processing circuits, please refer to the description shown in Figure 1a above, which will not be repeated here. Yes, the above-mentioned basic processing circuits can also be arranged in branch processing circuits. In addition, the number of basic processing circuits connected to each branch processing circuit may not be limited to four, and manufacturers can configure them according to actual needs. The above-mentioned main processing circuit and/or the N basic processing circuits may each include a data type conversion operation circuit. Specifically, the main processing circuit may include The data type operation circuit may also include N basic processing circuits or a part of them including the data type conversion circuit, or may include both the main processing circuit and the N basic processing circuits or a part of them. The above-mentioned main processing circuit can dynamically allocate the operation entity of the data type conversion step according to the neural network calculation instruction. Specifically, the main processing circuit can determine whether to perform the data type conversion step on the received data according to its own load. The load value is set to multiple intervals, and each interval corresponds to the execution body of the data type conversion step. For example, taking 3 intervals as an example, the load value of interval 1 is low, and the data type conversion step can be executed by the main processing circuit alone. The load value in interval 2 is between interval 1 and interval 3, and the data type conversion step can be performed by the main processing circuit or N basic processing circuits. The load value in interval 3 is higher, and the data type conversion step can be performed by N basic processing circuits. . In this regard, it can be performed in an explicit manner. For example, the main processing circuit can be configured with a special instruction or instruction. When the basic processing circuit receives the special instruction or instruction, it is determined to execute the data type conversion step. If the basic processing circuit does not receive the special instruction or instruction When special instructions or instructions, make sure not to perform the data type conversion step. For another example, it can be performed in an implicit manner, for example, when the basic processing circuit receives data of a floating-point type and determines that an inner product operation needs to be performed, it converts the data type to data of a fixed-point type.

The following provides a method for computing by using the device shown in FIG. 1a. The computing method may be a neural network computing method, such as the forward operation of the neural network, the training of the neural network. In practical applications, the forward operation Operations such as matrix multiplication, convolution operations, activation operations, transformation operations, etc. may be performed according to different input data, and the above operations may be implemented by the device shown in FIG. 1a.

The data conversion operation circuit of the main processing circuit first converts the type of data and then transmits it to the basic processing circuit for operation by the control circuit. For example, the data conversion operation circuit of the main processing circuit can convert floating-point numbers into fixed-point numbers with lower bit width. It is then transmitted to the basic processing circuit, which has the advantage of being able to By reducing the bit width of the transmitted data and reducing the total number of bits transmitted, the basic processing circuit has higher efficiency and lower power consumption for performing position-wide fixed-point operations.

If the data received by the basic processing circuit is floating-point data, then the basic processing circuit can first convert the data type and then perform the calculation by the data conversion operation circuit after receiving the data. For example, the basic processing circuit receives the transmission from the main processing circuit. The floating-point number, the data conversion operation circuit is then converted into a fixed-point number, and then the inner product operator circuit, vector operator circuit or accumulator circuit of the basic processing circuit performs operations to improve operational efficiency and reduce power consumption.

After the basic processing circuit calculates the result, it can first perform data type conversion and then transmit it to the main processing circuit. For example, the floating-point operation result calculated by the basic processing circuit can be converted into a fixed-point number with a low bit width first and then transmitted to the main processing circuit. , the benefit is that the data bit width of the transmission process is reduced, the efficiency is higher, and the power consumption is saved.

The main processing circuit transmits the data to be calculated to all or part of the basic processing circuit; taking matrix multiplication by vector calculation as an example, the control circuit of the main processing circuit can split the matrix data into each column as a basic data, such as m*n The matrix can be split into n vectors of m rows, and the control circuit of the main processing circuit distributes the split vectors of n m rows to multiple basic processing circuits. For vectors, the control circuit of the main processing circuit can broadcast the vector as a whole to each of the base processing circuits. If the value of m is relatively large, then the control circuit can first split the m*n matrix into x*n vectors. Taking x=2 as an example, it can be split into 2n vectors, each of which contains m/ 2 lines, that is, each vector in the n vectors of m lines is divided into 2 vectors, taking the first line as an example, if the first vector of the n vectors of m lines is 1000 lines, then the dividing into 2 vectors can be , the first 500 lines are composed of the first vector, the last 500 lines are composed of the second vector, and the control circuit broadcasts the two vectors to multiple basic processing circuits through two broadcasts.

The data transmission method can be broadcast or distribution, or any other possible transmission method; after the basic processing circuit receives the data, it performs an operation to obtain the operation result; the basic processing circuit transmits the operation result back to the main processing circuit; the operation The result can be an intermediate operation result or a final operation result.

Use the device shown in Figure 1a to complete the operation of matrix multiplication by vector; (matrix multiplication by vector can be that each row in the matrix and the vector are respectively subjected to the inner product operation, and these results are arranged in the order of the corresponding rows into a vector.)

The following describes the operation of calculating the multiplication of a matrix S of size M rows and L columns and a vector P of length L, as shown in Figure 2a below, (each row in the matrix S has the same length as the vector P, and the data in them is sorted by position one One corresponding) The neural network computing device has K basic processing circuits: refer to FIG. 2, which provides a method for realizing matrix multiplication by vector, which may specifically include: Step S201, the data conversion operation circuit of the main processing circuit converts the matrix Each line of data in S is converted into fixed-point type data, and the control circuit of the main processing circuit distributes it to one of the K basic processing circuits, and the basic processing circuit saves the received distributed data in the on-chip cache of the basic processing circuit and / or register; in an optional solution, if the number of rows M<=K of the matrix S, the control circuit of the main processing circuit distributes a row of the matrix S to the K basic processing circuits respectively; in an optional solution, If the number of rows of the matrix S is M>K, the control circuit of the main processing circuit distributes data of one or more rows in the matrix S to each basic processing circuit.

The set of rows in S distributed to the i-th basic processing circuit is Ai, and there are Mi rows in total. Figure 2c shows the computation to be performed on the i-th basic processing circuit.

In an optional solution, in each basic processing circuit, such as the ith basic processing circuit, the received distribution data, such as the matrix Ai, may be stored in a register and/or an on-chip cache of the ith basic processing circuit ; The advantage is that the data transmission amount of the distributed data is reduced, the calculation efficiency is improved, and the power consumption is reduced.

Step S202, the data type operation circuit of the main processing circuit converts the vector P into fixed-point type data, and the control circuit of the main processing circuit transmits each part of the fixed-point type vector P to K basic processing circuits by broadcasting; In an alternative solution, the control circuit of the main processing circuit can broadcast each part of the vector P only once to the registers or the on-chip cache of each basic processing circuit, and the i-th basic processing circuit can fully perform the data of the vector P obtained this time. Ground multiplexing to complete the inner product operation corresponding to each row in the matrix Ai. The advantage is that the data transmission amount of the repeated transmission of the vector P from the main processing circuit to the basic processing circuit is reduced, the execution efficiency is improved, and the transmission power consumption is reduced.

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the vector P to the registers or the on-chip cache of each basic processing circuit for multiple times, and the i-th basic processing circuit can evaluate the data of the vector P obtained each time. No multiplexing is performed, and the inner product operation corresponding to each row in the matrix Ai is completed in stages; the advantage is that the data transmission amount of the vector P transmitted in a single transmission inside the basic processing circuit can be reduced, and the basic processing circuit cache and/or can be reduced. Or the capacity of the register, improve the execution efficiency, reduce the transmission power consumption, and reduce the cost.

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the vector P to the registers or the on-chip cache of each basic processing circuit for multiple times, and the i-th basic processing circuit can evaluate the data of the vector P obtained each time. Partial multiplexing is performed to complete the Inner product operation; the advantage is that it reduces the amount of data transmission from the main processing circuit to the basic processing circuit, and also reduces the amount of data transmission inside the basic processing circuit, improves execution efficiency, and reduces transmission power consumption.

Step S203, the inner product operator circuit of the K basic processing circuits calculates the inner product of the data of the matrix S and the vector P, for example, the ith basic processing circuit calculates the inner product of the data of the matrix Ai and the data of the vector P; Step S204 , the accumulator circuits of the K basic processing circuits accumulate the results of the inner product operation to obtain the accumulated result, and transmit the accumulated result back to the main processing circuit in the form of fixed-point type.

In an optional solution, the partial sum (the partial sum is a part of the accumulated result) obtained by each time the basic processing circuit performs the inner product operation, for example, the accumulated result is: F1*G1+F2*G2+F3*G3+F4* G4+F5*G5, then the partial sum can be: the value of F1*G1+F2*G2+F3*G3) is transmitted back to the main processing circuit for accumulation; the advantage is that it reduces the amount of operations inside the basic processing circuit and improves the basic processing The operational efficiency of the circuit.

In an optional solution, the partial sum obtained by the inner product operation performed by the basic processing circuit each time can also be stored in the register and/or the on-chip cache of the basic processing circuit, and transferred back to the main processing circuit after the accumulation is completed; the advantage is that, The data transmission amount between the basic processing circuit and the main processing circuit is reduced, the operation efficiency is improved, and the data transmission power consumption is reduced.

In an optional solution, the partial sum obtained by the inner product operation performed by the basic processing circuit each time can also be stored in the register and/or on-chip cache of the basic processing circuit for accumulation in some cases, and transferred to the main The processing circuit performs accumulation, and after the accumulation is completed, it is transmitted back to the main processing circuit; the advantage is that the amount of data transmission between the basic processing circuit and the main processing circuit is reduced, the operation efficiency is improved, the power consumption of data transmission is reduced, and the basic processing circuit is reduced. The amount of internal operation improves the operation efficiency of the basic processing circuit.

Referring to Figure 2b, the device as shown in Figure 1a is used to complete the operation of matrix multiplication by matrix; the following describes the operation of the multiplication of the matrix S of M rows and L columns and the size of the matrix W of L rows and N columns, (in the matrix S Each row of the matrix W has the same length as each column of the matrix W, as shown in Figure 2d) The neural network computing device has K basic processing circuits: step S201b, the control circuit of the main processing circuit distributes the data of each row in the matrix S to On one of the K basic processing circuits, the basic processing circuit saves the received data in the on-chip cache and/or register; in an optional solution, if the number of rows of S M<=K, the main processing circuit The control circuit of the main processing circuit distributes a row of the S matrix to the M basic processing circuits; in an optional solution, if the number of rows of S M>K, the control circuit of the main processing circuit distributes a row of the S matrix to each basic processing circuit. or multiple rows of data.

There are Mi rows in S that are distributed to the i-th basic processing circuit, and the set of Mi rows is called Ai. Figure 2e shows the computation to be performed on the i-th basic processing circuit.

In an optional solution, in each basic processing circuit, such as the ith basic processing circuit: the received matrix Ai distributed by the main processing circuit is stored in the ith basic processing circuit register and/or In the on-chip cache; the advantage is that the amount of subsequent data transmission is reduced, the computing efficiency is improved, and the power consumption is reduced.

In step S202b, the control circuit of the main processing circuit transmits each part in the matrix W to each basic processing circuit by broadcasting; in an optional solution, each part in the matrix W can be broadcast only once to the registers of each basic processing circuit. Or in the on-chip cache, the i-th basic processing circuit is responsible for the moment obtained this time. The data of the matrix W is fully multiplexed, and the inner product operation corresponding to each row in the matrix Ai is completed; the multiplexing in this embodiment can be used repeatedly in the calculation for the basic processing circuit, for example, the multiplexing of the data of the matrix W , which can be used multiple times for the data of matrix W.

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the matrix W to the registers or the on-chip cache of each basic processing circuit for multiple times, and the i-th basic processing circuit can evaluate the data of the matrix W obtained each time by the i-th basic processing circuit. No multiplexing is performed, and the inner product operation corresponding to each row in the matrix Ai is completed in stages; in an optional solution, the control circuit of the main processing circuit can broadcast each part in the matrix W to the basic processing circuit multiple times. In the register or on-chip cache, the i-th basic processing circuit partially multiplexes the data of the matrix W obtained each time, and completes the inner product operation corresponding to each row in the matrix Ai; The processing circuit, such as the ith basic processing circuit, calculates the inner product of the data of the matrix Ai and the data of the matrix W; step S203b, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit.

In an optional solution, the basic processing circuit may transmit the partial sums obtained by performing the inner product operation each time back to the main processing circuit for accumulation; in an optional solution, the inner product operation performed by the basic processing circuit each time may also be The obtained partial sum is stored in the register and/or on-chip cache of the basic processing circuit, and is transmitted back to the main processing circuit after the accumulation; and in some cases stored in registers and/or on-chip caches of the underlying processing circuit. Line accumulation, in some cases, it is transmitted to the main processing circuit for accumulation, and then transmitted back to the main processing circuit after the accumulation; referring to Figure 3a, the device shown in Figure 1a is used to complete the fully connected operation: if the input data of the fully connected layer is a vector (that is, when the input of the neural network is a single sample), the weight matrix of the fully connected layer is used as the matrix S, the input vector is used as the vector P, and the matrix multiplication vector shown in Figure 2 is performed according to the method of using the device 1. operation; if the input data of the fully connected layer is a matrix (that is, the input of the neural network is a case where multiple samples are used as batches), then the weight matrix of the fully connected layer is used as the matrix S, and the input vector is used as the matrix W, or with The weight matrix of the fully connected layer is taken as the matrix W, and the input vector is taken as the matrix S. According to the use of the device, the operation of the matrix multiplied by the matrix shown in Figure 2c is used; referring to Figure 3b, the volume is completed using the device shown in Figure 1a. Product operation: for a convolution layer, record the number of its convolution kernels as M; in step S301, the control circuit of the main processing circuit distributes the weight of each convolution kernel in the weights of the convolution layer to the K basic processing circuits On a certain one, it is stored in the on-chip cache and/or register of the basic processing circuit; in an optional solution, if the number of convolution kernels M<=K, the control circuit of the main processing circuit gives M basic processing The circuit distributes the weights of a convolution kernel respectively; in an optional solution, if the number of convolution kernels M>K, the control circuit of the main processing circuit distributes one or more convolution kernels to each basic processing circuit. weight value.

A total of Mi convolution kernels are distributed to the i-th basic processing circuit, and the set of these convolution kernel weights is called Ai.

In an optional solution, in each basic processing circuit, such as the ith basic processing circuit: the received convolution kernel weights Ai distributed by the main processing circuit are stored in its register and/or on-chip cache Step S302, the control circuit of the main processing circuit transmits each part in the input data T to each basic processing circuit by broadcasting; In a kind of optional scheme, the control circuit of the main processing circuit Broadcast once to the registers or on-chip cache of each basic processing circuit, the i-th basic processing circuit fully multiplexes the data of the input data T obtained this time, and completes the inner product operation corresponding to each convolution kernel in Ai ; In an optional scheme, the control circuit of the main processing circuit can broadcast each part of the input data T to the registers or the on-chip cache of each basic processing circuit for many times, and the i-th basic processing circuit is responsible for the input data obtained each time. The data of T is not multiplexed, and the inner product operation corresponding to each convolution kernel in Ai is completed in stages; in an optional solution, the control circuit of the main processing circuit can broadcast each part of the input data T multiple times. In the register or on-chip cache of each basic processing circuit, the i-th basic processing circuit performs partial multiplexing on the data of the input data T obtained each time, and completes the inner product operation corresponding to each convolution kernel in Ai; step S303, each basic processing circuit calculates the data inner product of the convolution kernel and the input data T, such as the i-th basic processing circuit, calculates the inner product of each convolution kernel of Ai and the data of the input data T; Step S304, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit: Transfer back to the main processing circuit for accumulation; in an optional solution, the basic processing circuit can also save the partial sum obtained by each inner product operation performed in the register and/or on-chip cache of the basic processing circuit, and transmit after the accumulation. back to the main processing circuit; in an optional solution, the basic processing circuit can also store the partial sums obtained by each inner product operation performed in some cases in the registers and/or on-chip cache of the basic processing circuit for accumulation, and some In this case, it is transmitted to the main processing circuit for accumulation, and after the accumulation is completed, it is transmitted back to the main processing circuit; the method of updating the weights using the device shown in Figure 1a: using the vector operator circuit of the main processing circuit to realize the weights in the neural network training process. The value update function, specifically, the weight update refers to a method of updating the weight using the gradient of the weight.

In an optional solution, the vector operator circuit of the main processing circuit is used to perform addition and subtraction operations on the two vectors of the weight value and the weight value gradient to obtain an operation result, and the operation result is the updated weight value.

In an optional solution, the vector operator circuit of the main processing circuit is used to multiply or divide the weight and the gradient of the weight by a number to obtain the intermediate weight and the gradient of the intermediate weight, and the vector operator circuit compares the intermediate weight and the gradient of the intermediate weight. The intermediate weight gradient value is added and subtracted to obtain the operation result, and the operation result is the updated weight value.

In an optional solution, a set of momentums can be calculated first using the gradient of the weights, and then the updated weights can be obtained by adding and subtracting the momentum and the weights; A method for implementing the reverse operation of the fully connected layer using the device shown in Figure 1a

The reverse operation of the fully connected layer can be divided into two parts, as shown in Figure 4a below. The solid line arrow represents the forward calculation process of the fully connected layer, as shown in Figure 4b, which represents the reverse calculation process of the fully connected layer.

The reverse operation of the fully connected layer shown in Fig. 4a and Fig. 4b can be accomplished by using the device shown in Fig. 1a and the matrix-by-matrix method shown in Fig. 2b; Reverse operation; the reverse operation of the convolutional layer can be divided into two parts, as shown in Figure 5a below, the solid line arrow represents the forward calculation process of the convolutional layer, as shown in Figure 5b, represents the reverse calculation process of the convolutional layer.

For the reverse operation of the convolution layer shown in FIG. 5a and FIG. 5b, the reverse operation of the convolution layer can be completed by using the device shown in FIG. 1a and the method shown in FIG. 3b.

A method for implementing the BLAS (Basic Linear Algebra Subprograms) function using the device shown in Figure 1a

GEMM calculation refers to: the operation of matrix-matrix multiplication in the BLAS library. The usual representation of this operation is: C=alpha*op(S)*op(P)+beta*C, where S and P are the two input matrices, C is the output matrix, alpha and beta are scalars, and op Represents a certain operation on the matrix S or P, in addition, there will be some auxiliary integers as parameters to describe the width and height of S and P of the matrix; the steps of using the device as shown in Figure 1a to realize the GEMM calculation include: the main processing circuit The data type conversion operation circuit can perform data type conversion on the matrix S and the matrix W; the conversion circuit of the main processing circuit performs respective corresponding op operations on the input matrix S and the matrix W; In an optional solution, op can be the transpose operation of the matrix; the matrix transpose operation can be realized by using the matrix transpose circuit of the main processing circuit; in an optional solution, after the OP of the matrix S and the matrix W is executed After the operation, the data type conversion operation can also be performed by the data conversion operation circuit of the main processing circuit, that is, the data conversion operation circuit converts the data types of op(S) and op(P) from floating-point type data to fixed-point type data, and then Perform a matrix multiplication operation as shown in Figure 2b.

In an optional solution, the op of a certain matrix can be empty, and the op operation is not performed; the op(S) and op are completed by using the device shown in FIG. 1a using the matrix multiplication matrix calculation method as shown in FIG. 2b Matrix multiplication calculation between (P); use the arithmetic logic unit of the main processing circuit to multiply each value in the result of op(S)*op(P) by alpha; in an alternative solution, alpha In the case of 1, the operation of multiplying by alpha is not performed; the arithmetic logic unit of the main processing circuit is used to realize the operation of beta*C; in an optional solution, when beta is 1, the operation of multiplying by beta is not performed; The vector operator circuit of the main processing circuit is used to realize the step of adding the corresponding positions between the matrices alpha*op(s)*op(P) and beta*C to obtain the result of the GEMM calculation.

In an optional solution, when beta is 0, this step is not performed; GEMV calculation refers to the operation of matrix-vector multiplication in the BLAS library. The usual representation of this operation is: C=alpha*op(S)*P+beta*C, where S is the input matrix, P is the input vector, C is the output vector, alpha and beta are scalars, and op represents the pair A certain operation of matrix S; the steps to implement GEMV calculation using the device as shown in Figure 1a are: The data type conversion operation circuit of the main processing circuit can perform data type conversion on the input matrix S and the matrix W; the conversion circuit of the main processing circuit performs the corresponding op operation on the input matrix S; Transpose operation; use the conversion circuit of the main processing circuit to realize the matrix transposition operation; in an optional scheme, the op of a certain matrix can be empty, and the transposition operation is not performed; use the device shown in Figure 1a to use Figure 2a The calculation method of the matrix multiplication vector described in completes the matrix-vector multiplication calculation between the matrix op(S) and the vector P; utilizes the arithmetic logic unit of the main processing circuit to carry out each value in the result of op(S)*P. The operation of multiplying by alpha; in an optional solution, the operation of multiplying by alpha is not performed when alpha is 1; the operation of beta*C is realized by the arithmetic logic unit of the main processing circuit; in an optional solution, beta In the case of 1, the operation of multiplying beta is not performed; the vector operator circuit of the main processing circuit is used to realize the step of adding the corresponding positions between the matrices alpha*op(S)*P and beta*C to obtain the result of GEMV.

In an optional solution, when beta is 0, the step operation of addition is not performed; the activation function of the device as shown in Figure 1a is used to input a vector using the activation circuit of the main processing circuit, and the activation of the vector is calculated. vector; in an optional solution, the main processing circuit activation circuit passes each value in the input vector through an activation function (the input of the activation function is a numerical value, and the output is also a numerical value), and calculates a value that is output to the output vector. corresponding position; In an alternative, the activation function can be: y=max(m,x), where x is the input value, y is the output value, and m is a constant; in an alternative, the activation function can be: y =tanh(x), where x is the input value and y is the output value; in an alternative, the activation function can be: y=sigmoid(x), where x is the input value and y is the output value; in an alternative In an alternative scheme, the activation function can be a piecewise linear function; in an alternative scheme, the activation function can be any function that inputs a number and outputs a number.

In an optional solution, the sources of the input vector include (including but not limited to): an external data source of the device; in an optional solution, the input data comes from the operation result of the device performing matrix multiplication with a vector; in a In an optional solution, the input data comes from the operation result of the device performing matrix multiplication; the main processing circuit of the device calculates the result; in an optional solution, the input data comes from the main processing circuit of the device after adding bias calculation result.

It should be noted that the above activation operation can be implemented by an arithmetic logic circuit and an accumulator circuit in the main processing circuit, or an activation circuit can be separately added to the main processing circuit to implement the activation operation.

The biasing operation is achieved using the setup shown in Figure 1a: Using the vector operator circuit of the main processing circuit can realize the function of adding two vectors or two matrices; using the vector operator circuit of the main processing circuit can realize adding a vector to each row or each column of a matrix function on.

In an alternative, the matrix may come from the result of a matrix-multiply-matrix operation performed by the apparatus; in an alternative, the matrix may come from a result of a matrix-multiplied-vector operation performed by the apparatus; in an alternative In the scheme, the matrix may be from data received from the outside by the main processing circuit of the device.

In an alternative, the vector may come from data received externally by the main processing circuit of the device.

Including but not limited to the above data sources.

Use the device as shown in Figure 1a to realize data type conversion: use the data type conversion operation circuit of the main processing circuit to realize the conversion of data types; Data type conversion; in an optional solution, the form of data type conversion includes but is not limited to: floating-point number to fixed-point number and fixed-point number to floating-point number, etc.; the present invention also provides a chip, the chip includes a computing device, the The computing device includes: including a main processing circuit, the data involved in the main processing circuit can be data of any data type, in an optional solution, it can be data represented by floating-point numbers of any bit width. It can be data represented by a fixed-point number of any bit width; all the arithmetic circuits and storage circuits involved can be arithmetic circuits and storage circuits of any data type. In an optional solution, it can be a floating point number of any bit width. The arithmetic circuit and storage circuit can also be a fixed-point arithmetic circuit and storage circuit of any bit width.

In an optional solution, the main processing circuit includes a data type conversion operation circuit; in an optional solution, the main processing circuit includes a vector operation unit that performs data type conversion; specifically, includes a data input interface for receiving input data; In an optional solution, the source of the received data may be: the outside of the neural network computing circuit device or part or all of the basic processing circuits of the neural network computing circuit device; in an optional solution, the data There may be multiple input interfaces; specifically, it may include a data output interface for outputting data; in an optional solution, the destination of the output data may be: the outside of the neural network computing device or the neural network computing device Part or all of the basic processing circuits of the circuit device; in an optional solution, there may be multiple data output interfaces; in an optional solution, the main processing circuit includes on-chip caches and/or registers; in an optional solution In an alternative, the main processing circuit includes an arithmetic unit that can perform data operations; in an alternative, the main processing circuit includes an arithmetic operation unit; in an alternative, the main processing circuit Contains a vector operation unit, which can perform operations on a group of data at the same time; specifically, the arithmetic operations and/or vector operations can be any Types of operations, including but not limited to: addition, subtraction, multiplication and division of two numbers, addition, subtraction, multiplication and division of a number and a constant, performing exponential operations on a number, exponentiation, logarithmic operations, and various nonlinear operations, performing comparison operations on two numbers , logical operations, etc. Two vectors are added, subtracted, multiplied and divided. Each element in a vector is added, subtracted, multiplied and divided with a constant. Exponential operations, power operations, logarithmic operations, and various nonlinear operations are performed on each element in the vector. Two corresponding elements perform comparison operations, logical operations, etc.

In an optional solution, the main processing circuit includes a data rearrangement unit for transmitting data to the basic processing circuit in a certain order, or rearranging the data in a certain order in place; in an optional solution, all The order of data arrangement includes: transforming a multidimensional data block in the order of dimensions; the order of data arrangement may further include: dividing a data block into blocks to send to different basic processing circuits.

The computing device also includes a plurality of basic processing circuits: each basic processing circuit is used to calculate the inner product of two vectors, and the calculation method is that the elements in the two groups of numbers are corresponding to the two groups of numbers received by the basic processing circuit. Multiply, and accumulate the results of the multiplication; the result of the inner product is transmitted, and it is transmitted here. According to the position of the basic processing circuit, it may be transmitted to other basic processing circuits, or it can be directly transmitted to the main processing circuit.

The data involved in the basic processing circuit can be data of any data type. In an optional solution, it can be data represented by floating-point numbers of any bit width or data represented by fixed-point numbers of any bit width; All arithmetic circuits and storage circuits can be arithmetic circuits and storage circuits of any data type. In an optional solution, they can be arithmetic circuits and storage circuits of floating-point numbers of any bit width, or they can be of any bit width. Fixed-point arithmetic and storage circuits. In an optional solution, the basic processing circuit includes a data type conversion operation circuit; in an optional solution, the basic processing circuit includes a vector operation unit that performs data type conversion; storage unit; specifically, including one or more data input interfaces for receiving data; in an optional solution, including two data input interfaces, one or more data can be obtained from the two data input interfaces each time; In an optional solution, the basic processing circuit may receive input data from the data input interface and store it in a register and/or an on-chip cache; the source of the data received by the data input interface may be: other basic processing circuits and/or main processing circuit.

The main processing circuit of the neural network arithmetic circuit device; other basic processing circuits of the neural network arithmetic circuit device (the neural network arithmetic circuit device has a plurality of basic processing circuits); specifically, including one or more transmission outputs Data output interface for data; in an optional solution, one or more data can be transmitted from the data output interface; specifically, the data transmitted through the data output interface can be: data received from the data input interface, One or any combination of the data stored in the on-chip cache and/or registers, the result of the multiplier operation, the result of the accumulator operation, or the result of the inner product operator.

In an optional solution, three data output interfaces are included, two of which correspond to two data input interfaces respectively, each layer outputs the data received from the data input interface on the previous layer, and the third data output interface is responsible for the output operation As a result; specifically, the destination of the data transmitted by the data output interface may be: the above data source and the data destination here determine the connection relationship of the basic processing circuit in the device.

The main processing circuit of the neural network operation circuit device; other basic processing circuits of the neural network operation circuit device, the neural network operation circuit device has a plurality of basic processing circuits; The circuit may specifically be: one or more multiplier circuits, one or more accumulator circuits, one or more circuits for performing inner product operation of two groups of numbers, or any combination thereof.

In an alternative, a multiplication of two numbers can be performed, and the result can be stored in on-chip caches and/or registers, or directly accumulated in registers and/or on-chip caches; in an alternative, it can be Perform the inner product operation of two sets of data, and the result can be stored in the on-chip cache and/or register, or can be directly accumulated into the register and/or on-chip cache; The data is accumulated in the on-chip cache and/or register; specifically, the data accumulated by the accumulator circuit can be: data received from the data input interface, data stored in the on-chip cache and/or register, multiplier operation results, One or any combination of the operation result of the accumulator and the operation result of the inner product operator.

It should be noted that the "data input interface" and "data output interface" used in the above description of the basic processing circuit refer to the data input and output interface of each basic processing circuit, rather than the data input and output of the entire device. interface.

The present disclosure also discloses a neural network computing device, which includes one or more chips as shown in FIG. 1a or FIG. 1b, for obtaining data to be operated and control information from other processing devices, and executing a specified neural network Operation, the execution result is passed to the peripheral device through the I/O interface. Peripherals such as camera, monitor, mouse, keyboard, network card, wifi interface, server. When more than one chip as shown in Figure 1a or Figure 1b is included, the chips shown in Figure 1a or Figure 1b can be linked and transmitted through a specific structure, for example, interconnected and transmitted through the PCIE bus data to support larger-scale neural network operations. At this time, the same control system can be shared, or there can be independent control systems; memory can be shared, or each accelerator can have its own memory. In addition, the interconnection method can be any interconnection topology.

The neural network computing device has high compatibility and can be connected with various types of servers through the PCIE interface.

The present disclosure also discloses a combined processing device, which includes the above-mentioned neural network computing device, a universal interconnection interface, and other processing devices (ie, general-purpose processing devices). The neural network computing device interacts with other processing devices to jointly complete the operation specified by the user. Figure 4c below is a schematic diagram of the combined processing device.

Other processing devices include one or more processor types among general-purpose/special-purpose processors such as a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processor. The number of processors included in other processing devices is not limited. Other processing devices serve as the interface between the neural network computing device and external data and control, including data transfer, to complete the start and stop of the neural network computing device and other basic controls; other processing devices can also cooperate with neural network computing devices to complete computing tasks.

The universal interconnection interface is used to transmit data and control instructions between the neural network computing device and other processing devices. The neural network computing device obtains the required input data from other processing devices, and writes it into the memory device on-chip of the neural network computing device; it can obtain control instructions from other processing devices and write it into the control cache on the neural network computing device chip; The data in the memory module of the neural network computing device can be read and transmitted to other processing devices.

As shown in FIG. 4d, optionally, the structure further includes a storage device for storing the data required by the operation unit/operation device or other operation units, especially suitable for the data required for operation in the neural network operation device. or other data that cannot be fully stored in the internal storage of the processing device.

The combined processing device can be used as an SOC system for mobile phones, robots, drones, video surveillance equipment and other equipment, effectively reducing the core area of the control part, improving the processing speed and reducing the overall power consumption. In this case, the general interconnection interface of the combined processing device is connected to some components of the device. Some components such as camera, monitor, mouse, keyboard, network card, wifi interface.

Embodiments of the present disclosure provide a neural network processor board that can be used in numerous general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, smart homes, appliances, multiprocessor systems, microprocessor-based systems, robotics, programmable consumer electronics, network personal computers , PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

Please refer to FIG. 5c, which is a schematic structural diagram of a neural network processor board according to an embodiment of the present disclosure. As shown in FIG. 5 c , the above-mentioned neural network processor board 10 includes a neural network chip package structure 11 , a first electrical and non-electrical connection device 12 and a first substrate 13 .

The present disclosure does not limit the specific structure of the neural network chip packaging structure 11. Optionally, as shown in FIG. 5d, the neural network chip packaging structure 11 includes: a neural network chip 111, a second electrical and non-electrical connection device 112, a first Two substrates 113 .

The specific form of the neural network chip 111 involved in the present disclosure is not limited. The above-mentioned neural network chip 111 includes but is not limited to a neural network chip that integrates a neural network processor. The above-mentioned chip can be made of silicon material, germanium material, quantum material or molecular material. materials etc. According to the actual situation (for example: harsh environment) and different application requirements, the above neural network chip can be packaged, so that most of the neural network chip is wrapped, and the pins on the neural network chip are passed through gold wires. The other conductors are connected to the outside of the package structure for circuit connection with the outer layers.

The present disclosure does not limit the specific structure of the neural network chip 111. Optionally, please refer to the device shown in FIG. 1a or FIG. 1b.

The present disclosure does not limit the types of the first substrate 13 and the second substrate 113, which may be a printed circuit board (PCB) or a printed wiring board (PWB), and may also be other circuit boards. The material for making the PCB is also not limited.

The second substrate 113 involved in the present disclosure is used to carry the above-mentioned neural network chip 111 , and the neural network chip package structure 11 is obtained by connecting the above-mentioned neural network chip 111 and the second substrate 113 through the second electrical and non-electrical connection device 112 . , which is used to protect the neural network chip 111 and facilitate further packaging of the neural network chip packaging structure 11 and the first substrate 13 .

The packaging method and the structure corresponding to the packaging method of the above-mentioned specific second electrical and non-electrical connection device 112 are not limited, and an appropriate packaging method can be selected according to the actual situation and different application requirements and can be simply improved, for example: flip chip Ball Grid Array Package (Flip Chip Ball Grid Array Package, FCBGAP), Low-profile Quad Flat Package (LQFP), Quad Flat Package with Heat Sink (HQFP), None Pin quad flat package (Quad Flat Non-lead Package, QFN) or small pitch quad flat package (Fine-pitch Ball Grid Package, FBGA) and other packaging methods.

Flip Chip is suitable for situations where the area after the package is high or is sensitive to the inductance of the wire and the transmission time of the signal. In addition, a wire bonding (Wire Bonding) packaging method can be used to reduce the cost and improve the flexibility of the packaging structure.

Ball Grid Array (Ball Grid Array) can provide more pins, and the average lead length of the pins is short, which has the function of high-speed signal transmission. Among them, the package can be packaged with Pin Grid Array (PGA) , zero insertion force (Zero Insertion Force, ZIF), single edge contact connection (Single Edge Contact Connection, SECC), contact array (Land Grid Array, LGA), etc. instead.

Optionally, the neural network chip 111 and the second substrate 113 are packaged in a flip chip ball grid array (Flip Chip Ball Grid Array) packaging method. Refer to FIG. 6 for a schematic diagram of a specific neural network chip packaging structure. As shown in FIG. 6 , the above-mentioned neural network chip package structure includes: a neural network chip 21 , pads 22 , solder balls 23 , a second substrate 24 , connection points 25 and pins 26 on the second substrate 24 .

Wherein, the pad 22 is connected to the neural network chip 21, and solder balls 23 are formed by welding between the pad 22 and the connection point 25 on the second substrate 24, and the neural network chip 21 and the second substrate 24 are connected. Packaging of the neural network chip 21 .

The pin 26 is used to connect with the external circuit of the package structure (for example, the first substrate 13 on the neural network processor board 10 ), which can realize the transmission of external data and internal data, and is convenient for the neural network chip 21 or the neural network chip 21 The corresponding neural network processor processes the data. The present disclosure also does not limit the type and quantity of pins, and different pin forms can be selected according to different packaging technologies, and are arranged in accordance with certain rules.

Optionally, the above-mentioned neural network chip package structure further includes insulating fillers, which are placed in the gaps between the pads 22 , the solder balls 23 and the connection points 25 , to prevent interference between the solder balls and the solder balls.

Wherein, the material of the insulating filler may be silicon nitride, silicon oxide or silicon oxynitride; the interference includes electromagnetic interference, inductive interference, and the like.

Optionally, the above-mentioned neural network chip packaging structure further includes a heat dissipation device for dissipating heat during operation of the neural network chip 21 . Wherein, the heat dissipation device may be a metal sheet, a heat sink or a heat sink with good thermal conductivity, such as a fan.

For example, as shown in FIG. 6a, the neural network chip package structure 11 includes: a neural network chip 21, pads 22, solder balls 23, a second substrate 24, connection points 25 on the second substrate 24, pins 26, Insulation filler 27 , thermal paste 28 and metal shell heat sink 29 . Among them, the heat dissipation paste 28 and the metal shell heat dissipation fins 29 are used to dissipate the heat of the neural network chip 21 during operation.

Optionally, the above-mentioned neural network chip package structure 11 further includes a reinforcing structure, which is connected to the pads 22 and embedded in the solder balls 23 to enhance the connection strength between the solder balls 23 and the pads 22 .

Wherein, the reinforcing structure may be a metal wire structure or a columnar structure, which is not limited herein.

The present disclosure also does not limit the specific form of the first electrical and non-electrical device 12. Reference can be made to the description of the second electrical and non-electrical device 112, that is, the neural network chip package structure 11 is packaged by welding, or a connection can be used. The way of connecting the second substrate 113 and the first substrate 13 by wire connection or plugging is convenient for subsequent replacement of the first substrate 13 or the neural network chip package structure 11 .

Optionally, the first substrate 13 includes an interface and the like for a memory unit used to expand the storage capacity, for example: a synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), a double-rate synchronous dynamic random access memory (Double Date Rate SDRAM, DDR), etc., the processing power of the neural network processor is improved by expanding the memory.

The first substrate 13 may further include a Peripheral Component Interconnect-Express (PCI-E or PCIe) interface, a Small Form-factor Pluggable (SFP) interface, an Ethernet interface, A controller area network bus (Controller Area Network, CAN) interface, etc., is used for data transmission between the package structure and the external circuit, which can improve the operation speed and the convenience of operation.

The neural network processor is packaged into a neural network chip 111, the neural network chip 111 is packaged into a neural network chip package structure 11, and the neural network chip package structure 11 is packaged into a neural network processor board 10, through the interface ( socket or ferrule) for data interaction with an external circuit (eg, computer motherboard), that is, directly using the neural network processor board 10 to realize the function of the neural network processor and protect the neural network chip 111 . In addition, other modules can be added to the neural network processor board 10, which improves the application scope and operation efficiency of the neural network processor.

In one embodiment, the present disclosure discloses an electronic device including the above-mentioned neural network processor board 10 or neural network chip package structure 11 .

Electronic devices include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cameras, video cameras, projectors, watches, headphones, mobile storage , wearable devices, vehicles, home appliances, and/or medical devices.

The vehicles include airplanes, ships and/or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lamps, gas stoves, and range hoods; the medical equipment includes nuclear magnetic resonance device, B-ultrasound and/or electrocardiograph.

The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present disclosure in further detail. It should be understood that the above are only specific embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this disclosure shall be included within the protection scope of this disclosure.

Claims (15)

An integrated circuit chip device, wherein the integrated circuit chip device comprises: a main processing circuit, k branch circuits and k groups of basic processing circuits, the main processing circuit is respectively connected with the k branch circuits, among the k branch circuits Each branch circuit corresponds to a group of basic processing circuits in the k groups of basic processing circuits, and the group of basic processing circuits includes at least one basic processing circuit; the branch circuit includes: a data type operation circuit for executing floating-point type data conversion with fixed-point type data; the main processing circuit is used to perform each successive operation in the neural network operation and transmit data with the k branch circuits connected to it; the k branch circuits are used to perform in the main A transmission data is forwarded between the processing circuit and the k groups of basic processing circuits, and according to the operation control of the transmission data, whether to start the data type operation circuit to perform conversion of the type of the transmission data; the k groups of basic processing circuits are used for according to the transmission. The data or the converted transmission data execute the operation in the neural network in parallel, and transmit the operation result to the main processing circuit through the branch circuit connected with the main processing circuit; the main processing circuit is used to obtain a to-be-calculated A data block and an operation instruction, according to the operation instruction, the data block to be calculated is divided into a distribution data block and a broadcast data block; the distribution data block is divided and processed to obtain a plurality of basic data blocks, and the plurality of basic data blocks are obtained. The data block is distributed to the k branch circuits connected to it, and the broadcast data block is broadcast to the k branch circuits connected to it; the k branch circuits are used to receive the basic data block and the broadcast data block, and activate the The data type operation circuit converts the basic data block and the broadcast data block into a fixed-point data type; forwards the basic data block and the broadcast data block to the k groups of basic processing circuits in the fixed-point data type; The basic processing circuit is used to perform an inner product operation on the basic data block and the broadcast data block with fixed-point data type to obtain an operation result, and send the operation result to the k branch circuits; the k branch circuits are used for Convert the operation result into a floating-point type operation result, and send the floating-point type operation result to the main processing circuit; the main processing circuit is used for processing the floating-point type operation result to obtain the data to be calculated The block and the instruction result of the operation instruction. The integrated circuit chip device according to claim 1, wherein the main processing circuit is specifically configured to broadcast the broadcast data block to the k branch circuits at one time. The integrated circuit chip device according to claim 1, wherein the main processing circuit is specifically configured to divide the broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the k branch circuits. The integrated circuit chip device according to item 3 of the patent application scope, wherein the basic processing circuit is specifically configured to perform an inner product processing on the part of the broadcast data block and the basic data block in a fixed-point type to obtain an inner product processing result, Accumulate the inner product processing result to obtain a part of the operation result, send the partial operation result to the k branch circuits, and the k branch circuits are used to convert the partial operation result into floating-point type data and send it to the main processing circuit . The integrated circuit chip device according to claim 3, wherein the basic processing circuit is specifically configured to multiplex the part of the broadcast data block n times to perform the inner product of the part of the broadcast data block and the n basic data blocks in a fixed-point data type The operation obtains n partial processing results of the fixed-point data type. After accumulating the n partial processing results of the fixed-point data type respectively, the n partial operation results of the fixed-point type are obtained, and the n partial operation results of the fixed-point type are sent to the branch circuit. ; The branch circuit is used to convert the n partial operation results of the fixed point type into n partial operation results of the floating point type, and send the n partial operation results of the floating point type to the main processing circuit, where n is greater than or equal to An integer of 2. The integrated circuit chip device according to claim 1, wherein the main processing circuit comprises: a main register or a main on-chip buffer circuit; or the branch circuit comprises: a basic register or a basic on-chip buffer circuit; or the basic processing circuit comprises: Basic registers or basic on-chip cache circuits. The integrated circuit chip device according to claim 6, wherein the main processing circuit includes: a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, and a data type operation circuit Or one or any combination of data rearrangement circuits. The integrated circuit chip device according to claim 1, wherein the data is one or any combination of vectors, matrices, three-dimensional data blocks, four-dimensional data blocks, and n-dimensional data blocks. The integrated circuit chip device according to claim 1, wherein, if the operation command is a multiplication command, the main processing circuit determines that the multiplier data block is the broadcast data block, and the multiplicand data block is the distribution data block; such as The operation instruction is a convolution instruction, the main processing circuit determines that the input data block is the broadcast data block, and the convolution kernel is the distribution data block. A neural network computing device, wherein the neural network computing device includes one or more integrated circuit chip devices according to any one of items 1 to 9 of the patent application scope. A combined processing device, wherein the combined processing device comprises: a neural network computing device, a general interconnection interface and a general processing device as claimed in item 10 of the patent application scope; the neural network computing device communicates with the general purpose through the general interconnection interface Handling device connection. A chip for performing the operation of a neural network, wherein the chip integrates the device according to any one of items 1 to 9 of the scope of the application. A smart device, wherein the smart device includes the chip as claimed in item 12 of the patent application scope. A method for computing a neural network, wherein the method is applied in an integrated circuit chip device, the integrated circuit chip device comprising: the integrated circuit chip device according to any one of items 1 to 9 in the scope of the application, the integrated circuit chip device using for performing neural network operations. The method according to item 14 of the scope of the application, wherein the operation of the neural network includes: convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, bias operation, full connection operation, GEMM operation, GEMV operation, and activation operation. one or any combination.

Images