WO2025107800A1 - Ai processor, data processing method, and computer device - Google Patents

Abstract

The embodiments of the present application belong to the technical field of processors. Disclosed are an AI processor, a data processing method, and a computer device. The AI processor comprises: a first memory, which is used for storing first operation data and second operation data; a data transfer engine, which is used for reading first operation sub-data and second operation sub-data from the first memory on the basis of a bus width, and transferring by means of a bus the first operation sub-data and the second operation sub-data into a second memory in a matrix operation engine; and the matrix operation engine, which is used for performing, by means of a matrix operation unit, matrix operation on the first operation sub-data and the second operation sub-data which are in the second memory, so as to obtain sub-operation results, wherein the matrix operation engine is further used for accumulating the sub-operation results by means of an accumulator, so as to obtain a matrix operation result of the first operation data and the second operation data. By adopting the solution provided in the embodiments of the present application, the bandwidth utilization rate during data transfer is increased.

Description

AI processor, data processing method and computer device

This application claims priority to the Chinese patent application filed on November 22, 2023, with application number 202311579613.0 and invention name “AI processor, data processing method and computer device”, all contents of which are incorporated by reference into this application.

Technical Field

The embodiments of the present application relate to the field of processor technology, and in particular to an AI processor, a data processing method, and a computer device.

Background Art

The core calculations in deep learning algorithms mainly include convolution calculations and matrix calculations. Convolution calculations can be equivalent to matrix calculations. Therefore, by accelerating matrix calculations, deep learning algorithms can be accelerated.

In the related art, a matrix acceleration engine is set to increase the speed of matrix calculation, and input data is moved based on the array dimension of the systolic array in the matrix acceleration engine. Since the data dimension represented by the data format used by different input data often does not match the array dimension of the systolic array, when the data is moved through the bus, the data is moved directly according to the systolic array dimension, which often leads to the problem of low bus bandwidth utilization.

Summary of the invention

The embodiments of the present application provide an AI processor, a data processing method and a computer device. The technical solution is as follows:

On the one hand, an embodiment of the present application provides an AI processor, the AI processor comprising: a matrix operation engine, a data handling engine and a first memory, the matrix operation engine and the data handling engine being connected via a bus;

The first memory is used to store first operation data and second operation data, the first operation data and the second operation data are in a target data format, and the data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine;

The data transport engine is configured to read the first operator data and the second operator data from the first memory based on a bus bit width, and transport the first operator data and the second operator data to a second memory inside the matrix operation engine through the bus, wherein the bus bit width is greater than a data bit width corresponding to the array dimension;

The matrix operation engine is used to perform a matrix operation on the first operator data and the second operator data in the second memory through a matrix operation unit to obtain a sub-operation result, wherein the data dimension of the sub-operation result matches the array dimension;

The matrix operation engine is further used to accumulate the sub-operation results through an accumulator to obtain a matrix operation result of the first operation data and the second operation data.

On the other hand, an embodiment of the present application provides a data processing method, which is used for an AI processor, wherein the AI processor includes a matrix operation engine, a data handling engine, and a first memory, wherein the matrix operation engine is connected to the data handling engine via a bus; the method includes:

storing first operation data and second operation data by the first memory, wherein the first operation data and the second operation data are in a target data format, and the data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine;

Based on a bus bit width, reading first operator data and second operator data from the first memory through the data transfer engine, and transferring the first operator data and the second operator data to a second memory inside the matrix operation engine through the bus, wherein the bus bit width is greater than a data bit width corresponding to the array dimension;

Performing a matrix operation on the first operator data and the second operator data in the second memory through a matrix operation unit in the matrix operation engine to obtain a sub-operation result, wherein the data dimension of the sub-operation result matches the array dimension;

The sub-operation results are accumulated by an accumulator in the matrix operation engine to obtain a matrix operation result of the first operation data and the second operation data.

On the other hand, an embodiment of the present application provides a computer device, comprising a memory and an AI processor as described in the above aspects, wherein the memory stores at least one instruction, and the at least one instruction is used to be executed by the AI processor.

In an embodiment of the present application, by storing the first operation data and the second operation data in the target data format in the first memory, the data dimension of the first operation data and the second operation data matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine, so that the data handling engine can read the first operator data and the second operator data from the first memory according to the bus width, and carry the first operator data and the second operator data to the second memory inside the matrix operation engine through the bus, and then the matrix operation engine performs matrix operation on the first operator data and the second operator data through the matrix operation unit to obtain the sub-operation result, and accumulates the sub-operation result through the accumulator to obtain the matrix operation result of the first operation data and the second operation data. Using the AI processor provided in the embodiment of the present application, by storing the operation data in the target data format in the first memory, it is possible to realize the reading of operation data according to the bus width. Since the bus width is greater than the data width corresponding to the array dimension, compared with reading data according to the data width corresponding to the array dimension, the AI processor provided in the embodiment of the present application can improve the bandwidth utilization of data handling during matrix operation, thereby improving the efficiency of matrix operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a schematic diagram of the structure of an AI processor provided by an exemplary embodiment of the present application;

FIG2 shows a schematic diagram of a systolic array operation provided by an exemplary embodiment of the present application;

FIG3 is a schematic diagram showing the logical expression of convolutional network data and the physical storage form of data in different formats in the related art;

FIG4 is a schematic diagram showing a physical storage form of a first target data format provided by an exemplary embodiment of the present application;

FIG5 is a schematic diagram showing data processing by a systolic array based on a general data format in the related art;

FIG6 is a schematic diagram showing data processing by a systolic array based on a target data format provided by an exemplary embodiment of the present application;

FIG7 is a schematic diagram showing data processing by a systolic array based on a general data format in another related art;

FIG8 is a schematic diagram showing data processing by a systolic array based on a target data format provided by another exemplary embodiment of the present application;

FIG9 is a schematic diagram showing a data format conversion provided by an exemplary embodiment of the present application;

FIG10 shows a flow chart of a data processing method provided by an exemplary embodiment of the present application;

FIG. 11 shows a structural block diagram of a computer device provided by an exemplary embodiment of the present application.

DETAILED DESCRIPTION

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

It should be understood that the "several" mentioned in this article refers to one or more, and "multiple" refers to two or more. "And/or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

For ease of understanding, the nouns involved in the embodiments of the present application are explained below.

Operation data: refers to the data involved in the operation. The operation data in the embodiment of the present application is used for convolution operation or matrix multiplication operation. In the convolution operation process, the operation data includes the convolution input data and the convolution weight data, which can also be called the convolution kernel data; in the matrix multiplication operation process, the operation data includes the left matrix input data and the right matrix input data.

Pulse Sequence: An array in the Matrix Arithmetic Logic Unit (MALU) used to implement matrix multiplication operations. The core idea of the systolic array is to divide the matrix into blocks, and then complete the entire matrix multiplication through a series of data movement and local multiplication operations. The input of the systolic array includes left input and right input, and the number of rows of the systolic array is represented as MALU_K, and the number of columns of the array is represented as MALU_N.

Dimension: refers to the number of data in a certain dimension, such as the number of rows, columns, height, width, channels, etc. When the data format is HWC, if the data is 3×4×5, the dimension of the data in the H (height) dimension is 3, the dimension in the W (width) dimension is 4, and the dimension in the C (channel) dimension is 5.

Deep learning methods are widely used in various fields such as image processing, video processing, speech processing, content generation, etc., and the core characteristics of deep learning are large amount of calculation and large number of parameters. Deep learning networks can be divided into three categories: Convolutional Neural Networks (CNN), Transformer and ViT (Vision Transformer). The core calculations in these three categories of networks are mainly concentrated in convolution calculations and matrix calculations, and convolution calculations can be equivalent to matrix calculations. Therefore, the core of accelerating deep learning algorithms is to accelerate matrix operations, and how to perfectly match the core matrix calculations with other commonly used vector calculations is the key issue in designing AI processors.

Optionally, the AI processor can be a processor based on neural network algorithms and acceleration, such as a neural network processor (Neural network Processing Unit, NPU).

In the related art, the AI processor is integrated into a general deep learning framework, and matrix operations are performed through the matrix operation engine in the AI processor. Among them, the structure of the matrix operation unit in the matrix operation engine is generally in the form of a two-dimensional systolic array. Therefore, in order to use the matrix operation unit to perform data operations on the operation data, the related art generally carries out data transfer on the operation data according to the array dimension of the systolic array in the matrix operation unit. For a bus whose bus width is larger than the data width corresponding to the array dimension, the problem of insufficient bus bandwidth utilization often occurs during data transfer, resulting in low bus bandwidth utilization.

In an embodiment of the present application, by storing the first operation data and the second operation data in the target data format (the data dimension represented matches the array dimension of the systolic array in the matrix operation unit) in the first memory, the first operator data and the second operator data can be directly read from the first memory according to the bus bit width through the data transfer engine, and the first operator data and the second operator data can be transferred to the second memory inside the matrix operation engine through the bus, thereby fully improving the utilization rate of the bus bit width and optimizing the data transfer process.

Please refer to Figure 1, which shows a structural diagram of an AI processor provided by an exemplary embodiment of the present application. The AI processor 100 mainly includes a matrix operation engine 110, a data transfer engine 120 and a first memory 130, wherein the matrix operation engine 110 and the data transfer engine 120 are connected via a bus.

The first memory 130 is used to store the first operation data and the second operation data. The first operation data and the second operation data are in a target data format. The data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine.

In the related art, the AI processor stores the operation data of the dimensional expression method unique to the deep learning network through the first memory, resulting in insufficient bandwidth utilization in the process of data transfer through the data transfer engine. For example, in PyTorch, the data format of the convolutional network is NCHW, and the weight format is CoCiKhKw; in TensorFlow, the data format of the convolutional network is NHWC, and the weight format is KhKwCiCo. Among them, N is the number of batches, C is the number of data channels, H is the height, W is the width, Kh is the height of the convolution kernel, Kw is the width of the convolution kernel, Ci is the number of input channels, and Co is the number of output channels.

In the embodiment of the present application, the first operation data and the second operation data in the first memory are both stored in the target data format. The original data format of the first operation data and the second operation data can be the target data format, or the first operation data and the second operation data in the original data format are format converted and stored in the first memory in the target data format.

Optionally, the architecture of the matrix operation engine is a multi-dimensional systolic array consisting of multiple physical matrices of multiply accumulate (MAC) operations, which is used to perform computational processing on a series of matrix operations of convolutional neural networks.

Optionally, the data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine, which may mean that the data dimension of each data channel represented by the target data format is equal to the array dimension of the systolic array, or that the data dimension of some data channels among the data channels represented by the target data format is equal to the array dimension of the systolic array.

Optionally, the array dimension of the systolic array may be the number of array rows or the number of array columns. When the operation data is convolution data, the data dimension represented by the target data format may be the number of data channels; when the operation data is matrix data, the data dimension represented by the target data format may be the number of matrix rows or the number of matrix columns.

For example, the number of data channels represented by the target data format is equal to the array dimension of the systolic array, or the number of data rows or columns represented by the target data format is equal to the number of array rows or columns of the systolic array.

Illustratively, the first operation data is matrix data, the number of matrix rows and the number of matrix columns of the matrix data are both 128, and the number of array rows of the systolic array is 64. Then, when the target data format is adopted, the number of matrix rows of the first operation data can be converted to 2×64, so that the number of matrix rows of the first operation data is equal to the number of array rows of the systolic array.

Optionally, the systolic array is in the form of a two-dimensional matrix, which may be a two-dimensional systolic, a one-dimensional systolic plus a one-dimensional broadcast, or a two-dimensional systolic and broadcast mixed structure.

The data transfer engine 120 is used to read the first operator data and the second operator data from the first memory 130 based on the bus bit width, and transfer the first operator data and the second operator data to the second memory 111 inside the matrix operation engine 110 through the bus, and the bus bit width is greater than the data bit width corresponding to the array dimension.

In some embodiments, the bus width is an integer multiple of the data width corresponding to the array dimension. For example, the bus width is twice the data width corresponding to the array dimension. Of course, the bus width may not be an integer multiple of the data width corresponding to the array dimension. The following embodiments are only illustrated by taking integer multiples as an example, but are not limited thereto.

Optionally, the data transfer engine may be a direct memory access (DMA) controller for transferring data between different memories, including an address bus, a data bus, and a control register. The data transfer engine may read the first operator data and the second operator data from the first memory, and transfer the first operator data and the second operator data to the second memory through the bus.

Since the amount of operation data is huge, the matrix operation engine cannot complete the operation in a single time, so it is necessary to divide the operation data into operation sub-data and perform operations on the operation sub-data, that is, to split the operation process of the first operation data and the second operation data into the operation process of multiple operation sub-data. Accordingly, the first operation data and the second operation data in the first memory need to be moved to the second memory in segments.

Optionally, the first operator data belongs to the first operation data and is a part of the first operation data, and the second operator data belongs to the second operation data and is a part of the second operation data. That is, the AI processor can realize the segmented transmission of the first operation data and the second operation data from the first memory to the second memory inside the matrix operation engine through the data transfer engine.

In the related art, when the data dimension represented by the data format adopted by the first operation data and the second operation data does not match the array dimension of the systolic array, in order not to affect the matrix operation process in the matrix operation unit, the AI processor performs data transfer according to the array dimension through the data transfer engine, resulting in the sub-data transferred in adjacent time not having data continuity, which in turn leads to low bus bandwidth utilization when the bus bit width is larger than the data bit width corresponding to the array dimension.

In an embodiment of the present application, after storing the first operation data and the second operation data in the target data format, the AI processor can read the first operator data and the second operator data directly from the first memory according to the bus width through the data transfer engine, thereby fully utilizing the bus width and ensuring the continuity of data transfer.

Schematically, when the operation data does not adopt the target data format and the bus width is larger than the data width corresponding to the array dimension, in order to enable the pulse array of the matrix operation unit to directly perform matrix operations on the sub-operation data transported by the data transport engine, the data transport engine transports the data in sequence according to the data width corresponding to the array dimension of the pulse array and the number of data rows or columns. Since the data width of each data transport is smaller than the bus width, the bus width is not fully utilized. In the case of adopting the target data format, the operation data can be transported in the array dimension of the pulse array. The data is stored continuously in physical storage using the number as the unit. The data transfer engine can directly transfer the data according to the bus width, that is, the data width of the transferred data is equal to the bus width, thereby making full use of the bus width.

In an illustrative example, when the bus bit width is twice the data bit width corresponding to the array dimension, if the target data format is not used to store the operation data, the operation data is moved according to the data bit width corresponding to the array dimension, and the bandwidth utilization rate is only 50%; after the operation data is stored in the target data format, the operation data can be stored continuously in the physical storage in units of the array dimension of the systolic array, so that the operation data can be moved according to the bus bit width, so that the bandwidth utilization rate reaches 100%.

For example, when the bus width is 2048 bits, the array dimension of the systolic array is 64, and the data in each dimension are all fp16 (i.e., all are 16-bit floating point numbers), the data width corresponding to the array dimension of the systolic array is 1024 bits, i.e., the bus width (2048 bits) is greater than the data width (1024 bits) corresponding to the array dimension of the systolic array. When the target data format is not used to store the operation data, the operation data is transferred according to the data width corresponding to the array dimension, and only 1024 bits of bus width can be occupied each time, resulting in a bandwidth utilization rate of only 50%; when the target data format is used to store the operation data, the operation sub-data with a data width of 2048 bits equal to the bus width can be read from the first memory according to the bus width.

In some embodiments, a second memory 111 is provided in the matrix operation engine 110 , and the data transfer engine 120 transfers the first operator data and the second operator data to the second memory 111 inside the matrix operation engine 110 during data transfer via the bus.

Optionally, the data transfer process may be understood as transmitting the first operator data and the second operator data to the second memory inside the matrix operation engine through the bus.

The matrix operation engine 110 is used to perform matrix operations on the first operator data and the second operator data in the second memory 111 through the matrix operation unit 112 to obtain sub-operation results, and the data dimension of the sub-operation results matches the array dimension.

In some embodiments, the matrix operation engine also includes a matrix operation unit. After storing the first operator data and the second operator data in the second memory, the matrix operation engine can perform matrix operations on the first operator data and the second operator data through the matrix operation unit using a systolic array to obtain a sub-operation result.

The data dimension of the sub-operation result matches the array dimension of the systolic array. For example, the number of data rows of the sub-operation result is equal to the number of array rows of the systolic array, and the number of data columns of the sub-operation result is equal to the number of array columns of the systolic array.

In some embodiments, the matrix operation engine may use the first operator data as the left input data of the systolic array and the second operator data as the right input data of the systolic array, and perform matrix operation through the matrix operation unit to obtain a sub-operation result.

Optionally, the matrix operation unit processes the first operator data and the second operator data through a matrix multiplication operation, so as to obtain a sub-operation result.

Schematically, as shown in FIG2 , the result data output by the matrix operation unit through the matrix operation unit is in the form of a two-dimensional matrix [Matrix_M, MALU_N], where, taking a calculation based on the systolic array as an example, after determining the left input data 21 and the right input data 22 of the systolic array, the matrix operation unit performs data operations from left to right and from top to bottom based on the properties of the systolic array, thereby outputting a set of data on the MALU_N side. Among them, the PEs (Process Element) in the systolic array is the smallest unit of the array, which is used to implement one-dimensional multiplication and addition calculations.

The matrix operation engine 110 is further used to accumulate the sub-operation results through the accumulator 113 to obtain the matrix operation result of the first operation data and the second operation data.

In some embodiments, the matrix operation engine also includes an accumulator (ACC). After performing matrix operations through the matrix operation unit to obtain a large number of sub-operation results, the matrix operation engine can also accumulate the sub-operation results through the accumulator to obtain the matrix operation results of the first operation data and the second operation data.

Optionally, the accumulator's accumulation process of sub-operation results refers to temporarily storing the sub-operation results output by the matrix operation unit, and after obtaining all the sub-operation results, performing data splicing on each sub-operation result according to the matrix element position of each sub-operation result in the operation result matrix, thereby obtaining the operation results of the first operation data and the second operation data.

Schematically, the first operation data and the second operation data are both 2×2 matrix data, and the matrix operation unit performs matrix operation to obtain: the first row matrix data of the first operation data and the first column matrix data of the second operation data The first sub-operation result corresponding to the first row matrix data of the first operation data and the second column matrix data of the second operation data, the third sub-operation result corresponding to the second row matrix data of the first operation data and the first column matrix data of the second operation data, and the fourth sub-operation result corresponding to the second row matrix data of the first operation data and the second column matrix data of the second operation data, so that the matrix element position of the first sub-operation data in the operation result matrix is the first row and the first column, the matrix element position of the second sub-operation data in the operation result matrix is the first row and the second column, the matrix element position of the third sub-operation data in the operation result matrix is the second row and the first column, and the matrix element position of the fourth sub-operation data in the operation result matrix is the second row and the second column, and then the accumulator can splice the sub-operation data according to the matrix element positions corresponding to each sub-operation data, so as to obtain the matrix operation results of the first operation data and the second operation data.

In some embodiments, after obtaining the operation results of the first operation data and the second operation data, the AI processor may also use the data transfer engine to transfer the operation results to the first memory through the bus.

In summary, in the embodiment of the present application, by storing the first operation data and the second operation data in the target data format in the first memory, the data dimension of the first operation data and the second operation data matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine, so that the data handling engine can read the first operator data and the second operator data from the first memory according to the bus width, and carry the first operator data and the second operator data to the second memory inside the matrix operation engine through the bus, and then the matrix operation engine performs matrix operation on the first operator data and the second operator data through the matrix operation unit to obtain the sub-operation result, and accumulates the sub-operation result through the accumulator to obtain the matrix operation result of the first operation data and the second operation data. Using the AI processor provided in the embodiment of the present application, by storing the operation data in the target data format in the first memory, it is possible to realize the reading of operation data according to the bus width. Since the bus width is greater than the data width corresponding to the array dimension, compared with reading data according to the data width corresponding to the array dimension, the AI processor provided in the embodiment of the present application can improve the bandwidth utilization of data handling during matrix operation, thereby improving the efficiency of matrix operation.

In some embodiments, the deep learning network may include a convolutional network for performing operations on convolutional data; or a matrix network for performing operations on matrix data. Therefore, after the AI processor is integrated into the deep learning network framework, the AI processor needs to process the convolutional data and the matrix data separately so that both the convolutional data and the matrix data conform to the target data format.

The target data formats used in convolution and matrix operation scenarios are described below.

In a possible implementation, in the process of processing convolution data, the first operation data may be convolution input data, the second operation data may be convolution weight data, and the operation result of the first operation data and the second operation data may be convolution output data.

Optionally, the convolution input data may be an image or feature map to be convolved, and the convolution weight data may be a convolution kernel used to convolve the image or feature map. Typically, the size of the convolution input data is smaller than the size of the convolution weight data. For example, the convolution input data is 100×100 matrix data, and the convolution weight data is a 3×3 convolution kernel.

In some embodiments, in the expression of dimensional information of a convolutional network, H represents the height of an image or a feature map, W represents the width of an image or a feature map, C represents the feature dimension, and N represents a batch of images or feature maps.

Schematically, as shown in Figure 3, it shows the logical expression of convolutional network data and the physical storage form of data in different formats in the related technology. Among them, the logical expression forms corresponding to NCHW format data, NHWC format data and CHWN format data are the same, that is, the data arrangement methods in the C direction, H direction and W direction are the same, and in the physical storage process, for NCHW format data, the W direction is taken first, followed by the H direction, then the C direction, and finally the N direction; for NHWC format data, the C direction is taken first, followed by the W direction, then the H direction, and finally the N direction; for CHWN format data, the N direction is taken first, followed by the W direction, then the H direction, and finally the C direction.

In the related art, taking NHWC format data as an example, when the number of array rows of the systolic array Ck0=3, it is necessary to carry out data transfer according to 000, 020, 040 as a group and 001, 021, 041 as a group. It can be seen that 000, 020, 040 and 001, 021, 041 are in discrete storage states, so the data handling engine can only handle one set of data at a time. Even if the bus bandwidth is greater than the data bit width corresponding to the number of array rows, the data handling engine can only read 3 data values at a time.

In some embodiments, when the first operation data is convolution input data, in order to fully utilize the bus bandwidth in the process of data transfer of the convolution input data, the convolution input data can be stored in a first target data format, wherein the number of sub-input channels represented by the first target data format is equal to the number of array rows of the systolic array.

In one possible implementation, the number of array rows of the systolic array can be expressed as Ck0, and the convolution input data in the first target data format can be expressed as NCi1HiWiCk0, wherein N represents the number of groups of images or feature maps, Hi represents the height of the input image or feature map, Wi represents the width of the input image or feature map, Ci1 is the value of Ci/Ck0 rounded up, and Ci is the total number of input channels (as shown in FIG4 ).

Schematically, as shown in Figure 4, it shows the physical storage form of the first target data format provided by an exemplary embodiment of the present application. As can be seen from Figure 4, for the target data format NCi1HiWiCk0, the data is arranged in the Ck0 direction first, then the Wi direction, then the Hi direction, then the Ci1 direction, and finally the N direction.

For example, when the number of array rows of the systolic array Ck0=3, first take the three values 000, 020, and 040 in the Ck0 direction, and then take 001, 021, and 041 from the Wi direction. After taking all the values in the Wi direction (that is, taking 003, 023, and 043), transfer from 043 to 004 on Hi, and store the data in this order, thereby ensuring that in the process of data transfer based on the number of array rows of the systolic array, the data transferred in adjacent times are in a continuous state.

In some embodiments, when the second operation data is convolution weight data, in order to fully utilize the bus bandwidth in the process of data transfer of the convolution weight data, the convolution weight data may be stored in a second target data format, wherein the number of sub-input channels represented by the second target data format is equal to the number of array rows of the systolic array, and the number of sub-output channels represented by the second target data format is equal to the number of array columns of the systolic array.

In one possible implementation, the number of array rows of the systolic array can be expressed as Ck0, the number of array columns of the systolic array can be expressed as Cn0, and the convolution weight data in the second target data format can be expressed as Co1Ci1KhKwCk0Cn0, wherein Kh represents the height of the convolution kernel, Kw represents the width of the convolution kernel, Ci1 is the value of Ci/Ck0 rounded up, Co1 represents the value of Co/Cn0 rounded up, Ci is the total number of input channels, and Co is the total number of output channels.

In some embodiments, when the operation result of the first operation data and the second operation data is convolution output data, convolution output data in a third target data format can be obtained by performing matrix operation through a matrix operation unit according to the convolution input data in the first target data format and the convolution weight data in the second target data format, wherein the number of sub-output channels represented by the third target data format is equal to the number of array columns of the systolic array.

In one possible implementation, the number of array columns of the systolic array can be represented by Cn0, and the convolution output data under the third target data format can be represented as NCo1HoWoCn0, wherein N represents the number of groups of images or feature maps, Ho represents the height of the output image or feature map, Wo represents the width of the output image or feature map, Co1 represents the value of Co/Cn0 rounded up, and Co is the total number of output channels.

In some embodiments, after the convolution input data in the first target data format and the convolution weight data in the second target data format are stored in the first memory of the AI processor, the AI processor can use the data transfer engine to transfer the convolution input sub-data and the convolution weight sub-data according to the bus bit width, and store them in the second memory inside the matrix operation engine.

In one possible implementation, the AI processor uses a data transfer engine to sequentially read the first convolution input sub-data and the second convolution input sub-data from the first memory according to the bus bit width, wherein the first convolution input sub-data and the second convolution input sub-data are continuously stored data, and the data bit width of the first convolution input sub-data is equal to the bus bit width, and the data bit width of the second convolution input sub-data is equal to the bus bit width.

Schematically, as shown in FIG4, when the number of array rows of the systolic array is 3 and the bus width is twice the data width corresponding to the number of array rows, the AI processor uses the data transfer engine to take six data 000, 020, 040, 001, 021, and 041 as the first convolution input sub-data according to the bus width, and transfers the six data at the same time, and when continuing to transfer data according to the number of array rows of the systolic array, it can continue to take 002, 022, 042, 003, 023, and 043 as the second convolution input sub-data. It can be seen that the first convolution input sub-data and the second convolution input sub-data are physically stored in the same memory. The stored data is arranged as continuous data.

In one possible implementation, the AI processor uses a data transfer engine to sequentially read the first convolution weight sub-data and the second convolution weight sub-data from the first memory according to the bus bit width, wherein the first convolution weight sub-data and the second convolution weight sub-data are continuously stored data, and the data bit width of the first convolution weight sub-data is equal to the bus bit width, and the data bit width of the second convolution weight sub-data is equal to the bus bit width.

Schematically, as shown in Figure 5, it shows a schematic diagram of data processing through a systolic array based on a general data format in the related art. Taking the use of a data handling engine to carry out data handling of convolution input data as an example, in the process of using the data handling engine to carry out data handling, in order to obtain a data value of a length of MALU_K from the convolution input data, the AI processor needs to read data from the convolution input data in discrete segments and carry out data handling through the bus in sequence, resulting in a problem of low bandwidth utilization when the bus bandwidth is greater than MALU_K.

Schematically, taking the logical expression form corresponding to the convolution input data in Figure 5 as an example, when the data is arranged in NHiWiCi format, it includes Wi direction 501, Hi direction 502 and Ci direction 503, and the data is first continuous in the Ci direction 503 during physical storage. When the number of array rows of the systolic array is MALU_K and the number of array columns is MALU_N, in the related art, the convolution input data is directly read in the Ci direction 503 according to the number of array rows 504 of MALU_K, so that a data block 505 (height is Cut_Hi, width is Cut_Wi, number of channels is MALU_K) can be obtained, and each segment of data in the MALU_K direction in the data block 505 is discretely distributed in the physical storage of the convolution input data, and when the bus bandwidth is greater than MALU_K, the data handling process cannot fully utilize the bus bandwidth.

Schematically, taking the logical expression form corresponding to the convolution weight data in Figure 5 as an example, when the data is arranged in the KhKwCiCo format, including the Ci direction 506 and the Co direction 507, the data is first continuous in the Co direction 507 during the physical storage process. When the number of array rows of the systolic array is MALU_K and the number of array columns is MALU_N, in the related art, the convolution weight data is directly read in the Ci direction 506 according to the number of array rows of MALU_K and in the Co direction 507 according to the number of array columns of MALU_N, so that each segment of data in the MALU_N direction in each data block 508 (with a height of MALU_K and a width of MALU_N) is discretely distributed in the physical storage of the convolution weight data, and when the bus bandwidth is greater than MALU_N, the bus bandwidth cannot be fully utilized during data transportation.

Schematically, as shown in FIG6 , a schematic diagram of data processing by a systolic array based on a target data format provided by an exemplary embodiment of the present application is shown. Taking the use of a data handling engine to handle convolution input data as an example, in the process of using the data handling engine to handle data, the AI processor, in order to take a data value of a length of MALU_K from the convolution input data, because the data is stored continuously along the MALU_K and then along the Wi direction under the target data format, when the bus bandwidth is greater than MALU_K, the AI processor can directly read the data values greater than MALU_K along the MALU_K and then along the Wi direction, thereby making full use of the bus bandwidth.

Schematically, taking the logical expression form corresponding to the convolution input data in Figure 6 as an example, when the data arrangement is the target data format NCi1HiWiCk0, it includes Wi direction 601, Hi direction 602 and Ci1 (Ci/Ck0) direction 603, and the data is first continuous in the Ci1 direction 603 during the physical storage process.

When the number of array rows of the systolic array is MALU_K and the number of array columns is MALU_N, since the data is first continuous in the Ci1 direction 603 during the physical storage process, the embodiment of the present application can directly read the data block 604 (height is Cut_Hi, width is Cut_Wi, number of channels is MALU_K) according to the bus bit width continuity, thereby improving the bus bandwidth utilization.

Schematically, taking the logical expression form corresponding to the convolution weight data in Figure 6 as an example, when the data is arranged in the target data format of Co1Ci1KhKwCk0Cn0, including Ci1 (Ci/Ck0) direction 605 and Co1 (Co/Cn0) direction 606, the data is first continuous in the Co1 direction 606 during the physical storage process. When the number of array rows of the systolic array is MALU_K and the number of array columns is MALU_N, since the data is first continuous in the Co1 direction 606 during the physical storage process, the embodiment of the present application can directly read the data block 607 (height is MALU_K and width is MALU_N) continuously according to the bus bit width, thereby improving the bus bandwidth utilization.

In some embodiments, considering that the channel dimensions of convolution data in different deep learning networks may be different, and the total number of channels of convolution data may be less than the number of array rows or array columns of the systolic array, for the convolution input data whose total input channel number is less than the number of array rows of the systolic array, or the convolution input data whose total output channel number is less than the number of array columns of the systolic array, To output data, different target data formats need to be used to store the convolution data.

In one possible implementation, when the total number of input channels of the convolution input data is less than the number of array rows of the systolic array (i.e., Ci/Ck0 is less than 1), the convolution input data may be stored in a fourth target data format, wherein the number of input channels represented by the fourth target data format is the total number of input channels.

In one possible implementation, the number of array rows of the systolic array can be represented as Ck0, and the total number of input channels of the convolution input data is Ci, which is less than Ck0. The convolution input data can be represented as NHiWiCi, where N represents the number of groups of images or feature maps, Hi represents the height of the input image or feature map, and Wi represents the width of the input image or feature map.

In a possible implementation, when the total number of input channels of the convolution input data is less than the number of array rows of the systolic array, and the total number of output channels of the convolution output data is not less than the number of array columns of the systolic array, the convolution weight data may be stored in a fifth target data format, wherein the number of input channels represented by the fifth target data format is the total number of input channels, and the number of sub-output channels represented by the fifth target data format is equal to the number of array columns of the systolic array. When the fourth target data format and the fifth target data format are used to store the convolution input data and the convolution weight data, additional data padding for the convolution input data and the convolution weight data can be avoided.

In a possible implementation, the number of array rows of the systolic array can be represented as Ck0, the number of array columns of the systolic array can be represented as Cn0, Ci is the total number of input channels, Co is the total number of output channels, and Ci is less than Ck0, and Co is greater than Cn0. The convolution weight data can be represented as Co1KwCiKhCn0, where Kh represents the convolution kernel height, Kh represents the convolution kernel width, and Co1 represents the value of Co/Cn0 rounded up.

In one possible implementation, when the total number of output channels of the convolution output data is less than the number of array columns of the systolic array (i.e., Co/Cn0 is less than 1), the convolution output data may be stored in a sixth target data format, wherein the number of output channels represented by the sixth target data format is the total number of output channels.

In one possible implementation, the number of array columns of the systolic array may be represented as Cn0, and the total number of output channels of the convolution output data may be Co, where Co is less than Cn0. The convolution output data may be represented as NHoWoCo, where N represents a set of images or feature maps, Ho represents the height of the output image or feature map, and Wo represents the width of the output image or feature map.

In a possible implementation, when the total number of output channels of the convolution output data is less than the number of array columns of the systolic array, and the total number of input channels of the convolution input data is not less than the number of array rows of the systolic array, the convolution weight data may be stored in a seventh target data format, wherein the number of sub-input channels represented by the seventh target data format is equal to the number of array rows of the systolic array, and the number of output channels represented by the seventh target data format is the total number of output channels.

In one possible implementation, the number of array rows of the systolic array can be represented as Ck0, the number of array columns of the systolic array can be represented as Cn0, Ci is the total number of input channels, Co is the total number of output channels, and Ci is greater than Ck0, Co is less than Cn0, and the convolution weight data can be expressed as Ci1KhKwCk0ECn0, wherein Kh represents the height of the convolution kernel, Kw represents the width of the convolution kernel, Ci1 is the value of Ci/Ck0 rounded up, and W is the expansion coefficient in the Co small special optimization algorithm, which is jointly determined by the computing power, bandwidth and convolution parameters.

When the sixth target data format and the seventh target data format are used to store the convolution output data and the convolution weight data, additional data padding (padding) of the convolution weight data can be avoided during the operation. In the above embodiment, for the convolution data, by comparing the total number of input channels of the convolution input data with the number of array rows of the systolic array, and the total number of output channels of the convolution output data with the number of array columns of the systolic array, the target data formats corresponding to the convolution input data, the convolution weight data and the convolution output data are determined, thereby realizing the determination of different target data formats according to different data situations. Since the bus bit width is larger than the data bit width corresponding to the array dimension, compared with data reading according to the data bit width corresponding to the array dimension, for the convolution data, data reading can be realized directly according to the bus bit width, so that the data bit width of the input sub-data and the weight sub-data are equal to the bus bit width, thereby optimizing the data handling efficiency.

In a possible implementation, during processing of matrix data, the first operation data may be left matrix input data, the second operation data may be right matrix input data, and the operation results of the first operation data and the second operation data are matrix output data.

In some embodiments, in the expression of the dimensional information of the matrix network, the left matrix is represented as MK (M rows and K columns), the right matrix is represented as KN (K rows and N columns), and the result matrix is represented as MN (M rows and N columns). In the network, batch data of other dimensions will be added on this basis.

In some embodiments, when the first operation data is the left matrix input data, in order to fully utilize the bus bandwidth in the process of data transfer of the left matrix input data, the left matrix input data may be stored in an eighth target data format, wherein the number of submatrix columns represented by the eighth target data format is equal to the number of array rows of the systolic array.

In one possible implementation, the number of array rows of the systolic array is Wk0, and the left matrix input data in the eighth target data format can be expressed as B0B1W1HWk0, wherein B0 and B1 respectively represent the number of batches in two dimensions, H represents the number of matrix rows of the left matrix, W1 is the value rounded up of K/Wk0, and K is the number of matrix columns of the left matrix.

In some embodiments, when the second operation data is right matrix input data, in order to fully utilize the bus bandwidth during data transfer of the right matrix input data, the right matrix input data can be stored in a ninth target data format, wherein the number of submatrix columns represented by the ninth target data format is equal to the number of array columns of the systolic array.

In a possible implementation, the number of array columns of the systolic array is Wn0, and the right matrix input data in the ninth target data format can be expressed as B0B1W1HWn0, where B0 and B1 represent the number of batches in two dimensions, respectively. H represents the number of matrix rows of the right matrix, W1 is the value of N/Wn0 rounded up, and N is the number of matrix columns of the right matrix.

It should be noted that the batches in the left matrix input data and the right matrix input data may not be equal, but when they are not equal, one of them must be 1.

In some embodiments, when the operation result of the first operation data and the second operation data is matrix output data, matrix output data using the tenth target data format can be obtained by performing matrix operations through a matrix operation unit based on left matrix input data using the eighth target data format and right matrix input data using the ninth target data format, wherein the number of matrix rows represented by the tenth target data format is equal to the number of matrix rows represented by the eighth target data format, and the number of matrix columns represented by the tenth target data format is equal to the number of matrix columns represented by the ninth target data format.

In one possible implementation, under the tenth target data format, the matrix output data can be expressed as B0B1W1HWn0, wherein B0 in the matrix output data is the maximum value of B0 in the left matrix input data and B0 in the right matrix input data, B1 in the matrix output data is the maximum value of B1 in the left matrix input data and B1 in the right matrix input data, W1 and Wn0 are the same as the W1 and Wn0 values of the right matrix input data, and H is the same as the H value of the left matrix input data.

In some embodiments, after the left matrix input data in the eighth target data format and the right matrix input data in the ninth target data format are stored in the first memory of the AI processor, the AI processor can use the data transfer engine to transfer the left matrix input sub-data and the right matrix input sub-data according to the bus bit width, and store them in the second memory inside the matrix operation engine.

In one possible implementation, the AI processor uses a data transfer engine to sequentially read the first left matrix input sub-data and the second left matrix input sub-data from the first memory according to the bus bit width, wherein the first left matrix input sub-data and the second left matrix input sub-data are continuously stored data, and the data bit width of the first left matrix input sub-data is equal to the bus bit width, and the data bit width of the second left matrix input sub-data is equal to the bus bit width.

In one possible implementation, the AI processor uses a data transfer engine to sequentially read the first right matrix input sub-data and the second right matrix input sub-data from the first memory according to the bus bit width, wherein the first right matrix input sub-data and the second right matrix input sub-data are continuously stored data, and the data bit width of the first right matrix input sub-data is equal to the bus bit width, and the data bit width of the second right matrix input sub-data is equal to the bus bit width.

Schematically, as shown in Figure 7, it shows a schematic diagram of data processing through a systolic array based on a general data format in the related art. Taking the use of a data handling engine to carry out data handling on the left matrix input data as an example, in the process of using the data handling engine to carry out data handling, in order to obtain a data value of a length of MALU_K from the left matrix input data, the AI processor needs to read data from the left matrix input data in segments along the M direction discretely, and carry out data handling through the bus in sequence, resulting in a problem of low bandwidth utilization when the bus bandwidth is greater than MALU_K.

Schematically, taking the left matrix input data in FIG. 7 as an example, when the data arrangement mode is in MK format, it includes M direction 701 and K direction 702, and the data is first stored continuously in K direction 702 during physical storage (i.e., first stored in K direction, then stored in M direction). When the number of array rows of the systolic array is MALU_K and the number of array columns is MALU_N, in the related art, data is directly read from K direction 702 according to the number of array rows of MALU_K, and each segment of data in the MALU_K direction in the data block 703 is discretely distributed in the physical storage of the left matrix input data ( FIG. The data block 703 in FIG. 7 spans multiple rows in the M direction, so the data in the data block 703 is discretely distributed), and when the bus bandwidth is greater than MALU_K, the problem of low bandwidth utilization will also occur during data transfer. The right matrix input data in FIG. 7 is similar, and the embodiment of the present application will not be described in detail here.

Schematically, as shown in FIG8, it shows a schematic diagram of data processing through a systolic array based on a target data format provided by another exemplary embodiment of the present application. Taking the use of a data handling engine to handle data on the left matrix input data as an example, in the process of using the data handling engine to handle data, the AI processor, in order to obtain a data value of a length of MALU_K from the left matrix input data, because the data is stored continuously along the MALU_K and then along the H_Left direction in the target data format, when the bus bandwidth is greater than MALU_K, the AI processor can directly read the data values greater than MALU_K along the MALU_K and then along the H_Left direction, thereby making full use of the bus bandwidth.

Schematically, taking the left matrix input data in FIG8 as an example, when the data arrangement mode is the target data format of B0B1W1HWk0, including the H direction 801 (vertical) and the W1 (K/Wk0) direction 802 (horizontal), the data is first continuous in the W1 direction 802 during the physical storage process (i.e., first stored along the W1 direction, and then stored along the H direction). When the number of array rows of the systolic array is MALU_K and the number of array columns is MALU_N, since the data is first continuous in the W1 direction 802 during the physical storage process, the embodiment of the present application can directly read the data block 803 according to the bus width continuity (although the data block 803 in FIG8 spans multiple rows in the H direction, the data in the target data format is continuously stored in the H direction, so the data in the data block 803 is continuous), thereby improving the bus bandwidth utilization. The same is true for the right matrix input data in FIG8, and the embodiment of the present application will not be repeated here.

In the above embodiment, for matrix data, the specific target data format adopted by the left matrix input data, the right matrix input data and the matrix output data is determined according to the matrix column number of the left matrix input data and the right matrix input data, and in combination with the array row number and the array column number of the systolic array. Since the bus width is larger than the data width corresponding to the array dimension, compared with data reading according to the data width corresponding to the array dimension, for matrix data, data reading can be directly performed according to the bus width, so that the data width of the input sub-data is equal to the bus width, thereby optimizing the efficiency of data transfer for matrix data.

In some embodiments, considering that in deep learning networks there are often scenarios where convolution operators and matrix operators are used alternately, and there are differences in the data storage formats of convolution data and matrix data, in order to improve the data processing efficiency in the AI processor, a data format converter may also be provided in the AI processor.

Optionally, the data format converter is used to perform unidirectional conversion on the data storage formats of convolution data and matrix data (if mutual conversion is to be achieved, data format converters with different conversion directions need to be set), or bidirectional conversion.

In one possible scenario, when it is necessary to first use the convolution kernel to convolve the feature map (convolution input data), and then use the convolution result as the left matrix input data to perform matrix multiplication with the right matrix input data, it is necessary to convert the convolution output data into the data format used by the matrix input data.

In a possible implementation, when the operation result of the first operation data and the second operation data is convolution output data, and the convolution output data is used for subsequent matrix calculations, the data format converter can be used to convert the data format of the convolution output data based on the target data format corresponding to the matrix input data.

In one possible implementation, the data format of the convolution output data is NC1HWC0, and the data format of the matrix input data is B0B1W1HW0, then B0 can be set to 1, B1 can be set to N, W1 can be set to C1, H can be set to H×W, and W0 can be set to C0, thereby converting the convolution output data into equivalent matrix input data.

In a possible scenario, when it is necessary to first perform matrix multiplication on two matrices (left matrix input data and right matrix input data) and then use a convolution kernel to convolve the matrix multiplication result, it is necessary to convert the matrix data into the data format used by the convolution input data.

In a possible implementation, when the operation result of the first operation data and the second operation data is matrix output data, and the matrix output data is used for subsequent convolution calculations, the data format converter can be used to convert the data format of the matrix output data based on the target data format corresponding to the convolution input data.

In a possible implementation, the data format of the matrix output data is B0B1W1HW0, and the data format of the convolution input data is NC1HWC0, then N can be set to B0×B1, C1 to W1, H to 1, W to H, and C0 Set to W0 to make the matrix output data equivalent to the convolution input data.

In the above embodiments, when the convolution output data or the matrix output data adopts the target data format and there are subsequent operations with different data types, the data equivalence relationship between the convolution output data and the matrix input data, or the data equivalence relationship between the matrix output data and the convolution input data can be determined, that is, the data equivalence between the convolution output data and the matrix input data, or the data equivalence between the matrix output data and the convolution input data can be achieved, thereby improving the data processing efficiency in the AI processor.

In some embodiments, considering that after the AI processor is integrated into the deep learning network framework, data calculation through the matrix operation engine is only one of the links, that is, the calculation data obtained based on the previous process may not be in the target data format, so directly moving the data may affect the efficiency of data movement. Therefore, a data format converter can also be provided in the AI processor to convert the data format of the calculation data into the target data format.

In a possible implementation, a data format converter is used to read first operation data and second operation data from a first memory, the first operation data and the second operation data are in an original data format, and the data dimension represented by the original data format does not match the array dimension of the systolic array in the matrix operation unit in the matrix operation engine, and then the data format converter converts the data format of the first operation data and the second operation data from the original data format to the target data format by performing data format conversion on the first operation data and the second operation data, and writes the first operation data and the second operation data into the first memory.

In an illustrative example, the data format converter first reads the convolution input data in the original data format (NCHW) from the first memory, and converts the convolution input data according to the target data format (NCi1HiWiCk0), thereby storing the converted convolution input data in the target data format back in the first memory.

It should be noted that before performing data format conversion, the data needs to be converted into the target data format corresponding to the user based on the purpose of the data. For example, the convolution input data is converted into the first target data format, the convolution weight data is converted into the second target data format, the left matrix input data is converted into the eighth target data format, the right matrix input data is converted into the ninth target data format, and so on.

In addition, for convolution data, it is also necessary to determine the target data format used for the convolution input data, convolution weight data, and convolution output data based on the relationship between the total number of input channels of the convolution input data and the number of array rows of the pulse array, and based on the relationship between the total number of output channels of the convolution output data and the number of array columns of the pulse array. This embodiment will not be elaborated here.

In one possible implementation, when an AI processor is integrated into a general deep learning framework as a computing acceleration device, in order to reduce the complexity of integrating the AI processor into the deep learning framework, corresponding format processing can also be provided at the operator layer and the framework layer. For example, at the operator layer, full support for the original data format and the target data format is added; at the framework layer, format information about the original data format and the target data format is provided.

Schematically, as shown in FIG9 , Math Dim (Math Dimention) is set to correspond to the operation data in the original data format (general format) in the algorithm framework, and Data Dim (Data Dimention) corresponds to the operation data in the target data format (special format) on the AI processor. In the process of data calculation through the deep learning framework, the operation data in the target data format can be directly used for data processing, and in the debugging process, when the data is transmitted from the device (Device) to the host (Host), the operation data in the target data format can be converted into the operation data in the original data format (that is, the format conversion is completed during the transmission process) according to the format information of the original data format and the target data format, so as to facilitate viewing and analysis.

Please refer to FIG. 10 , which shows a flow chart of a data processing method provided by an exemplary embodiment of the present application. The method is used in the AI processor in the above embodiment, and the method includes:

Step 1001, storing first operation data and second operation data in a first memory, wherein the first operation data and the second operation data are in a target data format, and the data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine.

Step 1002, based on the bus bit width, read the first operator data and the second operator data from the first memory through the data transfer engine, and transfer the first operator data and the second operator data to the second memory inside the matrix operation engine through the bus, and the bus bit width is greater than the data bit width corresponding to the array dimension.

Step 1003, through the matrix operation unit in the matrix operation engine, the first operator data and the second operator data in the second memory are matrix operated to obtain a sub-operation result, and the data dimension of the sub-operation result matches the array dimension.

Step 1004, accumulating the sub-operation results through an accumulator in the matrix operation engine to obtain a matrix operation result of the first operation data and the second operation data.

In some embodiments, in the process of processing convolution data, the first operation data is the convolution input data, the second operation data is the convolution weight data, and the operation result of the first operation data and the second operation data is the convolution output data; in the process of processing matrix data, the first operation data is the left matrix input data, the second operation data is the right matrix input data, and the operation result of the first operation data and the second operation data is the matrix output data.

In some embodiments, when the first operation data is convolution input data, the second operation data is convolution weight data, and the operation results of the first operation data and the second operation data are convolution output data, the convolution input data adopts a first target data format, and the number of sub-input channels represented by the first target data format is equal to the number of array rows of the systolic array; the convolution weight data adopts a second target data format, and the number of sub-input channels represented by the second target data format is equal to the number of array rows of the systolic array, and the number of sub-output channels represented by the second target data format is equal to the number of array columns of the systolic array; the convolution output data adopts a third target data format, and the number of sub-output channels represented by the third target data format is equal to the number of array columns of the systolic array.

In some embodiments, when the total number of input channels of the convolution input data is less than the number of array rows of the systolic array, the convolution input data adopts a fourth target data format, and the number of input channels represented by the fourth target data format is the total number of input channels; the convolution weight data adopts a fifth target data format, and the number of input channels represented by the fifth target data format is the total number of input channels, and the number of sub-output channels represented by the fifth target data format is equal to the number of array columns of the systolic array.

In some embodiments, when the total number of output channels of the convolution output data is less than the number of array columns of the systolic array, the convolution output data adopts a sixth target data format, and the number of output channels represented by the sixth target data format is the total number of output channels; the convolution weight data adopts a seventh target data format, and the number of sub-input channels represented by the seventh target data format is equal to the number of array rows of the systolic array, and the number of output channels represented by the seventh target data format is the total number of output channels.

In some embodiments, during data transfer, through a data transfer engine, based on the bus bit width, the first convolution input sub-data and the second convolution input sub-data are read from the first memory in sequence, the first convolution input sub-data and the second convolution input sub-data are continuously stored data, and the data bit width of the first convolution input sub-data and the data bit width of the second convolution input sub-data are both equal to the bus bit width; based on the bus bit width, the first convolution weight sub-data and the second convolution weight sub-data are read from the first memory in sequence, the first convolution weight sub-data and the second convolution weight sub-data are continuously stored data, and the data bit width of the first convolution weight sub-data and the data bit width of the second convolution weight sub-data are both equal to the bus bit width.

In some embodiments, when the first operation data is left matrix input data, the second operation data is right matrix input data, and the operation results of the first operation data and the second operation data are matrix output data, the left matrix input data adopts an eighth target data format, and the number of submatrix columns represented by the eighth target data format is equal to the number of array rows of the systolic array; the right matrix input data adopts a ninth target data format, and the number of submatrix columns represented by the ninth target data format is equal to the number of array columns of the systolic array; the matrix output data adopts a tenth target data format, and the number of matrix rows represented by the tenth target data format is equal to the number of matrix rows represented by the eighth target data format, and the number of matrix columns represented by the tenth target data format is equal to the number of matrix columns represented by the ninth target data format.

In some embodiments, during data transfer, through a data transfer engine, based on the bus bit width, the first left matrix input sub-data and the second left matrix input sub-data are read from the first memory in sequence, the first left matrix input sub-data and the second left matrix input sub-data are continuously stored data, and the data bit width of the first left matrix input sub-data and the data bit width of the second left matrix input sub-data are both equal to the bus bit width; based on the bus bit width, the first right matrix input sub-data and the second right matrix input sub-data are read from the first memory in sequence, the first right matrix input sub-data and the second right matrix input sub-data are continuously stored data, and the data bit width of the first right matrix input sub-data and the data bit width of the second right matrix input sub-data are both equal to the bus bit width.

In some embodiments, the AI processor further includes a data format converter;

The method also includes: when the operation results of the first operation data and the second operation data are convolution output data, and the convolution output data is used for subsequent matrix calculation, converting the data format of the convolution output data based on the target data format corresponding to the matrix input data through a data format converter; when the operation results of the first operation data and the second operation data are matrix output data, and the matrix output data is used for subsequent convolution calculation, converting the data format of the matrix output data based on the target data format corresponding to the convolution input data through a data format converter.

In some embodiments, the AI processor further includes a data format converter;

The method further includes: reading first operation data and second operation data from a first memory through a data format converter, the first operation data and the second operation data being in an original data format, the data dimension represented by the original data format not matching the array dimension of a systolic array in a matrix operation unit in a matrix operation engine;

The data formats of the first operation data and the second operation data are converted by a data format converter, so that the data formats of the first operation data and the second operation data are converted from the original data format to the target data format, and the first operation data and the second operation data are written into the first memory.

In summary, in the embodiment of the present application, by storing the first operation data and the second operation data in the target data format in the first memory, the data dimension of the first operation data and the second operation data matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine, so that the data handling engine can read the first operator data and the second operator data from the first memory according to the bus width, and carry the first operator data and the second operator data to the second memory inside the matrix operation engine through the bus, and then the matrix operation engine performs matrix operation on the first operator data and the second operator data through the matrix operation unit to obtain the sub-operation result, and accumulates the sub-operation result through the accumulator to obtain the matrix operation result of the first operation data and the second operation data. Using the AI processor provided in the embodiment of the present application, by storing the operation data in the target data format in the first memory, it is possible to realize the reading of operation data according to the bus width. Since the bus width is greater than the data width corresponding to the array dimension, compared with reading data according to the data width corresponding to the array dimension, the AI processor provided in the embodiment of the present application can improve the bandwidth utilization of data handling during matrix operation, thereby improving the efficiency of matrix operation.

Please refer to FIG. 11 , which shows a block diagram of a computer device 1100 provided by an exemplary embodiment of the present application. The computer device 1100 may be a portable mobile terminal, such as a smart phone, a tablet computer, a Moving Picture Experts Group Audio Layer III (MP3) player, or a Moving Picture Experts Group Audio Layer IV (MP4) player. The computer device 1100 may also be called a user device, a portable terminal, a workstation, a server, or other names.

Typically, the computer device 1100 includes: an AI processor 1101 and a memory 1102 .

The AI processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The AI processor 1101 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The AI processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the AI processor 1101 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the AI processor 1101 may also be used to process computing operations related to machine learning.

The memory 1102 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash memory storage devices.

In some embodiments, the computer device 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device.

Those skilled in the art will appreciate that the structure shown in FIG. 11 does not limit the computer device 1100 , and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims (19)

An AI processor, comprising: a matrix operation engine, a data handling engine and a first memory, wherein the matrix operation engine and the data handling engine are connected via a bus; The first memory is used to store first operation data and second operation data, the first operation data and the second operation data are in a target data format, and the data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine; The data transport engine is configured to read the first operator data and the second operator data from the first memory based on a bus bit width, and transport the first operator data and the second operator data to a second memory inside the matrix operation engine through the bus, wherein the bus bit width is greater than a data bit width corresponding to the array dimension; The matrix operation engine is used to perform a matrix operation on the first operator data and the second operator data in the second memory through a matrix operation unit to obtain a sub-operation result, wherein the data dimension of the sub-operation result matches the array dimension; The matrix operation engine is further used to accumulate the sub-operation results through an accumulator to obtain a matrix operation result of the first operation data and the second operation data. The AI processor according to claim 1, wherein: In the process of processing convolution data, the first operation data is convolution input data, the second operation data is convolution weight data, and the operation result of the first operation data and the second operation data is convolution output data; In the process of processing matrix data, the first operation data is left matrix input data, the second operation data is right matrix input data, and the operation results of the first operation data and the second operation data are matrix output data. The AI processor according to claim 2, wherein, when the first operation data is convolution input data, the second operation data is convolution weight data, and the operation results of the first operation data and the second operation data are convolution output data, The convolution input data adopts a first target data format, and the number of sub-input channels represented by the first target data format is equal to the number of array rows of the systolic array; The convolution weight data adopts a second target data format, the number of sub-input channels represented by the second target data format is equal to the number of array rows of the systolic array, and the number of sub-output channels represented by the second target data format is equal to the number of array columns of the systolic array; The convolution output data adopts a third target data format, and the number of sub-output channels represented by the third target data format is equal to the number of array columns of the systolic array. The AI processor of claim 3, wherein, when the total number of input channels of the convolution input data is less than the number of array rows of the systolic array, The convolution input data adopts a fourth target data format, and the number of input channels represented by the fourth target data format is the total number of input channels; The convolution weight data adopts a fifth target data format, the number of input channels represented by the fifth target data format is the total number of input channels, and the number of sub-output channels represented by the fifth target data format is equal to the number of array columns of the systolic array. The AI processor of claim 3, wherein, when the total number of output channels of the convolution output data is less than the number of array columns of the systolic array, The convolution output data adopts a sixth target data format, and the number of output channels represented by the sixth target data format is the total number of output channels; The convolution weight data adopts the seventh target data format, the number of sub-input channels represented by the seventh target data format is equal to the number of array rows of the systolic array, and the number of output channels represented by the seventh target data format is equal to the total output Number of channels. The AI processor according to any one of claims 3 to 5, wherein the data handling engine is used to: Based on the bus bit width, first convolution input sub-data and second convolution input sub-data are sequentially read from the first memory, where the first convolution input sub-data and the second convolution input sub-data are continuously stored data, and the data bit width of the first convolution input sub-data and the data bit width of the second convolution input sub-data are both equal to the bus bit width; Based on the bus bit width, the first convolution weight sub-data and the second convolution weight sub-data are read from the first memory in sequence, the first convolution weight sub-data and the second convolution weight sub-data are continuously stored data, and the data bit width of the first convolution weight sub-data and the data bit width of the second convolution weight sub-data are both equal to the bus bit width. The AI processor according to any one of claims 2 to 6, wherein, when the first operation data is left matrix input data, the second operation data is right matrix input data, and the operation results of the first operation data and the second operation data are matrix output data, The left matrix input data adopts an eighth target data format, and the number of submatrix columns represented by the eighth target data format is equal to the number of array rows of the systolic array; The right matrix input data adopts a ninth target data format, and the number of submatrix columns represented by the ninth target data format is equal to the number of array columns of the systolic array; The matrix output data adopts the tenth target data format, the number of matrix rows represented by the tenth target data format is equal to the number of matrix rows represented by the eighth target data format, and the number of matrix columns represented by the tenth target data format is equal to the number of matrix columns represented by the ninth target data format. The AI processor according to claim 7, wherein the data handling engine is used to: Based on the bus bit width, first left matrix input sub-data and second left matrix input sub-data are sequentially read from the first memory, the first left matrix input sub-data and the second left matrix input sub-data are continuously stored data, and the data bit width of the first left matrix input sub-data and the data bit width of the second left matrix input sub-data are both equal to the bus bit width; Based on the bus bit width, the first right matrix input sub-data and the second right matrix input sub-data are read from the first memory in sequence, the first right matrix input sub-data and the second right matrix input sub-data are continuously stored data, and the data bit width of the first right matrix input sub-data and the data bit width of the second right matrix input sub-data are both equal to the bus bit width. The AI processor according to any one of claims 1 to 8, wherein the AI processor further comprises a data format converter; The data format converter is used for: When the operation results of the first operation data and the second operation data are convolution output data, and the convolution output data is used for subsequent matrix calculation, based on the target data format corresponding to the matrix input data, converting the data format of the convolution output data; When the operation results of the first operation data and the second operation data are matrix output data, and the matrix output data is used for subsequent convolution calculation, the data format of the matrix output data is converted based on the target data format corresponding to the convolution input data. The AI processor according to any one of claims 1 to 8, wherein the AI processor further comprises a data format converter; The data format converter is used to read the first operation data and the second operation data from the first memory. calculation data, the first calculation data and the second calculation data are in original data format, and the data dimension represented by the original data format does not match the array dimension of the systolic array in the matrix calculation unit in the matrix calculation engine; The data format converter is also used to convert the data formats of the first operation data and the second operation data, so that the data formats of the first operation data and the second operation data are converted from the original data format to the target data format, and the first operation data and the second operation data are written into the first memory. A data processing method, the method is executed by an AI processor, the AI processor comprises a matrix operation engine, a data handling engine and a first memory, the matrix operation engine and the data handling engine are connected via a bus; The method comprises: storing first operation data and second operation data by the first memory, wherein the first operation data and the second operation data are in a target data format, and the data dimension represented by the target data format matches the array dimension of the systolic array in the matrix operation unit in the matrix operation engine; Based on a bus bit width, reading first operator data and second operator data from the first memory through the data transfer engine, and transferring the first operator data and the second operator data to a second memory inside the matrix operation engine through the bus, wherein the bus bit width is greater than a data bit width corresponding to the array dimension; Performing a matrix operation on the first operator data and the second operator data in the second memory through a matrix operation unit in the matrix operation engine to obtain a sub-operation result, wherein the data dimension of the sub-operation result matches the array dimension; The sub-operation results are accumulated by an accumulator in the matrix operation engine to obtain a matrix operation result of the first operation data and the second operation data. The method according to claim 11, wherein In the process of processing convolution data, the first operation data is convolution input data, the second operation data is convolution weight data, and the operation result of the first operation data and the second operation data is convolution output data; In the process of processing matrix data, the first operation data is left matrix input data, the second operation data is right matrix input data, and the operation results of the first operation data and the second operation data are matrix output data. The method according to claim 12, wherein, when the first operation data is convolution input data, the second operation data is convolution weight data, and the operation results of the first operation data and the second operation data are convolution output data, The convolution input data adopts a first target data format, and the number of sub-input channels represented by the first target data format is equal to the number of array rows of the systolic array; The convolution weight data adopts a second target data format, the number of sub-input channels represented by the second target data format is equal to the number of array rows of the systolic array, and the number of sub-output channels represented by the second target data format is equal to the number of array columns of the systolic array; The convolution output data adopts a third target data format, and the number of sub-output channels represented by the third target data format is equal to the number of array columns of the systolic array. The method of claim 13, wherein, when the total number of input channels of the convolution input data is less than the number of array rows of the systolic array, The convolution input data adopts a fourth target data format, and the number of input channels represented by the fourth target data format is the total number of input channels; The convolution weight data adopts a fifth target data format, the number of input channels represented by the fifth target data format is the total number of input channels, and the number of sub-output channels represented by the fifth target data format is equal to the number of array columns of the systolic array. The method of claim 13, wherein, when the total number of output channels of the convolution output data is less than the number of array columns of the systolic array, The convolution output data adopts a sixth target data format, and the number of output channels represented by the sixth target data format is the total number of output channels; The convolution weight data adopts a seventh target data format, the number of sub-input channels represented by the seventh target data format is equal to the number of array rows of the systolic array, and the number of output channels represented by the seventh target data format is the total number of output channels. The method according to claim 12, wherein, when the first operation data is left matrix input data, the second operation data is right matrix input data, and the operation results of the first operation data and the second operation data are matrix output data, The left matrix input data adopts an eighth target data format, and the number of submatrix columns represented by the eighth target data format is equal to the number of array rows of the systolic array; The right matrix input data adopts a ninth target data format, and the number of submatrix columns represented by the ninth target data format is equal to the number of array columns of the systolic array; The matrix output data adopts the tenth target data format, the number of matrix rows represented by the tenth target data format is equal to the number of matrix rows represented by the eighth target data format, and the number of matrix columns represented by the tenth target data format is equal to the number of matrix columns represented by the ninth target data format. The method according to any one of claims 11 to 16, wherein the AI processor further comprises a data format converter; The method further comprises: When the operation results of the first operation data and the second operation data are convolution output data, and the convolution output data is used for subsequent matrix calculation, converting the data format of the convolution output data based on the target data format corresponding to the matrix input data by the data format converter; When the operation results of the first operation data and the second operation data are matrix output data, and the matrix output data is used for subsequent convolution calculation, the data format of the matrix output data is converted through the data format converter based on the target data format corresponding to the convolution input data. The method according to any one of claims 11 to 16, wherein the AI processor further comprises a data format converter; The method further comprises: Reading the first operation data and the second operation data from the first memory through the data format converter, wherein the first operation data and the second operation data are in original data format, and the data dimension represented by the original data format does not match the array dimension of the systolic array in the matrix operation unit in the matrix operation engine; The data formats of the first operation data and the second operation data are converted by the data format converter, so that the data formats of the first operation data and the second operation data are converted from the original data format to the target data format, and the first operation data and the second operation data are written into the first memory. A computer device, characterized in that the computer device comprises a memory and an AI processor according to any one of claims 1 to 10, wherein the memory stores at least one instruction, and the at least one instruction is used to be executed by the AI processor.