WO2024222115A1 - Neural network pruning method and apparatus, and data processing method and apparatus - Google Patents

Abstract

A neural network pruning method and apparatus, and a data processing method and apparatus. A complementary sparse mode is provided. Therefore, every K pruning vectors with a length of M in weight tensors are spliced into one data block (or vector), and after sparsity, at the position of a non-zero element in one pruning vector, the other pruning vectors in a data block to which the pruning vector belongs are zero elements. The method improves the accuracy of application of a network model, reduces the calculation amount of the model, and can balance the accuracy and acceleration.

Description

Neural network pruning method, data processing method and device

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on April 25, 2023, with application number 202310470198.9 and invention name "Neural Network Pruning Method, Data Processing Method and Device", all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of deep learning technology, and in particular to a neural network pruning method, a data processing method and a device.

Background Art

At present, the scale of neural networks is constantly increasing, and the number of parameters and the amount of calculation are also increasing. In order to reduce the number of parameters and the amount of calculation, pruning, quantization, distillation or lightweight model design can be used. Pruning is widely used. Pruning refers to resetting some non-critical weights to 0 in redundant large models to achieve a sparse model. The selection method of the part that needs to be pruned can adopt any sparse method, but the distribution position of the 0 element in any pruning method is relatively random. Since there is no rule to follow, it does not bring about the optimization of reasoning performance. The selection method of the part that needs to be pruned can also adopt a structured sparse method, but in the structured sparse method, the weights of the convolution are pruned according to the channel, the sparse granularity is coarse, and the accuracy is low.

Summary of the invention

The embodiments of the present application provide a neural network pruning method, a data processing method and a device, so as to provide a pruning method that balances reasoning performance and accuracy.

In a first aspect, an embodiment of the present application provides a pruning method for a neural network, which can be performed by a pruning device, such as a processor in the pruning device. The method includes: obtaining a first weight tensor of the neural network to be pruned; performing at least one pruning process on the first weight tensor to obtain a second weight tensor; wherein the second weight tensor includes multiple data blocks, each of which is composed of K pruning vectors, and each pruning vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruning vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruning vectors in the first data block except the first pruning vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruning vector is any pruning vector in the first data block.

Compared with the single-channel or block-based sparse method, the embodiment of the present application uses a smaller granularity for sparseness, which can improve the accuracy of the application of the network model. Compared with the use of any sparse method, sparseness is performed in a complementary manner, with certain rules, which is understood as a kind of semi-structured sparseness. Therefore, there is no need to use dense calculations in the reasoning stage, which reduces the amount of calculation of the model. In the pruning method provided in the embodiment of the present application, only one non-zero element is selected for the same position of each K pruning vector to participate in the calculation. Since the other weights at the same position are all zero elements, the operation speed can be improved and the power consumption can be reduced when applied to acceleration hardware. The solution provided in the embodiment of the present application can balance accuracy and acceleration.

In one possible design, the sparsity S of the second weight tensor satisfies: S = 1-1/K. In the above scheme, the number of pruned vectors can be determined based on the sparsity, providing a feasible way to make the position where the non-zero element in a pruned vector is located, and other pruned vectors in the data block to which the pruned vector belongs have zero elements at that position.

In a possible design, the pruning process is performed N times in a manner of increasing sparsity. The accuracy of the network can be improved by using a progressive sparsification method.

In one possible design, the number of times N that the pruning process is performed is related to the number K of pruning vectors included in the data block and/or the sparsity of the second weight tensor.

In one possible design, the pruning process is performed a number of times that satisfies: N=S/(1-S), or N=K-1.

It can be understood that the weight tensor is pruned along each M*K. Each pruning can be understood as pruning the number of weights included in one pruning vector. Compared with pruning a considerable number of weights at a time, the accuracy of the network can be improved.

In one possible design, N = K-1, and the sparsity of the i-th pruning process satisfies:

Wherein, _Si represents the sparsity of the i-th pruning process, i is a positive integer, 1≤i＜K.

In one possible design, the i-th pruning process is performed, where i is less than or equal to N, including:

A deformed three-dimensional weight tensor obtained by deforming an input four-dimensional weight tensor, wherein the three-dimensional weight tensor includes G data blocks, each data block is composed of K pruning vectors, and each pruning vector is composed of M weights;

Pruning the three-dimensional weight tensor along the K dimension, resetting the q _i weights with the least importance in the K dimension to zero, so as to obtain the three-dimensional weight vector after the i-th pruning;

Then the three-dimensional weight tensor after the i-th pruning is deformed to obtain a deformed four-dimensional weight tensor;

Training the deformed four-dimensional weight tensor to obtain a four-dimensional weight tensor that has undergone the i-th pruning process;

Among them, the input four-dimensional weight tensor is the first weight tensor or the four-dimensional weight tensor that has undergone the i-1th pruning process.

In the above design, a progressive pruning method is adopted, and a part of the weights are reduced in each generation of training. Compared with pruning a considerable number of weights at one time, the accuracy of the network can be improved.

In a possible design, the shape of the input four-dimensional weight tensor is (C _out ,C _in ,FH,FW), and the shape of the deformed three-dimensional weight tensor is (G, K, M); where G satisfies:

Wherein, C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; C _in *FH*FW=K*M, or C _in =K*M.

In the above design, the embodiment of the present application performs sparse processing on the dimension of C _in or C _in *FH*FW, which can improve the accuracy compared with the outer vector wise (OVW) sparse method on a single vector. The OVW sparse method performs sparse processing on the C _out dimension, and the embodiment of the present application can improve the acceleration efficiency compared with this method.

In one possible design, the q _i weights with the least importance in the K dimension are reset to zero, including:

Reset the q _i weights with the smallest L1 norm in K dimensions to zero; or,

Reset the q _i weights with the smallest L2 norm in K dimensions to zero.

In one possible design, when the training process adjusts all weights, q _i =i; or,

When the training process only adjusts the unpruned weights, q _i =1.

In a second aspect, an embodiment of the present application provides a data processing method, including:

Acquire data to be processed, and generate a vector representation of the data to be processed;

Performing calculations on the vector representation of the data to be processed and the weight tensor included in the neural network to obtain a processing result of the data to be processed;

Among them, the weight tensor of the neural network is obtained through pruning processing, and the weight tensor includes multiple data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruned vectors in the first data block except the first pruned vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruned vector is any pruned vector in the first data block.

In the solution provided by the embodiment of the present application, only one non-zero element is selected at the same position of each K pruning vectors to participate in the calculation. Since the other weights at the same position are all zero elements, the calculation speed can be increased and the power consumption can be reduced when applied to acceleration hardware. The solution provided by the embodiment of the present application can balance accuracy and acceleration.

In one possible design, performing calculations on the vector representation of the data to be processed and the weight tensor included in the neural network includes:

The vector representation of the data to be processed is a first matrix, and the weight tensor is processed to obtain a second matrix and the index of the non-zero elements in the second matrix;

A multiplication operation is performed on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result.

In a possible design, each row element in the second matrix corresponds to the weight of a data block, and the number of column elements in the first matrix is K*M.

In a possible design, performing a multiplication operation on the second matrix and the first matrix to obtain a calculation result according to an index of each non-zero element in the second matrix includes:

Determine the non-zero elements in the first matrix corresponding to the i-th row in the second matrix according to the index of the non-zero elements in the second matrix. The elements of the row operation are used to perform a multiplication operation on the second matrix and the first matrix to obtain a calculation result.

In a possible design, the number of non-zero elements in each data block is M; the index value of the k-th non-zero element in each data block is k-1, which is used to indicate the index of the pruning vector to which the k-th non-zero element belongs;

When the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M.

In a possible design, the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel height of the neural network; the number of column elements of the second matrix is C _out *FH*FW, and the number of row elements of the second matrix is C _in ; C _in =K*M, and the number of data blocks included in the weight tensor is

In a possible design, the shape of the weight tensor is (C _out ,C _in ,FH,FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; the number of column elements of the second matrix is C _out , and the number of row elements of the second matrix is C _in *FH*FW; C _in *FH*FW=K*M, and the number of data blocks included in the weight tensor is

In a third aspect, an embodiment of the present application provides an accelerator, including:

A storage circuit, used to store a first matrix corresponding to the data to be processed, a second matrix corresponding to the weight tensor of the neural network, and the index of the non-zero elements in the second matrix;

Wherein, the weight tensor of the neural network is obtained through pruning processing, and the weight tensor includes multiple data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruned vectors in the first data block except the first pruned vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruned vector is any pruned vector in the first data block;

An operation circuit is used to perform a multiplication operation on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result.

In a possible design, each row element in the second matrix corresponds to the weight of a data block, and the number of column elements in the first matrix is K*M.

In one possible design, the operation circuit includes a selection unit and a calculation unit;

The selection unit is used to read the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements in the second matrix, and read the elements in the i-th column of the first matrix that are operated on the non-zero elements of the second matrix from the storage circuit according to the index of the non-zero elements of the second matrix;

The calculation unit is used to perform multiplication and addition operations on the non-zero elements in the i-th row of the second matrix and the elements in the i-th column of the read first matrix.

In a possible design, the storage circuit includes a storage unit and a selection unit, the storage unit is used to store a first matrix corresponding to the data to be processed, a second matrix corresponding to the weight tensor included in the neural network, and the index of non-zero elements in the second matrix;

The selection unit is used to read the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements in the second matrix, and send the non-zero elements of the i-th row in the second matrix to the operation circuit; and read the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements of the second matrix, and send the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix to the operation circuit;

The operation circuit is specifically used to perform multiplication and addition operations on the non-zero elements in the i-th row of the second matrix and the elements in the i-th column of the first matrix and the non-zero elements in the i-th row of the second matrix.

In a possible design, the number of non-zero elements in each data block is M; the index value of the k-th non-zero element in each data block is k-1, which is used to indicate the index of the pruning vector to which the k-th non-zero element belongs;

When the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M.

In one possible design, the shape of the weight tensor is (C _out , C _in , FH, FW), where C _out represents the output stimulus of the neural network. C _in represents the number of activations of the neural network input, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; the number of column elements of the second matrix is C _out *FH*FW, and the number of row elements of the second matrix is C _in ; C _in =K*M, the number of data blocks included in the weight tensor

In a possible design, the shape of the weight tensor is (C _out ,C _in ,FH,FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel height of the neural network; the number of column elements of the second matrix is C _out , and the number of row elements of the second matrix is C _in *FH*FW; C _in *FH*FW=K*M, and the number of data blocks included in the weight tensor is

In a fourth aspect, an embodiment of the present application provides a processing system, including:

A processor, configured to obtain data to be processed and generate a vector representation of the data to be processed;

An accelerator, configured to perform calculations on the vector representation of the data to be processed and the weight tensor included in the neural network to obtain a processing result of the data to be processed;

Among them, the weight tensor of the neural network is obtained through pruning processing, and the weight tensor includes multiple data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruned vectors in the first data block except the first pruned vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruned vector is any pruned vector in the first data block.

In one possible design, the processor is further used to process the vector representation of the data to be processed into a first matrix, and process the weight tensor to obtain a second matrix and the index of the non-zero elements in the second matrix, and send the first matrix, the second matrix and the index of the non-zero elements in the second matrix to the accelerator;

The accelerator is specifically used to perform a multiplication operation on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result.

In a possible design, each row element in the second matrix corresponds to the weight of a data block, and the number of column elements in the first matrix is K*M.

In one possible design, the accelerator is specifically used to determine the elements in the first matrix to be operated on the non-zero elements in the i-th row of the second matrix according to the indices of the non-zero elements in the second matrix, so as to perform multiplication operations on the second matrix and the first matrix to obtain a calculation result.

In a possible design, the number of non-zero elements in each data block is M; the index value of the k-th non-zero element in each data block is k-1, which is used to indicate the index of the pruning vector to which the k-th non-zero element belongs;

When the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M.

In a possible design, the shape of the weight tensor is (C _out ,C _in ,FH,FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; the number of column elements of the second matrix is C _out *FH*FW, and the number of row elements of the second matrix is C _in ; C _in =K*M, and the number of data blocks included in the weight tensor is

In a possible design, the shape of the weight tensor is (C _out ,C _in ,FH,FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel height of the neural network; the number of column elements of the second matrix is C _out , and the number of row elements of the second matrix is C _in *FH*FW; C _in *FH*FW=K*M, and the number of data blocks included in the weight tensor is

In a fifth aspect, the present application further provides a pruning device, which has the function of implementing the method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

In a possible design, the structure of the pruning device may include an acquisition unit and a pruning unit, which can perform the above For the corresponding functions in the method example, please refer to the detailed description in the method example, which will not be repeated here. The pruning device may also include a training unit for performing a training operation.

In a possible design, the structure of the pruning device may include a processor and a memory, wherein the processor is configured to execute the above-mentioned method. The memory is coupled to the processor and stores necessary program instructions and data for the pruning device.

In a sixth aspect, the present application further provides a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are called by the computer, the computer is used to enable the computer to execute any one of the methods mentioned in the first aspect above, or to execute any one of the methods mentioned in the second aspect above.

In a seventh aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods mentioned in the first aspect above, or execute any of the methods mentioned in the second aspect above.

In an eighth aspect, the present application also provides a chip, which is coupled to a memory and is used to read and execute program instructions stored in the memory to implement any of the methods mentioned in the first aspect above, or to execute any of the methods mentioned in the second aspect above.

Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the convolutional neural network structure;

FIG2 is a schematic diagram of a system architecture 10 provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of a data processing device 200 provided in an embodiment of the present application;

FIG4 is a schematic diagram of a pruning method flow chart provided in an embodiment of the present application;

FIG5 is a schematic diagram of a pruned data block provided in an embodiment of the present application;

FIG6 is a schematic diagram showing the correspondence between sparsity and the number of pruning times provided in an embodiment of the present application;

FIG7 is a schematic diagram of a progressive pruning process provided in an embodiment of the present application;

FIG8A is a schematic diagram of a data processing method provided in an embodiment of the present application;

FIG8B is a schematic diagram of a coding method for non-zero indexes of data blocks provided in an embodiment of the present application;

FIG9A is a schematic diagram of an accelerator structure provided in an embodiment of the present application;

FIG9B is a schematic diagram of another accelerator structure provided in an embodiment of the present application;

FIG10A is a schematic diagram of a processing system structure provided in an embodiment of the present application;

FIG10B is a schematic diagram of another processing system structure provided in an embodiment of the present application;

FIG11A is a schematic diagram of a data processing flow provided in an embodiment of the present application;

FIG11B is a schematic diagram of another data processing flow provided in an embodiment of the present application;

FIG. 12 is a schematic diagram of the structure of a pruning device provided in an embodiment of the present application.

DETAILED DESCRIPTION

Before describing a neural network pruning method, a data processing method, and a device provided in the embodiments of the present application, the related concepts involved in the embodiments of the present application are first described.

(1) Neural Network:

Neural networks are used to implement machine learning, deep learning, search, reasoning, decision making, etc. The neural networks mentioned in this application may include various types, such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), residual networks, neural networks using transformer models or other neural networks, etc. Some neural networks are introduced as examples below.

(2) Convolutional Neural Networks:

Convolutional neural networks (CNN) are deep neural networks with convolutional structures. They are deep learning architectures, which refer to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron processes the data input into it.

As shown in FIG1 , a convolutional neural network (CNN) 100 may include an input layer 110, a convolutional layer/pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130. As shown in FIG1 , the convolutional layer/pooling layer 120 may include layers 121-126, for example. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another ... In one implementation, 121 and 122 are convolution layers, 123 is a pooling layer, 124 and 125 are convolution layers, and 126 is a pooling layer. That is, the output of the convolution layer can be used as the input of the subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation. Taking the convolution layer 121 as an example, the convolution layer 121 can include many convolution operators, which are also called convolution kernels. The convolution operator can essentially be a weight matrix, which is usually predefined. Taking image processing as an example, different weight matrices extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unnecessary noise in the image.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values obtained through training can extract information from the input data, thereby helping the convolutional neural network 100 to make correct predictions.

When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, that is, as shown in the example of 120 in FIG. 1, each layer 121-126 may be a convolution layer followed by a pooling layer, or multiple convolution layers may be followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value. The maximum pooling operator can take the pixel with the largest value in the range within a specific range as the result of the maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

After being processed by the convolution layer/pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. Because as mentioned above, the convolution layer/pooling layer 120 will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (131, 132 to 13n as shown in Figure 1) and an output layer 140. The parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140. The output layer 140 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (as shown in Figure 1, the propagation from 110 to 140 is forward propagation) is completed, the back propagation (as shown in Figure 1, the propagation from 140 to 110 is back propagation) will begin to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

It should be noted that the convolutional neural network 100 shown in Figure 1 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models. For example, multiple convolutional layers/pooling layers are in parallel, and the extracted features are input into the neural network layer 130 for processing.

(3) Pruning/thinning:

Sparsity/pruning is to set the weight values of non-critical parts of a neural network (or neural network) to zero (0). Sparse methods can include arbitrary sparsity, structured sparsity, and unstructured sparsity. Arbitrary sparsity means that the weights chosen to be set to 0 can be at any position, and the inference calculation uses the sparse weight calculation. Structured sparsity is to prune according to certain rules, such as whole channel pruning in convolution, whole column pruning in full connection, and after pruning, the inference calculation still uses dense calculation through rearrangement. Unstructured sparsity (also called semi-structured sparsity) can be understood as between structured sparsity and arbitrary sparsity. Compared with structured sparsity, it has a lower granularity and needs to be pruned according to certain rules, but the rules are relatively weak, and customized code implementation is required to achieve good inference performance. For example, NVIDIA The proposed 2:4 sparseness has two positions with zero weights in the vector of length 4.

(4) Sparsity: the ratio of the number of 0 elements to the total number of elements.

In the description of the present application, unless otherwise specified, "plurality" means two or more than two. In addition, "/" indicates that the objects associated before and after are in an "or" relationship. For example, A/B can represent A or B. The "and/or" in the present application is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference. It should also be noted that, unless otherwise specified, the specific description of some technical features in one embodiment can also be used to explain the corresponding technical features mentioned in other embodiments.

2 is a schematic diagram of a system architecture 10 provided in an embodiment of the present application. The system architecture 10 includes a pruning device 100 and a data processing device 200.

The pruning device 100 is used to execute the pruning method of the neural network provided in the embodiment of the present application, and the data processing device 200 is used to implement the data processing method provided in the embodiment of the present application. After the pruning device 100 completes pruning of the neural network, the pruned neural network can be sent to the data processing device 200.

The pruning device 100 may be implemented by software or hardware.

The pruning device 100 is taken as an example of a software functional unit. The pruning device 100 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the pruning device 100 may include code running on multiple hosts/virtual machines/containers. It should be noted that the multiple hosts/virtual machines/containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts/virtual machines/containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Generally, a region may include multiple AZs.

Similarly, multiple hosts/virtual machines/containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

The pruning device 100 is an example of a hardware functional unit. The pruning device 100 may include at least one computing device, such as a server. The at least one computing device may be deployed in a cloud network or locally. Alternatively, the pruning device 100 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

In some scenarios, the multiple computing devices included in the pruning apparatus 100 may be distributed in the same region or in different regions.

The system architecture 10 also includes a data storage system 300. Optionally, the pruning device 100 cooperates with other computing devices, such as data storage, routers, load balancers, etc. The pruning device 100 can use the data in the data storage system 300, or call the program code in the data storage system 300 to implement the neural network pruning method provided in the present application.

FIG3 exemplarily shows a schematic diagram of the structure of a data processing device 200 provided by the present application. As shown in FIG3 , the data processing device 200 includes a processing system 210, a hardware accelerator 220, a data bus 230, and a control bus 240. The processing system 210 and the hardware accelerator 220 can transmit data through the data bus 230, and can transmit control information through the control bus 240. The processing system 210 is the control component of the entire system, including a processor 211, and optionally, the processing system 210 also includes a memory 212. The processor 211 is used to run the code on the software side and load the acceleration task (such as a matrix or a vector) onto the hardware accelerator 220. Optionally, the hardware accelerator 220 may include one or more intellectual property (IP) cores, and the hardware accelerator 220 may control the working state and data reading of the one or more IP cores. The functions of the IP core include, for example: designing some commonly used but relatively complex function blocks in digital circuits, such as synchronous dynamic random access memory (SDRAM) controllers, peripheral component interconnect (PCI) interfaces, etc., into circuits with modifiable parameters. The processor 211 may be a central processing unit (CPU), a network processor (NP) or a combination of a CPU and a NP. The processor 211 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The memory 212 is used to store data. The processor 211 can control the reading or writing of data to the memory 212. The memory 212 can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a random access memory (RAM), or a non-volatile memory (non-volatile memory), such as a flash memory, a hard disk, or a hard disk. The memory 212 may include a hard disk drive (HDD) or a solid-state drive (SSD). The memory 212 may also include a combination of the above-mentioned types of memories.

The hardware accelerator 220, which can also be referred to as an accelerator, is a hardware acceleration component of the entire system. It can design and implement a dedicated acceleration IP core according to a specific computing task. The IP core on the hardware accelerator can accelerate the algorithm. In the embodiment of the present application, the hardware accelerator 220 can be used in a neural network to accelerate the neural network algorithm. After the data interaction is completed, the hardware accelerator 220 and the processing system 210 can independently execute tasks in parallel. The hardware accelerator 220 can be a graphics processing unit (GPU), or an FPGA, or an integrated circuit (application specific integrated circuit, ASIC) for special applications, or a microcontroller (micro control unit, MCU), etc.

The data bus 230 is used for data transmission between the entire processing system 210 and the hardware accelerator 220. The data bus 230 may adopt the advanced extensible interface-Stream (AXI-Stream) protocol or the bus interface (PCI Express, PCI-E) bus protocol. AXI is a bus protocol, and the AXI-Stream protocol is a high-performance transmission protocol that allows unlimited data burst transmission.

The control bus 240 is used for transmission of control signals between the entire processing system 210 and the hardware accelerator 220. The control bus 40 may adopt the AXI-Lite protocol, which is a lightweight address-mapped single-transmission protocol suitable for transmission of control signals of hardware computing circuits.

In some possible implementations, the processing system 210 may have a hardware acceleration function, that is, the functions of the processing system 210 and the hardware accelerator 220 are implemented by one device, such as a CPU.

The data processing device 200 interacts with the pruning device 100 via a communication network of any communication mechanism/communication standard. The communication network may be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

The solution provided by the embodiment of the present application can be applied to deep learning scenarios, such as scenarios using convolutional neural networks and scenarios using deep neural networks. In addition, the embodiment of the present application is applicable to scenarios of Transformer networks or fusion networks of transformers + CNN.

The following is a description of the process of the neural network pruning method provided in the embodiment of the present application. See Figure 4 for a schematic diagram of the process of the neural network pruning method provided in the embodiment of the present application. The neural network pruning method can be executed by the pruning device 100, or by a module in the pruning device 100.

S401, obtaining a first weight tensor of the neural network to be pruned.

The neural network to be pruned can be a pre-trained neural network or an untrained neural network.

The neural network to be pruned may be a convolutional neural network, such as a neural network used for image classification, target recognition, image processing, image denoising, and the like.

The first weight tensor can be understood as a vector representation of the weight of the convolution kernel. The convolution kernel can be single-channel or multi-channel. For example, the shape of the convolution kernel is (C _out , C _in , FH, FW). C _out represents the number of output activations of the convolution kernel (or the number of output channels of the convolution kernel), C _in represents the number of input activations of the convolution kernel (or the number of input channels of the convolution kernel), FH represents the kernel height of the convolution kernel, and FW represents the kernel height of the convolution kernel.

S402, perform at least one pruning process on the first weight tensor to obtain a second weight tensor.

The second weight tensor includes multiple data blocks, each data block consists of K pruned vectors, and each pruned vector consists of M weights. K and M are both positive integers. The weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the pruned vectors other than the first pruned vector in the first data block is a zero element. The first data block is any data block among the multiple data blocks. The first pruned vector is any pruned vector in the first data block. A data block can also be understood as being composed of multiple pruned vectors. The size of the data block is related to the specifications of the hardware that performs neural network operation acceleration, such as the specifications of the hardware that performs neural network operation acceleration is the length of the vector included in the data block.

The pruning vector may also be referred to as a vector, or as a shortest vector, or may be named by other names. The embodiment of the present application does not specifically limit the name of the pruning vector.

It should be understood that each data block satisfies the conditions satisfied by the first data block, and the position where the non-zero element in a pruning vector is located is a zero element in other pruning vectors in the data block to which the pruning vector belongs. For example, as shown in FIG5, in the pruning vector of M=4, the data block includes two pruning vectors, the weights of the 0th position and the 3rd position of the first pruning vector are non-zero elements, and the weights of the 1st position and the 2nd position of the first pruning vector are zero elements, so that the weights of the 0th position and the 3rd position of the second pruning vector are both zero elements, and the weights of the 1st position and the 2nd position of the second pruning vector are non-zero elements. That is, the positions of the zero elements of the first pruning vector and the second pruning vector are complementary. Based on this, the pruning method provided in the embodiment of the present application can be called complementary sparse or complementary pruning, and of course other names can be used to name it, and the embodiment of the present application does not specifically limit this. For ease of description, the pruning method provided in the embodiment of the present application will be referred to as complementary sparse in the following.

Compared with the single-channel or block-based sparse method, the embodiment of the present application uses a smaller granularity for sparseness, which can improve the accuracy of the application of the network model. Compared with the use of any sparse method, the sparse method is complementary, and there is no need to use dense calculations in the reasoning stage, which reduces the amount of calculation of the model. The solution provided by the embodiment of the present application can balance accuracy and acceleration.

In complementary sparsity, K pruned vectors can be combined into a vector (i.e., data block) for operation, so that the relationship between the sparsity after pruning and K satisfies formula (1):

For example, when K=2, the sparsity S=50%. For another example, when K=4, the sparsity S=75%, and so on.

The embodiment of the present application may adopt a one-step approach when performing pruning, that is, perform pruning once, or may adopt a progressive pruning approach (or progressive thinning approach).

If the progressive pruning method is described, the progressive pruning method is performed N times in a manner of increasing sparsity. In some embodiments, the number K of pruning vectors included in each data block and/or the sparsity of the second weight tensor (i.e., the sparsity required for pruning) can be determined.

For example, the relationship between N and K satisfies the following formula (2):
N=K-1 formula (2).

For another example, the relationship between N and S satisfies the following formula (3):

Taking N=K-1 as an example, the sparsity of each pruning process can satisfy the following formula (4):

Wherein, _Si represents the sparsity of the i-th pruning process, i is a positive integer, 1≤i＜K.

For example, to train a neural network with a sparsity of 75%, after 3 prunings, the sparsity changes as follows: 0->25%->50%->75%. For another example, to train a neural network with a sparsity of 87.5%, after 7 prunings, the sparsity changes as follows: 0->12.5%->25%->37.5%->50%->62.5%->75%->87.5.

It can be understood that for an M*K data block (or matrix), when the progressive sparsity is performed, the increased sparsity is equivalent to subtracting the number of weights included in the shortest vector for each increase in the number of executions by 1. Based on this, the embodiment of the present application can also refer to the progressive sparsity method as vector-by-vector progressive sparsity. Of course, other names can also be used, and the embodiment of the present application is not limited to this.

In the embodiment of the present application, each time pruning is performed, one generation of training (also referred to as epoch) is performed. Based on this, the pruning process mentioned above may include pruning operations and training operations. In some embodiments, when a vector-by-vector progressive sparsity method is adopted, the number of training times at each sparsity value may be the same. As shown in FIG6 , different steps (stages) corresponding to different sparsities are of equal width, that is, the number of training times at each sparsity value is the same. The training corresponding to different sparsity values may use the same training sample set or different training sample sets. For example, the sample set may be split into multiple training sample subsets, and each time a pruning process is performed, a training sample subset is used to perform a generation of training. For example, different training sample subsets include the same number of batches (indicating a batch of data), and the number of training times at each sparsity value is the same. A small portion of samples in the training set is used to perform a back-propagation parameter update on the weights retained by the neural network. This small portion of samples is called a "batch of data."

The pruning process is described below. Taking the i-th pruning as an example, i is less than or equal to N. For example, each pruning operation can be divided into three steps, namely (1) deformation, (2) pruning and (3) restoration (also called inverse deformation or anti-deformation).

(1) Deformation: The input four-dimensional weight tensor is deformed to obtain a deformed three-dimensional weight tensor. The three-dimensional weight tensor includes G data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights.

It can be understood that when pruning is performed for the first time, the input four-dimensional weight tensor can be understood as the first weight tensor. When pruning is performed for the second to Nth times, the input four-dimensional weight tensor can be understood as the four-dimensional weight tensor that has undergone the i-th pruning process.

The shape of the input four-dimensional weight tensor can be described as (C _out , C _in , FH, FW). C _out represents the number of output activations of the convolution kernel (or the number of output channels of the convolution kernel), C _in represents the number of input activations of the convolution kernel (or the number of input channels of the convolution kernel), FH represents the kernel height of the convolution kernel, and FW represents the kernel height of the convolution kernel). Four sets of parameters are used to describe the weight tensor. Of course, in some scenarios, in the case of a single output channel, C _out = 1 in the weight tensor. In this case, the shape of the input weight tensor can be understood as three-dimensional, that is, (C _in , FH, FW). In the case of a single input channel, C _in = 1 in the weight tensor, and the shape of the input weight tensor can be understood as three-dimensional, that is, (C _out , FH, FW).

Taking the shape of the input weight tensor as (C _out ,C _in ,FH,FW) as an example, the shape of the deformed three-dimensional weight tensor is G, K, M); Where G satisfies the following formula (5):

In some scenarios, in order to achieve acceleration on general parallel computing hardware or dedicated acceleration hardware, K*M satisfies the following formula (6):
K*M＝L, L∈{C _in ,C _in *FH*FW} formula (6).

For example, dedicated acceleration hardware may be a circuit or hardware with independent functions that can accelerate the complementary sparsity provided in the embodiments of the present application, such as a dedicated MCU, a dedicated GPU, or a dedicated CPU, etc. The value of L is related to the specifications of the acceleration hardware (dedicated acceleration hardware or general parallel computing hardware). For example, the specifications of the acceleration hardware may be the size of the matrix of the matrix product supported by the acceleration hardware. For example, the acceleration hardware supports the product operation of the matrix A*L and the matrix L*B. For example, in the specifications of the acceleration hardware, L=C _in , and for another example, in the specifications of the acceleration hardware, L=C _in *FH*FW. In some embodiments, L may also be C _in *FH or C _in *FW.

In some embodiments, taking L=C _in as an example, when performing deformation, the input four-dimensional weight tensor can be first rearranged into (FH, FW, C _out , C _in ). The purpose of deformation is to facilitate pruning along the C _in dimension. Further, the rearranged weight tensor is deformed into a three-dimensional weight tensor of (G, K, M). At this time, G=C _out *FH*FW.

In some other embodiments, taking L=C _in *FH*FW as an example, when performing deformation, the input four-dimensional weight tensor can be first rearranged into (C _out , FH, FW, C _in ), and further, the rearranged weight tensor can be deformed into a three-dimensional weight tensor of (G, K, M). In this case, G=C _out .

(2) Pruning: Prune the three-dimensional weight tensor along the K dimension, reset the q _i weights with the least importance in the K dimension to zero, so as to obtain the three-dimensional weight vector after the i-th pruning.

The three-dimensional weight tensor of (G, K, M) can be understood as G K*M data blocks. Each K*M data block is pruned in the K dimension. It can be understood that the q _i weights with the least importance among the K weights in each column are reset to zero.

Exemplarily, the q _i weights with the least importance in the K dimension are reset to zero, which can be:

Reset the q _i weights with the smallest L1 norm in K dimensions to zero; or,

Reset the q _i weights with the smallest L2 norm in K dimensions to zero.

For example, said _qi =i, or _qi =1.

(3) Restoration: deforming the three-dimensional weight tensor after the i-th pruning to obtain a deformed four-dimensional weight tensor, wherein the input four-dimensional weight tensor is the first weight tensor or the four-dimensional weight tensor after the i-1-th pruning.

In a possible implementation, in each pruning process in the embodiment of the present application, the pruning operation may be performed first and then the training operation, or the training operation may be performed first and then the pruning operation. During the training process, the pruned weights may or may not be updated.

In one example, the pruning operation is performed first and then the training operation is performed, and the subtracted _weights are updated as an example.

Before training, the first pruning is performed, i=1, and the sparsity after the first pruning is S ₁ =1/K. It can be understood that when the first pruning is performed, M weights are subtracted from the data block composed of each K pruned vector. For each K*M data block, the least important weight among the K weights in each column is reset to zero, that is, M weights are subtracted from each data block. Then the first generation of training is performed for the pruned weight vector, and each weight is adjusted during the training process. Then continue to perform the second pruning, i=2, and the sparsity after the second pruning is S ₂ =2/K. It can be understood that when the second pruning is performed, 2*M weights are subtracted from the data block composed of each K pruned vector. For each K*M data block, the two least important weights among the K weights in each column are reset to zero, that is, 2*M weights are subtracted from each data block. Then the second generation of training is performed for the pruned weight vector, and each weight is adjusted during the training process. And so on.

It should be noted that after the last pruning, only the weights that have not been pruned can be adjusted during the last generation of training.

In another example, the pruning operation may be performed first and then the training operation may be performed, and the subtracted _weights may not be updated.

Before training, the first pruning is performed, i=1, and the sparsity after the first pruning is S ₁ =1/K. It can be understood that when the first pruning is performed, M weights are subtracted from the data block composed of each K pruned vectors. For each K*M data block, the least important weight among the K weights in each column is reset to zero, that is, M weights are subtracted from each data block. Then the first generation of training is performed on the pruned weight vectors, and the weights that have not been subtracted are adjusted during the training process. Then continue to perform the second pruning, i=2, and the sparsity after the second pruning is S ₂ =2/K. It can be understood that when the second pruning is performed, M weights are continued to be subtracted from the data block composed of each K pruned vectors. For each K*M data block, the least important weight among the K weights in each column except the subtracted weights is reset to zero, and at this time 2*M weights have been subtracted from each data block. Then the second generation of training is performed on the pruned weight vectors During training, the weights that were not lost are adjusted. And so on.

In another example, a training operation may be performed first and then a pruning operation may be performed, and the subtracted _weights may be updated as an example.

Before performing the first pruning, the first generation of training is performed, i=1, and each weight is adjusted during the training process. Then the first pruning operation is performed, and the sparsity after the first pruning is S ₁ =1/K. It can be understood that when performing the first pruning, M weights are subtracted from each data block consisting of K pruned vectors. For each K*M data block, the least important weight among the K weights in each column is reset to zero, that is, M weights are subtracted from each data block. Then the second generation of training is performed for the pruned weight vectors, and all weights are adjusted during the training process. Then continue to perform the second pruning, i=2, and the sparsity after the second pruning is S ₂ =2/K. It can be understood that when performing the second pruning, 2*M weights are subtracted from each data block consisting of K pruned vectors. For each K*M data block, the two least important weights among the K weights in each column are reset to zero, and at this time, 2*M weights have been subtracted in each data block. And so on.

In another example, the training operation may be performed first and then the pruning operation may be performed, and the subtracted _weights may not be updated.

Before performing the first pruning, perform the first generation of training, i=1, and adjust each weight during the training process. Then perform the first pruning operation, and the sparsity after the first pruning is S ₁ =1/K. It can be understood that when performing the first pruning, M weights are subtracted from each data block consisting of K pruned vectors. For each K*M data block, the least important weight among the K weights in each column is reset to zero, that is, M weights are subtracted from each data block. Then perform the second generation of training for the pruned weight vectors, and adjust the weights that have not been subtracted during the training process. Then continue to perform the second pruning, i=2, and the sparsity after the second pruning is S ₂ =2/K. It can be understood that when performing the second pruning, M weights are subtracted from each data block consisting of K pruned vectors. For each K*M data block, the K weights in each column are reset to zero after the weight with the least importance is subtracted. At this time, 2*M weights have been subtracted in each data block. And so on.

For example, referring to FIG7, the progressive pruning process of a tensor with a deformed shape of (1,4,4), it can be seen that as the sparsity increases, the number of retained K dimensions becomes smaller and smaller. It should be noted that FIG7 is only an example. In the actual pruning process, the value of the weight will change after training. FIG7 only illustrates the change in the number of pruning and the change in sparsity.

The data processing method provided by the embodiment of the present application is described below. See FIG. 8A for a schematic diagram of the data processing flow provided by the embodiment of the present application. The data processing method can be executed by the data processing device 200, or by a module in the data processing device 200. The processor in this embodiment can be the processor 211 in FIG. 3, and the accelerator can be the hardware accelerator 220 in FIG. 3.

S801: The processor 211 obtains data to be processed and generates a vector representation of the data to be processed.

The vector representation of the data to be processed may be a representation in matrix form, for example, the vector representation of the data to be processed is a first matrix.

S802, the hardware accelerator 220 performs calculations on the vector representation of the data to be processed and the weight tensor included in the neural network to obtain a processing result of the data to be processed. The weight tensor included in the neural network can be a matrix representation, such as a second matrix.

In some embodiments, the processor 211 may process the vector representation of the data to be processed into a first matrix, process the weight tensor to obtain a second matrix and the index of the non-zero elements in the second matrix, and send the first matrix, the second matrix, and the index of the non-zero elements in the second matrix to the accelerator. Then, the accelerator performs a multiplication operation (also described as a multiplication operation) on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result.

In some embodiments, the hardware accelerator 220 (referred to as accelerator 220) performs calculations on the vector representation of the data to be processed and the weight tensor included in the neural network, which can be understood as multiplying the first matrix of the data to be processed and the second matrix of the neural network.

Among them, the weight tensor of the neural network is obtained through pruning processing, and the weight tensor includes multiple data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruned vectors in the first data block except the first pruned vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruned vector is any pruned vector in the first data block.

In one possible implementation, the weight tensor may be encoded. A pruned weight tensor may be encoded as non-zero weights + indices. When storing, non-zero weights may be compactly stored in the order of the concatenated vectors. Each non-zero weight corresponds to an index. When the accelerator performs calculations on the vector representation of the data to be processed and the weight tensor included in the neural network, the second matrix may be multiplied by the first matrix according to the index of each non-zero weight in the second matrix to obtain a calculation result. In some embodiments, each row element in the second matrix corresponds to the weight of a data block, that is, the number of row elements in the second matrix is K*M. The number of column elements in the first matrix is K*M. When performing matrix multiplication, the second matrix × first matrix method is adopted. In other implementations, each column element in the second matrix may be multiplied by the first matrix. The weight of a data block, that is, the number of column elements of the second matrix is K*M. The number of row elements of the first matrix is K*M. When performing matrix multiplication, the first matrix×the second matrix is used.

In one possible implementation, when the accelerator performs a multiplication operation on the second matrix and the first matrix to obtain a calculation result according to the index of each non-zero element in the second matrix, it can determine the elements in the first matrix that are operated on the non-zero elements in the i-th row of the second matrix according to the index of the non-zero elements in the second matrix, so as to perform a multiplication operation on the second matrix and the first matrix to obtain a calculation result.

As an example, the index of the non-zero elements in the second matrix can be encoded as follows: the number of non-zero elements in each data block is M; the index value of the kth non-zero element in each data block is k-1, which is used to indicate the index of the pruning vector to which the kth non-zero element belongs.

For example, see FIG8B , which is a schematic diagram of the encoding method of the non-zero index of the data block provided in an embodiment of the present application. In FIG8B , taking the sparsity of 50% as an example, K=2, and M=8 as an example. When the non-zero elements of pruning vector 1 and pruning vector 2 are binary-encoded, the weight of the first non-zero element in the M dimension belongs to pruning vector 1, and the index is 0; the weight of the second non-zero element in the M dimension belongs to pruning vector 2, and the index is 1; the weight of the third non-zero element in the M dimension belongs to pruning vector 3, and the index is 1; the weight of the fourth non-zero element in the M dimension belongs to pruning vector 1, and the index is 0, and so on. In FIG8B , taking the sparsity of 75% as an example, K=4, and M=4 as an example. When binary encoding is performed on the non-zero elements of pruning vector 1-pruning vector 4, the weight of the first non-zero element in the M dimension belongs to pruning vector 1, the index is 0, and the binary encoding is 0; the weight of the second non-zero element in the M dimension belongs to pruning vector 3, the index is 2, and the binary encoding is 10; the weight of the third non-zero element in the M dimension belongs to pruning vector 2, the index is 1, and the binary encoding is 01; the weight of the fourth non-zero element in the M dimension belongs to pruning vector 4, the index is 3, and the binary encoding is 11.

Exemplarily, in combination with the above-mentioned encoding method, when determining the elements in the first matrix that are operated on with the non-zero elements in the i-th row of the second matrix according to the indices of the non-zero elements in the second matrix, for example, when the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M.

The shape of the weight tensor is (C _out ,C _in ,FH,FW) for illustration. Exemplarily, the weight tensor is described as a convolution kernel. In one example, taking L=K*M=C _in as an example, the processor can transform a pruned convolution kernel with a shape of (C _out ,C _in ,FH,FW) into a second matrix of (C _out *FH*FW, C _in ). The shape of the data to be processed is (C _in ,IW,IH), and the processor transforms the data to be processed into a matrix of (C _in ,OH*OW). Padding indicates the padding of zero elements. Stride indicates the stride, which is used to reduce the number of input parameters and the amount of calculation. The purpose of padding zero elements is to make all the input blocks serve as the center of the convolution kernel window. The accelerator then performs the operation of the first matrix × the second matrix. The subsequent descriptions all take the operation of the first matrix × the second matrix as an example. For the (i, j)th element w _ij of the convolution kernel matrix (i.e., the second matrix), for example, the w _ij is a non-zero element. When w _ij needs to be multiplied with the corresponding element x _rj of the jth column in the first matrix, when the index value of w _ij in the second matrix is g, r=g*M. Furthermore, after completing the matrix multiplication, the shape of the result matrix is (C _out *FH*FW, OH*OW). Further perform a parallel matrix row reduction operation, that is, the sum of each FH*FW number in the result matrix can be used to obtain a number. After the parallel matrix row reduction operation, the shape of the matrix is (C _out , OW×OH). Then, the matrix with the shape of (C _out , OW×OH) is converted into (C _out , OW, OH), which is the calculation result.

In another example, taking L=K*M=C _in *FW*FH as an example, the processor can transform a pruned convolution kernel of the shape (C _out ,C _in ,FH,FW) into a second matrix of (C _out ,C _in *FH*FW). The shape of the data to be processed is (C _in ,IW,IH), and the processor transforms the data to be processed into a matrix of (C _in *FH*FW,OH*OW). Padding indicates the padding of zero elements. Stride indicates the stride, which is used to reduce the number of input parameters and the amount of calculation. The purpose of padding with zero elements is to make all the input blocks serve as the center of the convolution kernel window. The accelerator then performs the operation of the first matrix × the second matrix. For the (i, j)th element w _ij of the convolution kernel matrix (i.e., the second matrix), for example, the w _ij is a non-zero element. When w _ij needs to be multiplied with the corresponding element x _rj of the jth column in the first matrix, when the index value of w _ij in the second matrix is g, r = g*M. Furthermore, After completing the matrix multiplication, the shape of the result matrix is (OC, OW×OH), which is converted to (OC, OW, OH), which is the calculation result.

As an example, the structure of the accelerator is described below. As shown in Figures 9A and 9B, the accelerator may include a storage circuit 910 and an operation circuit 920. The accelerator may be the hardware accelerator 220 in Figure 3. The storage circuit 910 may be used for the first matrix corresponding to the data to be processed, the second matrix corresponding to the weight tensor of the neural network, and the index of the non-zero elements in the second matrix. The operation circuit 920 is used to perform a multiplication operation on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result.

In some scenarios, not all elements in the input first matrix participate in the operation, but only elements that need to be multiplied with non-zero elements in the second matrix participate in the operation. Therefore, the elements of the first matrix that need to be operated can be selected through a selection circuit (it can also be understood that the first matrix is sparse). The selection circuit can also be called a selection unit. In the subsequent description, the selection circuit is called a selection unit for example.

In a possible implementation, the selection unit is deployed in the operation circuit. Referring to FIG. 9A , exemplarily, the operation circuit 920 may include a selection unit 921 and a calculation unit 922. The selection unit 921 reads the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements in the second matrix, and reads the elements of the i-th column of the first matrix from the storage circuit according to the index of the non-zero elements of the second matrix to be operated with the non-zero elements of the second matrix. The calculation unit 922 is used to perform multiplication and addition operations on the non-zero elements of the i-th row of the second matrix and the elements of the i-th column of the read first matrix. It can be understood that the selection unit 921 selects only one non-zero element for calculation at the same position of the K pruning vectors, and the other elements are all zero elements. The acceleration ratio on the hardware thus achieved is strictly related to the sparsity, for example, the convolution acceleration ratio of 50% sparsity is 2. The convolution acceleration ratio of 75% sparsity is 4, and so on.

In some possible application scenarios, the operation circuit 920 may use one or more arithmetic and logic units (ALUs). For example, the accelerator may be an MCU, a CPU, or a GPU, etc. ALU is the operation unit of the IP core of the accelerator, a data processing component, and the most special data channel unit in the data channel. It implements multiple operations such as addition, subtraction, and, or, XOR, NOT, left shift, right shift, and half-byte exchange. The selection unit 921 may be executed by a separate ALU or embedded in other ALUs that perform operations.

In another possible implementation, the selection unit is deployed in a storage circuit. Exemplarily, as shown in FIG9B , the storage circuit 910 includes a storage unit 911 and a selection unit 912. The storage unit 911 stores a first matrix corresponding to the data to be processed, a second matrix corresponding to the weight tensor included in the neural network, and the index of the non-zero elements in the second matrix. In some scenarios, the storage circuit 910 is also responsible for data reading, and the first matrix can be sparse during the data reading process of the storage circuit.

The selection unit 912 reads the non-zero elements of the i-th row in the second matrix from the storage unit 911 according to the index of the non-zero elements in the second matrix, and sends the non-zero elements of the i-th row in the second matrix to the operation circuit; and reads the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix from the storage unit 911 according to the index of the non-zero elements of the second matrix, and sends the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix to the operation circuit 920. Further, the operation circuit 920 performs multiplication and addition operations on the non-zero elements of the i-th row in the second matrix and the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix.

In some embodiments, taking the example of a selection unit being deployed in a storage circuit 910, that is, the storage circuit 910 includes a selection unit 912, as shown in FIG. 9A, the storage unit 911 in the storage circuit 910 may also include a first data cache area 9111, a second data cache area 9112, and an index information cache area 9113.

The first data cache area 9111 is used to cache data loaded from the processor's memory 212 into the hardware accelerator. For example, the first data cache area 9111 can cache the convolution kernel parameter matrix (including the second matrix) of each layer of the neural network. The second matrix can be compactly stored. The K*M non-zero elements that need to be multiplied with the first matrix are compactly stored, such as row storage, and the non-zero elements of the row are read each time.

The second data buffer area 9112 is used to cache data loaded from the memory 212 into the hardware accelerator. For example, when the hardware accelerator is applied to a neural network, the second data buffer area 9112 can cache the input matrix (including the first matrix). The index information buffer area 9113 can be used to cache the index of the non-zero elements of the second matrix and the index of the elements in the first matrix. The index of the non-zero elements in the second matrix can be used in the manner described above. The elements of the first matrix may include a row index (which can also be understood as a row address) and a column index (which can be understood as a column address).

The selection unit 912 is used to select data at a corresponding position according to the index cached in the index information cache area 9113, for example, according to the index The index of the second matrix cached in the information cache area 9113 determines the element of the first matrix that needs to be multiplied with the non-zero element in the first matrix.

The operation circuit 920 is the core component of the hardware accelerator, and various functional units of the operation circuit are solidified inside. Exemplarily, the operation circuit 920 may include a multiplication unit 9201, an accumulation unit 9202, an excitation function unit 9203 and a vector calculation unit 9204. The multiplication unit is used to multiply the elements of the corresponding positions of the input vector and the matrix to obtain an intermediate value, or to multiply the input matrix and the matrix to obtain an intermediate value. The multiplication unit 9201 can be a matrix and vector multiplication unit or a matrix and matrix multiplication unit. The accumulation unit 9202 is used to accumulate the intermediate values calculated by the multiplication unit 9201 to obtain each gate vector value. The excitation function unit 9203 is used to perform excitation function processing on the gate vector value obtained by the accumulation unit, and the vector calculation unit 9204 is used to calculate each gate vector to obtain the final result.

In some possible implementations, the hardware accelerator may further include an input buffer 923 and an output buffer 924 for temporarily storing input and output data in the hardware accelerator. After the operation circuit 920 completes the calculation, the result may be placed in the output buffer 924 and output, wherein the output buffer 924 may be written back to the processor at one time after the cache is full. The input buffer 923 and the output buffer 924 may be deployed inside the operation circuit 920 or outside the operation circuit 920.

In some embodiments, the hardware accelerator may include a memory access unit 913. The memory access unit 913 may use direct memory access (DMA). The memory access unit 913 may be used for data transmission between the hardware accelerator and other devices. Taking the hardware accelerator in FIG. 3 using the structure shown in FIG. 9A as an example, the software and hardware collaborative framework of the data processing device 200 is shown in FIG. 10A. The memory access unit 913 may be used for data transmission between the hardware accelerator 220 and the memory 212. The memory access unit 913 may be deployed inside the storage circuit 910 or outside the storage circuit 910. FIG. 10A takes the deployment outside the storage circuit 910 as an example.

In some embodiments, the memory access unit 913 adopts DMA as an example. The circuits of the hardware accelerator can be equipped with a DMA to realize parallel data reading. In some embodiments, the hardware accelerator may also include a control interconnection circuit for interconnecting the control signal line between the processing system 210. In conjunction with Figure 10A, under the control of the processor 211 in the processing system 210, the opening of DMA is controlled by the control bus and the data bus respectively, and the DMA transmits the first matrix of the data to be processed in the memory 212 and the second matrix corresponding to the weight tensor and the index of the non-zero elements in the second matrix to the storage circuit 910 in the hardware accelerator 220, such as transferring the second matrix to the first data cache area 9111, transferring the first matrix to the second data cache area 9112, and transferring the index of the non-zero elements of the second matrix to the index information cache unit 9113. After that, the target result is obtained by processing by the operation circuit 920. As shown in Figure 11A, the selection unit 912 reads the j-th row non-zero elements in the second matrix from the first data cache area 9111. It can be understood that each column of non-zero elements is compactly stored in rows in the first data buffer area 9111. The selection unit 912 can read the storage row where the j-th row is located from the first data buffer area 9111. The selection unit 912 reads the index of the non-zero element of the j-th row in the second matrix from the index information buffer area 9113, and then reads the element in the j-th column of the first matrix multiplied with the non-zero element of the j-th row in the second matrix from the second data buffer area 9112 according to the index of the non-zero element of the j-th row. The selection unit 912 can send the read data (including the non-zero element of the j-th row in the second matrix, and the element in the j-th column of the first matrix that needs to be multiplied with the non-zero element of the j-th row) to the operation circuit 920. For example, the selection unit 912 can output the read data to the input buffer area 923 (which can be inside the operation circuit 920 or inside the operation circuit 920). If outside the operation circuit 920, the operation circuit 920 can obtain read data from the input buffer area 923 for operation. For example, the multiplication unit performs multiplication operations on the non-zero elements in the j-th row of the second matrix and the elements in the j-th column of the first matrix that need to be multiplied with the non-zero elements in the j-th row to obtain multiple values, and then the accumulation unit 9202 performs addition operations on the multiple values.

In some other embodiments, taking the case where the selection unit is deployed in the operation circuit, that is, the operation circuit 920 includes a selection unit 921, as shown in FIG. 9B, the storage unit 911 in the storage circuit 910 may also include a first data buffer 9111, a second data buffer 9112, and an index information buffer 9113. The functions of the first data buffer 9111, the second data buffer 9112, and the index information buffer 9113 are as described above, and are not repeated here. Exemplarily, the calculation unit 922 in the operation circuit 920 may include a multiplication unit 9201, an accumulation unit 9202, an excitation function unit 9203, and a vector calculation unit 9204. The functions of the multiplication unit 9201, the accumulation unit 9202, the excitation function unit 9203, and the vector calculation unit 9204 are as described above, and are not repeated here.

In some possible implementations, the hardware accelerator may further include an input buffer 923 and an output buffer 924 for temporarily storing input and output data in the hardware accelerator. After the operation circuit 920 completes the calculation, the result may be placed in the output buffer 924 and output, wherein the output buffer 924 may be written back to the processor at one time after the cache is full. The input buffer 923 and the output buffer 924 may be deployed inside the operation circuit 920 or outside the operation circuit 920. FIG. 9B takes the input buffer 923 and the output buffer 924 deployed outside the operation circuit 920 as an example.

In some embodiments, the hardware accelerator may include a memory access unit 913. The memory access unit 913 may use direct memory access (DMA). The memory access unit 913 may be used for data transmission between the hardware accelerator and other devices. Taking the hardware accelerator in FIG3 adopting the structure shown in FIG9B as an example, the hardware and software collaborative framework of the data processing device 200 is shown in FIG10B. The memory access unit 913 can be used for data transmission between the hardware accelerator 220 and the memory 212. The memory access unit 913 can be deployed inside the storage circuit 910 or outside the storage circuit 910. FIG10B takes the deployment outside the storage circuit 910 as an example.

In some embodiments, the memory access unit 913 adopts DMA as an example. The circuits of the hardware accelerator can be equipped with a DMA to realize parallel data reading. In some embodiments, the hardware accelerator may also include a control interconnection circuit (not shown in FIG. 10B) for interconnecting the control signal line between the processing system 210. In conjunction with FIG. 10B, under the control of the processor 211 in the processor 210, the opening of DMA is controlled by the control bus and the data bus respectively, and the DMA transmits the first matrix of the data to be processed in the memory 212 and the second matrix corresponding to the weight tensor and the index of the non-zero elements in the second matrix to the storage circuit 910 in the hardware accelerator 220, such as transferring the second matrix to the first data cache 9111, transferring the first matrix to the second data cache 9112, and transferring the index of the non-zero elements of the second matrix to the index information cache unit 9113. After that, the target result is obtained by processing the operation circuit 920. Referring to FIG. 11B, the selection unit 921 reads the j-th row non-zero elements in the second matrix from the first data cache 9111. It can be understood that the non-zero elements of each column are compactly stored in rows in the first data buffer area 9111. The selection unit 921 can read the storage row where the j-th row is located from the first data buffer area 9111. The selection unit 921 obtains the index of the non-zero element of the j-th row in the second matrix from the index information buffer area 9113, and then reads the element in the j-th column of the first matrix multiplied with the non-zero element of the j-th row in the second matrix from the second data buffer area 9112 according to the index of the non-zero element of the j-th row. For example, the selection unit 921 can read the elements in the j-th column of the first matrix from the second data buffer area 9112, and each column element in the first matrix is stored in rows in the second data buffer area 9112. The storage circuit 910 can output the storage row where the j-th column in the first matrix is located, the non-zero element of the j-th row in the second matrix, and the index of the non-zero element of the j-th row to the input buffer area 923. Then the selection unit 921 obtains the elements that need to be multiplied with the non-zero elements in the j-th row from the elements in the j-th column of the first matrix according to the index of the non-zero elements in the j-th row. The calculation unit 922 can calculate the elements that need to be multiplied with the non-zero elements in the j-th row and the non-zero elements in the j-th row in the elements in the j-th column of the first matrix according to the selection result of the selection unit 921. For example, the multiplication unit 9201 performs multiplication operations on the non-zero elements in the j-th row of the second matrix and the elements in the j-th column of the first matrix that need to be multiplied with the non-zero elements in the j-th row to obtain multiple values, and then the accumulation unit 9202 performs addition operations on the multiple values.

Based on the above embodiments, see FIG. 12 for a schematic diagram of the structure of a pruning device provided in an embodiment of the present application. The pruning device is used to implement the pruning method shown in FIG. 4. The pruning device may include an acquisition unit 1201 and a pruning unit 1202. The acquisition unit may be used to acquire a first weight tensor of a neural network to be pruned. The pruning unit 1202 performs at least one pruning process on the first weight tensor to obtain a second weight tensor. In some embodiments, the pruning device may further include a training unit 1203. The pruning unit 1202 is specifically used to deform the input four-dimensional weight tensor to obtain a deformed three-dimensional weight tensor, wherein the three-dimensional weight tensor includes G data blocks, each of which is composed of K pruning vectors, and each pruning vector is composed of M weights; the three-dimensional weight tensor is pruned along the K dimension, and the q _i weights with the least importance in the K dimension are reset to zero to obtain the three-dimensional weight vector after the i-th pruning; and then the three-dimensional weight tensor after the i-th pruning is deformed to obtain the deformed four-dimensional weight tensor. The training unit 1203 is used to train the deformed four-dimensional weight tensor to obtain a four-dimensional weight tensor that has undergone the i-th pruning process.

The technical effects achieved by the solutions provided in the embodiments of the present application are described below.

at present In the hardware 2:4 sparse method, there are two positions in the vector of length 4 that are 0. There is no rule between the two positions. Architecture and The method designed for GPUs with architectures is therefore not suitable for general acceleration hardware structures, such as general GPUs or CPUs. The solution provided in the embodiments of the present application can be applied to general parallel computing hardware. Acceleration can also be achieved on dedicated hardware structures. The embodiments of the present application adopt a complementary sparse progressive pruning method, which can improve the accuracy of the trained network compared to a single pruning. In addition, the solution provided in the embodiments of the present application performs sparse processing on the dimensions of C _in or C _in *FH*FW, which can improve accuracy compared to sparse processing on a single vector using an outer vector wise (OVW) sparse method. The OVW sparse method performs sparse processing on the C _out dimension, and the embodiments of the present application can improve acceleration efficiency compared to this method.

In addition, it should be noted that the embodiments of the present application can also be applied to optimization scenarios using semi-structured sparse matrix multiplication. 1×1 convolution can also be understood as matrix multiplication, so any scenario where matrix multiplication is sparse can also use the solution provided by the embodiments of the present application.

It should be understood that the division of the various units of the above device is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated.

In the above embodiments, all or part of the embodiments may be implemented by software, hardware or a combination thereof. When implemented by a software program, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer may be a general-purpose computer, A dedicated computer, computer network, or other programmable device. Instructions can be stored in a computer storage medium or transmitted from one computer storage medium to another computer storage medium. For example, instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer storage media can be any medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more media integrated. The medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical medium (e.g., an optical disk), or a semiconductor medium (e.g., a ROM, an EPROM, an EEPROM, a solid state disk (SSD)), etc.

The present application is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one or more processes of the flowchart and/or one or more boxes of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (33)

A neural network pruning method, characterized by comprising: Obtain the first weight tensor of the neural network to be pruned; Perform at least one pruning process on the first weight tensor to obtain a second weight tensor; The second weight tensor includes a plurality of data blocks, each of which is composed of K pruning vectors, each of which is composed of M weights; K and M are both positive integers; The weight corresponding to the first position in the first pruning vector included in the first data block is a non-zero element, the weight corresponding to the first position in other pruning vectors in the first data block except the first pruning vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruning vector is any pruning vector in the first data block. The method according to claim 1 is characterized in that the sparsity S of the second weight tensor satisfies: S=1-1/K. The method according to claim 1 or 2 is characterized in that the pruning process is performed N times in an increasing sparsity manner. The method as claimed in claim 3 is characterized in that the number of times N that the pruning process is performed is related to the number K of pruning vectors included in the data block and/or the sparsity of the second weight tensor. The method according to claim 4, wherein the pruning process is performed a number of times that: N=S/(1-S), or, N=K-1. The method according to any one of claims 3 to 5, characterized in that N=K-1, and the sparsity of the i-th pruning process satisfies:
Wherein, _Si represents the sparsity of the i-th pruning process, i is a positive integer, 1≤i<K. The method according to any one of claims 3 to 6, wherein performing the i-th pruning process, where i is less than or equal to N, comprises: A deformed three-dimensional weight tensor obtained by deforming an input four-dimensional weight tensor, wherein the three-dimensional weight tensor includes G data blocks, each data block is composed of K pruning vectors, and each pruning vector is composed of M weights; Pruning the three-dimensional weight tensor along the K dimension, resetting the q _i weights with the least importance in the K dimension to zero, so as to obtain the three-dimensional weight vector after the i-th pruning; Then the three-dimensional weight tensor after the i-th pruning is deformed to obtain a deformed four-dimensional weight tensor; Training the deformed four-dimensional weight tensor to obtain a four-dimensional weight tensor that has undergone the i-th pruning process; Among them, the input four-dimensional weight tensor is the first weight tensor or the four-dimensional weight tensor that has undergone the i-1th pruning process. The method of claim 7, wherein the shape of the input four-dimensional weight tensor is (C _out , C _in , FH, FW), and the shape of the deformed three-dimensional weight tensor is (G, K, M); wherein G satisfies:
Wherein, C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; C _in *FH*FW=K*M, or C _in =K*M. The method according to claim 7 or 8, characterized in that resetting the q _i weights with the least importance in the K dimension to zero comprises: Reset the q _i weights with the smallest L1 norm in K dimensions to zero; or, Reset the q _i weights with the smallest L2 norm in K dimensions to zero. The method according to any one of claims 7 to 9, characterized in that when the training process adjusts all weights, said q _i =i; or, When the training process only adjusts the unpruned weights, q _i =1. A data processing method, comprising: Acquire data to be processed, and generate a vector representation of the data to be processed; Performing calculations on the vector representation of the data to be processed and the weight tensor included in the neural network to obtain a processing result of the data to be processed; The weight tensor of the neural network is obtained through pruning, and the weight tensor includes multiple data blocks, each of which is composed of K pruning vectors, and each pruning vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruning vector included in the first data block is a non-zero element, and the weights of other pruning vectors in the first data block except the first pruning vector are non-zero elements. The weight corresponding to the first position in the vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruning vector is any pruning vector in the first data block. The method of claim 11, wherein performing calculations on the vector representation of the data to be processed and the weight tensor included in the neural network comprises: The vector representation of the data to be processed is a first matrix, and the weight tensor is processed to obtain a second matrix and the index of the non-zero elements in the second matrix; A multiplication operation is performed on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result. The method as claimed in claim 12 is characterized in that each row element in the second matrix corresponds to the weight of a data block, and the number of column elements in the first matrix is K*M. The method of claim 13, wherein performing a multiplication operation on the second matrix and the first matrix to obtain a calculation result according to an index of each non-zero element in the second matrix comprises: The elements in the first matrix to be operated on with the non-zero elements in the i-th row in the second matrix are determined according to the indices of the non-zero elements in the second matrix, so as to perform a multiplication operation on the second matrix and the first matrix to obtain a calculation result. The method of claim 14, wherein the number of non-zero elements in each data block is M; the index value of the k-th non-zero element in each data block is k-1, which is used to indicate the index of the pruning vector to which the k-th non-zero element belongs; When the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M. The method according to any one of claims 12 to 15, characterized in that the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel height of the neural network; the number of column elements of the second matrix is C _out *FH*FW, and the number of row elements of the second matrix is C _in ; C _in =K*M, and the number of data blocks included in the weight tensor The method according to any one of claims 12 to 15, characterized in that the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; the number of column elements of the second matrix is C _out , and the number of row elements of the second matrix is C _in *FH*FW; C _in *FH*FW=K*M, and the number of data blocks included in the weight tensor is An accelerator, characterized in that it comprises: A storage circuit, used to store a first matrix corresponding to the data to be processed, a second matrix corresponding to the weight tensor of the neural network, and the index of the non-zero elements in the second matrix; Wherein, the weight tensor of the neural network is obtained through pruning processing, and the weight tensor includes multiple data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruned vectors in the first data block except the first pruned vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruned vector is any pruned vector in the first data block; An operation circuit is used to perform a multiplication operation on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result. The accelerator of claim 18, wherein each row element in the second matrix corresponds to the weight of a data block, and the number of column elements in the first matrix is K*M. The accelerator according to claim 19, wherein the operation circuit comprises a selection unit and a calculation unit; The selection unit is used to read the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements in the second matrix, and read the elements in the i-th column of the first matrix that are operated on the non-zero elements of the second matrix from the storage circuit according to the index of the non-zero elements of the second matrix; The calculation unit is used to perform a calculation on the non-zero elements of the i-th row of the second matrix and the elements of the i-th column of the first matrix read. Row multiplication and addition operation. The accelerator of claim 19, wherein the storage circuit comprises a storage unit and a selection unit, the storage unit being used to store a first matrix corresponding to the data to be processed, a second matrix corresponding to the weight tensor included in the neural network, and an index of non-zero elements in the second matrix; The selection unit is used to read the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements in the second matrix, and send the non-zero elements of the i-th row in the second matrix to the operation circuit; and read the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix from the storage circuit according to the index of the non-zero elements of the second matrix, and send the elements of the i-th column elements of the first matrix that are operated with the non-zero elements of the i-th row in the second matrix to the operation circuit; The operation circuit is specifically used to perform multiplication and addition operations on the non-zero elements in the i-th row of the second matrix and the elements in the i-th column of the first matrix that are operated on the non-zero elements in the i-th row of the second matrix. The accelerator according to claim 20 or 21, characterized in that the number of non-zero elements in each data block is M; the index value of the k-th non-zero element in each data block is k-1, which is used to indicate the index of the pruning vector to which the k-th non-zero element belongs; When the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M. The accelerator according to any one of claims 18 to 22, characterized in that the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; the number of column elements of the second matrix is C _out *FH*FW, and the number of row elements of the second matrix is C _in ; C _in =K*M, and the number of data blocks included in the weight tensor The accelerator according to any one of claims 18 to 22, characterized in that the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel height of the neural network; the number of column elements of the second matrix is C _out , and the number of row elements of the second matrix is C _in *FH*FW; C _in *FH*FW=K*M, and the number of data blocks included in the weight tensor is A processing system, characterized in that it comprises: A processor, configured to obtain data to be processed and generate a vector representation of the data to be processed; An accelerator, configured to perform calculations on the vector representation of the data to be processed and the weight tensor included in the neural network to obtain a processing result of the data to be processed; Among them, the weight tensor of the neural network is obtained through pruning processing, and the weight tensor includes multiple data blocks, each of which is composed of K pruned vectors, and each pruned vector is composed of M weights; K and M are both positive integers; the weight corresponding to the first position in the first pruned vector included in the first data block is a non-zero element, and the weight corresponding to the first position in the other pruned vectors in the first data block except the first pruned vector is a zero element, the first data block is any data block among the multiple data blocks, and the first pruned vector is any pruned vector in the first data block. The system of claim 25, wherein the processor is further configured to process the vector representation of the data to be processed into a first matrix, process the weight tensor to obtain a second matrix and the indices of non-zero elements in the second matrix, and send the first matrix, the second matrix, and the indices of non-zero elements in the second matrix to the accelerator; The accelerator is specifically used to perform a multiplication operation on the second matrix and the first matrix according to the index of each non-zero element in the second matrix to obtain a calculation result. The system as claimed in claim 26 is characterized in that each row element in the second matrix corresponds to the weight of a data block, and the number of column elements in the first matrix is K*M. The system as described in claim 27 is characterized in that the accelerator is specifically used to determine the elements in the first matrix that are operated on the non-zero elements of the i-th row in the second matrix according to the indices of the non-zero elements in the second matrix, so as to perform a multiplication operation on the second matrix and the first matrix to obtain a calculation result. The system of claim 28, wherein the number of non-zero elements in each data block is M; The index value of the kth non-zero element in the data block is k-1, which is used to indicate the index of the pruning vector to which the kth non-zero element belongs; When the index value of the first non-zero element in the second matrix is g, the row r of the element in the first matrix that is multiplied with the first non-zero element satisfies: r=g*M. The system according to any one of claims 26 to 29, characterized in that the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel width of the neural network; the number of column elements of the second matrix is C _out *FH*FW, and the number of row elements of the second matrix is C _in ; C _in =K*M, and the number of data blocks included in the weight tensor The system according to any one of claims 26 to 29, characterized in that the shape of the weight tensor is (C _out , C _in , FH, FW), C _out represents the number of output activations of the neural network, C _in represents the number of input activations of the neural network, FH represents the kernel height of the neural network, and FW represents the kernel height of the neural network; the number of column elements of the second matrix is C _out , and the number of row elements of the second matrix is C _in *FH*FW; C _in *FH*FW=K*M, and the number of data blocks included in the weight tensor is A pruning device, characterized in that it comprises: A memory for storing program instructions; A processor is used to be coupled with the memory, call program instructions in the memory, and execute the method according to any one of claims 1 to 10, or execute the method according to any one of claims 11 to 17. A computer storage medium, characterized in that a computer program is stored in the computer storage medium, and when the computer program is executed by a computer, the computer executes the method according to any one of claims 1 to 10, or the computer executes the method according to any one of claims 11 to 17.