WO2023005085A1 - Method and apparatus for pruning neural network, and device and storage medium - Google Patents

Abstract

A method and apparatus for pruning a neural network, and a device and a storage medium. By means of the method, when a neural network is pruned, a network layer to be pruned can be taken as a target network layer for channel pruning and convolution kernel reconstruction. Therefore, by means of the method, when a multi-branch structure is compressed, compression is not just limited to an intermediate layer, network layers such as an input layer, an output layer and a down-sampling layer can also be compressed, thereby greatly increasing the compression ratio of the neural network, reducing the calculation amount required by a neural network model to execute a task, and increasing the task processing speed of the neural network. In addition, the method is a data-independent asynchronous channel pruning method, and facilitates the maintaining of the robustness of a compressed neural network; and by means of the method, the pruning of different sparse granularities on different network layers of the neural network can also be realized, and the flexibility of compression can also be improved.

Description

A neural network pruning method, device, equipment and storage medium

This application claims the priority of the Chinese patent application submitted to the China Patent Office on July 29, 2021, with the application number 202110866324.3, and the title of the invention is "a neural network pruning method, device, equipment and storage medium", all of which The contents are incorporated by reference in this application.

technical field

The present application relates to the technical field of deep neural network compression and acceleration, and more specifically, to a neural network pruning method, device, device and storage medium.

Background technique

With the increase in depth and width of the neural network, it has shown excellent performance in various AI (Artificial Intelligence, artificial intelligence) application scenarios, such as machine vision tasks such as image recognition and target detection. Faced with the development trend of machine vision software based on deep learning in various embedded devices or mobile devices, deep neural networks with huge parameters are difficult to deploy in devices with limited computing and storage resources. Deep neural network compression and acceleration technology provides a solution for the long-term real-time application of deep learning in these resource-constrained devices. The deep neural network compression technology achieves the purpose of reducing the storage overhead of the neural network model and improving the reasoning speed by reducing the amount of parameters and calculation of the neural network model.

At present, the convolutional neural network has evolved a variety of multi-branch structures, but the pruning schemes for multi-branch structures only cut the input and output channels of the middle layer of the bottleneck structure. For the input and output of the entire multi-branch structure The channel is not compressed, and the number of channels in the middle layer of the bottleneck structure of the multi-branch structure is originally smaller than the number of input and output channels of the entire module. Therefore, only compressing the middle layer of the multi-branch structure is not conducive to improving the compression ratio of the multi-branch structure. .

Contents of the invention

The purpose of this application is to provide a neural network pruning method, device, equipment and storage medium to improve the compression ratio of the neural network, reduce the amount of calculation required for the neural network model to perform tasks, and speed up the task processing speed of the neural network.

In order to achieve the above purpose, the present application provides a pruning method of a neural network, including:

Determine the target network layer to be pruned in the neural network;

Using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel to determine the channel core set of the target network layer; wherein, the channel core set records the reserved input channel;

pruning the original input channel of the target network layer according to the channel kernel set, and reconstructing the convolution kernel of the target network layer.

Wherein, the use of the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel to determine the channel kernel set of the target network layer includes:

determining a second number of reserved input channels of the target network layer according to the first number of original input channels of the target network layer and the channel compression ratio;

Using the first quantity, the second quantity and the Frobenius norm of the convolution kernel weight of each original input channel to determine the importance value of each original input channel;

Determine the sampling probability of each original input channel according to the importance value of each original input channel;

R rounds of independent sampling are performed on each original input channel by using the sampling probability of each original input channel, and a channel core set corresponding to the target network layer is generated according to the sampling result.

Wherein, the Frobenius norm of the convolution kernel weight of the first quantity, the second quantity and each original input channel is used to determine the importance value of each original input channel, including:

determining the first number of weighting coefficients by using the first number and the second number;

Determine the Frobenius norm value of each original input channel according to the convolution kernel weight value of each original input channel, and assign corresponding weighting coefficients to each original input channel according to the Frobenius norm value of each original input channel; wherein, the Frobenius norm The weight coefficient assigned to the original input channel with a larger value is larger;

The importance value of each original input channel is determined according to the weight coefficient of each original input channel and the initial importance function of each original input channel.

Wherein, said reconstructing the convolution kernel of said target network layer includes:

Create an optimization function using the channel core set; the optimization function is:

代表分别计算卷积核的K个输出通道的特征图重构误差并求和，

代表Frobenius范数，W _ik代表卷积核在输入通道i和输出通道k的权值，

为在通道核集

中每个保留输入通道的输入数据x _i经卷积核输出通道k的输出特征图之和，*代表卷积操作； Among them, Y _k is the output feature map of the original convolution kernel in the output channel k, K is the total number of convolution kernel output channels of the target network layer,

Represents the feature map reconstruction errors of the K output channels of the convolution kernel respectively calculated and summed,

Represents the Frobenius norm, Wi _ik represents the weight of the convolution kernel in the input channel i and output channel k,

for the channel kernel set

The sum of the output feature maps of the input data x _i of each reserved input channel through the convolution kernel output channel k, * represents the convolution operation;

The optimization function is minimized, and the weight of the convolution kernel of the target network layer is updated.

In order to achieve the above purpose, the present application further provides a pruning method based on the ResNet downsampling module, including the pruning method in the above scheme; wherein, the determination of the target network layer to be pruned in the neural network includes:

The middle layer and the output layer of the residual branch in the down-sampling module are sequentially used as the target network layer to prune the original input channels of the middle layer and the output layer, and reconstruct the volume of the middle layer and the output layer. Accumulation;

Correspondingly, the pruning method also includes:

Using the channel compression ratio of the input layer of the down-sampling module, random sampling is performed on the original input channel of the input layer to obtain the screening result of the first input channel;

Use the screening result of the first input channel to prune the input layer of the down sampling module and the original input channel of the down sampling layer, and use the input layer and the down sampling layer of the down sampling module as the target network layer respectively Convolution kernel reconstruction.

In order to achieve the above purpose, the present application further provides a pruning method based on the ResNe residual module, including the pruning method in the above scheme; wherein, the ResNet includes stacked N residual modules, and the determined neural The target network layers to be pruned in the network include:

The intermediate layer and the output layer of each residual module in the N residual modules are sequentially used as the target network layer to prune the original input channels of the intermediate layer and the output layer, and reconstruct the intermediate layer and the output layer The convolution kernel;

Correspondingly, after pruning the original input channels of the intermediate layer and the output layer, and reconstructing the convolution kernels of the intermediate layer and the output layer, the pruning method further includes:

Using the channel compression ratio of the input layer of the residual module, random sampling is performed on the original input channel of the input layer of the residual module to obtain the screening result of the second input channel;

For each residual module in the N residual modules, the original input channel of the input layer of each residual module is pruned by using the screening result of the second input channel, and the residual module The input layer of the input layer is used as the target network layer to perform convolution kernel reconstruction; the original output channel of the output layer of the residual module is pruned using the screening result of the second input channel, and the output layer of the residual module is pruned Convolution kernel reconstruction is performed as the target network layer.

In order to achieve the above object, the present application further provides a pruning method based on SqueezeNet, including the pruning method in the above scheme; wherein, if the target Fire module of the SqueezeNet is pruned, then in the determined neural network The target network layers to be pruned include:

The Squeeze layer of the next Fire module of the target Fire module is used as the target network layer to prune with the original input channel of the Squeeze layer of the next Fire module, and reconstruct the volume of the Squeeze layer of the next Fire module Accumulation;

Correspondingly, the pruning method also includes:

According to the channel core set of the Squeeze layer of the next Fire module, the original output channels of the convolution kernels of different sizes in the Expand layer of the target Fire module are pruned;

Utilize the channel compression ratio of the Expand layer of described target Fire module, carry out random sampling to the original input channel of the Expand layer of described target Fire module, obtain the 3rd input channel screening result;

Utilize the screening result of the third input channel to prune the original input channels of the convolution kernels of different sizes in the Expand layer of the target Fire module, and use the Expand layer of the target Fire module as the target network layer, to Expand The convolution kernels of different sizes in the layer are reconstructed.

In order to achieve the above object, the present application further provides a neural network pruning device, including:

The network layer determination module is used to determine the target network layer to be pruned in the neural network;

A channel kernel set determination module, used to determine the channel kernel set of the target network layer by using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel; wherein, the channel kernel set records In order to reserve the input channel;

A channel pruning module, configured to prune the original input channel of the target network layer according to the channel core set;

The convolution kernel reconstruction module is used to reconstruct the convolution kernel of the target network layer.

In order to achieve the above purpose, the present application further provides an electronic device, including:

memory for storing computer programs;

A processor configured to implement the steps of the above pruning method when executing the computer program.

To achieve the above object, the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above pruning method are implemented.

It can be seen from the above scheme that a neural network pruning method provided by the embodiment of the present application includes: determining the target network layer to be pruned in the neural network; using the channel compression ratio of the target network layer and the volume of each original input channel The product kernel weight determines the channel kernel set of the target network layer; among them, the reserved input channel is recorded in the channel kernel set; the original input channel of the target network layer is pruned according to the channel kernel set, and the convolution of the target network layer is reconstructed nuclear.

It can be seen that when this scheme prunes the neural network, the network layer to be pruned can be used as the target network layer for channel pruning and convolution kernel reconstruction. Therefore, when this scheme compresses the multi-branch structure, it does not only Limited to the middle layer, it can also compress the input layer, output layer, downsampling layer and other network layers, which greatly improves the compression ratio of the neural network, reduces the amount of calculation required by the neural network model to perform tasks, and speeds up the tasks of the neural network Processing speed; and, this scheme is a data-independent pruning method, which is beneficial to maintain the robustness of the compressed neural network. This scheme is also an asynchronous channel pruning method, which can be implemented in different network layers of the neural network Pruning with different sparse granularity improves the flexibility of compression; this application also discloses a neural network pruning device, equipment and storage medium, which can also achieve the above technical effects.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flow diagram of a pruning method for a neural network disclosed in an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a downsampling module disclosed in an embodiment of the present application;

Fig. 3 is a schematic diagram of the overall pruning process of the downsampling module disclosed in the embodiment of the present application;

FIG. 4 is a schematic structural diagram of the residual module disclosed in the embodiment of the present application;

Fig. 5 is a schematic diagram of the overall process of residual module pruning disclosed in the embodiment of the present application;

Fig. 6 is a schematic structural diagram of the Fire module disclosed in the embodiment of the present application;

Fig. 7 is a schematic diagram of the overall flow of Fire module pruning disclosed in the embodiment of the present application;

Fig. 8 is a schematic structural diagram of a neural network pruning device disclosed in an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

This application proposes an asynchronous pruning method for convolutional neural networks based on the kernel set theory, which can use the selection criteria based on the kernel set theory layer by layer in the forward reasoning process of the neural network to cut channels, and optimize the feature map weight The structure error directly obtains the new weight of the compressed convolution kernel, and an asynchronous compression process is designed for multi-branch structures such as the ResNet residual module. branch. It should be noted that the core set theory described in this embodiment specifically refers to offline and streaming corset constructions (OSCC), which provides support for the derivation and proof of various formulas in the core set construction process , this scheme is modified on the basis of OSCC to realize the construction of the channel core set.

The method described in this application performs filter-level pruning on each convolutional layer of the image recognition network trained on the classified image. Different from the existing filter-level pruning method, the method of this application is implemented in the form of asynchronous channel pruning Filter-level pruning instead of direct filter-level pruning. In this application, the convolution kernel is defined as a four-dimensional tensor K×C×H×W with K output channels, C input channels and size H×W, where K×1×H×W corresponds to a specific The convolution kernel parameters of the input channel, 1×C×H×W correspond to the convolution kernel parameters of a specific output channel. When pruning a convolution kernel’s input and output channels in an asynchronous manner, the filter 1 can be realized. ×1×H×W pruning. The method described in this application adopts kernel set theory when selecting channels to be pruned, constructs a kernel set for input or output channels through weighted random sampling, and realizes data-independent channel pruning to ensure the robustness of the compressed image recognition network.

Referring to Fig. 1, a schematic flow chart of a neural network pruning method provided in the embodiment of the present application; referring to Fig. 1, the pruning method specifically includes the following steps:

S101. Determine the target network layer to be pruned in the neural network;

Specifically, the neural network in this embodiment can be a convolutional neural network applied to tasks such as image recognition. Since the parameters of the neural network have great redundancy, the convolutional neural network is pruned through this scheme. Branches can reduce the amount of calculation required for image recognition and speed up image recognition. The target network layer in this embodiment can be any network layer to be pruned in the neural network, can be a network layer in a single-branch network structure, or can be a network layer in a multi-branch structure, such as: input layer, middle layer, The output layer, down-sampling layer, etc. are not specifically limited here.

S102. Using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel to determine the channel core set of the target network layer; the channel core set records the reserved input channels;

It should be noted that the channel compression ratio in this embodiment is: the ratio of the number of input channels before compression to the number of input channels after compression. Through the channel compression ratio, the number of retained input channels after compression can be determined, for example: The number of original input channels of the target network layer is 50, and the channel compression ratio is 10:5, so the number of input channels that the target network layer needs to retain is 25. The channel kernel set in this embodiment records the input channels that need to be reserved. The specific method of determining the channel kernel set is not specifically limited here. It can be randomly selected from the original input channel, or it can be selected according to the convolution kernel of the target network layer. The weight value can be selected, or it can be selected according to the importance value of each original input channel and so on.

S103. Prune the original input channels of the target network layer according to the channel kernel set, and reconstruct the convolution kernel of the target network layer.

In this embodiment, after determining the channel core set of the target network layer, the original input channels recorded in the channel core set need to be retained, and the original input channels not recorded in the channel core set need to be removed, so as to achieve the original input channel pruning. Moreover, after the input channel is pruned, the convolution kernel of the target network layer needs to be reconstructed to update the weight of the convolution kernel. The convolution kernel reconstruction process does not require retraining the network, and only implemented during the forward inference of the network.

It should be noted that the heuristic screening rules currently used in saliency-based pruning algorithms usually depend on the importance of feature maps or convolution kernel weights, which correspond to data-driven pruning Algorithms are independent of data

There are two categories of (data-independent) pruning algorithms. Existing studies have shown that neural network compression will affect the generalization of the compressed network model. Even if the average accuracy of the compressed network model is similar to that of the uncompressed complete network, the compressed network model is not good in a certain category of samples or in a certain category. The generalization ability on some categories will decrease. Data-independent pruning algorithms can provide robust compression for arbitrary test data. For example, the compression method DINP (Data-Independent Neural Pruning via Coresets) uses the offline and streaming corset constructions (OSCC) to construct a core set for the hidden layer neurons of the fully connected layer, and the core sets in the core set Hidden layer neurons are regarded as important neurons to be retained and the weights of neurons are updated. In addition, DINP gives the neuron importance measurement rules based on the upper bound of the activation function value and the sampling probability of the kernel set. During the construction of the kernel set, the input data of the network layer and the weighting coefficient of each neuron sampled Computation and allocation are independent, so the neuron kernel set constructed by DINP has the property of being independent of data. In this application, instead of using feature maps for channel pruning, channel pruning is performed through convolution kernel weights. Therefore, this scheme is a data-independent pruning method, which is conducive to maintaining the robustness of the compressed neural network. sex.

In summary, this scheme discloses a structured pruning scheme. When pruning the neural network, the scheme can use the network layer to be pruned as the target network layer for channel pruning and convolution kernel reconstruction. It can compress the single-branch structure, and can also compress the multi-branch structure; when this scheme compresses the multi-branch structure, it is not limited to the middle layer of the module, but also the input layer, output layer, and downsampling of the module. Layers and other network layers are compressed, so as to realize the pruning of the input channel and output channel of the entire module, which greatly improves the compression ratio of the neural network, reduces the amount of calculation required for the neural network model to perform tasks, and speeds up the task processing of the neural network speed; and, this solution prunes different network layers in an asynchronous manner, so the asynchronous channel pruning method disclosed in this solution can realize pruning with different sparse granularity at different network layers, improving the flexibility of compression , based on channel pruning at some network layers, filter-level pruning at other network layers.

Based on the above-mentioned embodiment, in this embodiment, the process of determining the channel kernel set of the target network layer by using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel specifically includes:

Step 1: Determine the second number of reserved input channels of the target network layer according to the first number of original input channels of the target network layer and the channel compression ratio;

Step 2: Using the first quantity, the second quantity and the Frobenius norm of the convolution kernel weight of each original input channel to determine the importance value of each original input channel;

Step 3: Determine the sampling probability of each original input channel according to the importance value of each original input channel;

Step 4: Use the sampling probability of each original input channel to perform R rounds of independent sampling for each original input channel, and generate a channel core set corresponding to the target network layer according to the sampling results.

In this embodiment, if the target network layer is set as the l-th layer network layer, then the first number of the original input channels of the target network layer can also be understood as the number of output channels of the l-1-th layer network layer, in this embodiment Among them, the first number of original input channels of the target network layer is represented by n _l-1 , and the second number of reserved input channels of the target network layer is represented by a _l .

In this embodiment, when determining the importance value of each original input channel in step 2, the first number and the second number are used to determine the first number of weighting coefficients, and each weight coefficient is determined according to the convolution kernel weight value of each original input channel. The Frobenius norm value of the original input channel, and assign a corresponding weighting coefficient to each original input channel according to the Frobenius norm value of each original input channel; wherein,

The weight coefficient assigned to the original input channel with the larger Frobenius norm value is larger, and then the importance value of each original input channel is determined according to the weight coefficient of each original input channel and the initial importance function of each original input channel.

In this embodiment, n _l-1 weighting coefficients w _i (x) can be calculated by the following formula:

Among them, w _i (x) is the i-th weighting coefficient, which is the weighting coefficient constructed by the non-uniform sampling of each channel, and the w _i (x) has no correlation with the original input channel, and

Further, in order to associate n _l-1 weighting coefficients w _i (x) with n _l-1 original input channels, it is necessary to calculate the n _l-1 original input channel convolution kernel weights of the l-th layer convolution kernel The Frobenius norm value of the value, sort the Frobenius norm values of each original input channel according to the numerical order, and sort the n _l-1 weighting coefficients according to the numerical order, according to the sorting of the two, assign the weighting coefficient with the larger value Give the original input channel with a larger Frobenius norm value. After assigning the corresponding weighting coefficient to each original input channel, the importance value of each original input channel can be calculated according to the following formula:

s _i (x) = w _i (x) g _i (x) (2)

其中，i表示第i个通道，x _ij是当前批次输入数据x中某个数据x _j的第i通道，

是Frobenius范数，N是线性点卷积层的输入通道总数；w _i(x)为第i个原输入通道的加权系数。每个原输入通道的重要性值确定后，本实施例在步骤三根据如下公式3确定每个原输入通道的采样概率p _i： Among them, in Formula 2, s _i (x) is the importance function of the i-th original input channel, which is used to determine the importance value of the i-th original input channel, g _i (x) is the i-th original input channel In this embodiment, the initial importance function can be customized according to different neural networks, such as: for ResNet and SqueezeNet, g _i (x)=1 can be set, for mobilenet-v2 network Linear point-wise convolution layer (linear point-wise convolution layer), can be set:

Among them, i represents the i-th channel, x _ij is the i-th channel of a certain data x _j in the current batch of input data x,

is the Frobenius norm, N is the total number of input channels of the linear point convolution layer; w _i (x) is the weighting coefficient of the i-th original input channel. After the importance value of each original input channel is determined, this embodiment determines the sampling probability p _i of each original input channel according to the following formula 3 in step 3:

p _i =s _i (x)/t (3)

Among them, t is the sum of all channel importances, namely

而未被添加至通道核集

的原输入通道需要被剪枝。 In this embodiment, after determining the sampling probability of each original input channel, it is necessary to perform R rounds of independent sampling according to the sampling probability of each original input channel through step 4 to obtain the sampling result, which is the sampling result of each original input channel When the number of samples is reached, the greater the sampling probability of the original input channel, the more times it is sampled; after the sampling result is obtained, the al original input channels that have been sampled the most times in the sampling result can be added to the channel core set

without being added to the channel core set

The original input channel of needs to be pruned.

After channel pruning is performed on the target network layer, the convolution kernel of the target network layer needs to be reconstructed. In this embodiment, the process of reconstructing the convolution kernel of the target network layer is to update the weight of the convolution kernel process, which first uses the channel kernel set to create an optimization function; the optimization function is:

代表分别计算卷积核的K个输出通道的特征图重构误差并求和，

代表Frobenius范数，W _ik代表卷积核在输入通道i和输出通道k的权值，

为在通道核集

中每个保留输入通道的输入数据x _i经卷积核输出通道k的输出特征图之和，*代表卷积操作； Among them, Y _k is the output feature map of the original convolution kernel in the output channel k, K is the total number of convolution kernel output channels of the target network layer,

Represents the feature map reconstruction errors of the K output channels of the convolution kernel respectively calculated and summed,

Represents the Frobenius norm, Wi _ik represents the weight of the convolution kernel in the input channel i and output channel k,

for the channel kernel set

The sum of the output feature maps of the input data x _i of each reserved input channel through the convolution kernel output channel k, * represents the convolution operation;

Then the optimization function is minimized, and the weights of the convolution kernels of the target network layer are updated.

It should be noted that the current neural network pruning algorithm designs heuristic screening rules to select parameters, channels, filters and other objects for pruning, and compensates for the loss of accuracy caused by compression through training or feature map reconstruction. In addition to the accuracy rate and compression ratio of the network model, the evaluation criteria of the neural network pruning algorithm also has construction efficiency. Therefore, the pruning algorithm using the feature map reconstruction method in this embodiment can improve the neural network compression process. own operating efficiency.

To sum up, it can be seen that when pruning the passage, this scheme uses the characteristics of the convolution kernel instead of the characteristics of the feature map to construct the sampling probability of channel sampling, so it realizes the data-independent pruning method, which is conducive to maintaining the compressed The robustness of the neural network; and, this scheme reconstructs the convolution kernel by minimizing the output feature map reconstruction error corresponding to the channel kernel set, that is, the convolution kernel reconstruction is realized during the forward reasoning process of the network without the need for Retrain the network, save the reconstruction time of the convolution kernel, and improve the operating efficiency of the neural network compression process itself.

It should be noted that when pruning a network layer with a single-branch structure, the structured pruning method described in the above embodiment can be directly used to prune each network layer in a single-branch structure, but for For the multi-branch structure, in addition to pruning each network layer in the multi-branch structure using the structured pruning method described in the above embodiment, additional processing is required according to the special features of the multi-branch structure. In this embodiment In , the pruning process of the multi-branch structure is illustrated by taking the down-sampling module and residual module of ResNet and the Fire module of SqueezeNet as examples.

1. If the down-sampling module of ResNet is pruned, when executing S101 in the above embodiment, the solution needs to use the middle layer and the output layer of the residual branch in the down-sampling module as the target network layer in turn, and continue Execute S102 to S103 to prune the original input channels of the middle layer and the output layer, and reconstruct the convolution kernels of the middle layer and the output layer;

Correspondingly, due to the particularity of the down-sampling module, in this embodiment, the pruning method further includes: using the channel compression ratio of the input layer of the down-sampling module, randomly sampling the original input channel of the input layer to obtain the first Input channel screening results; use the first input channel screening results to prune the input layer of the down-sampling module and the original input channel of the down-sampling layer, and use the input layer and down-sampling layer of the down-sampling module as the target network layer for convolution Accumulation refactoring.

In this embodiment, for the convenience of description, the pruning method described in the above embodiments is referred to as the pruning method based on kernel set theory, and this solution is applicable to any network layer in the downsampling module, residual module, and Fire module The process of reconstructing the convolution kernel is the same as the process of reconstructing the target network layer described in the above embodiment, and will not be repeated here. Referring to Figure 2, it is a schematic structural diagram of the down-sampling module provided by the embodiment of the present application. The middle layer, output layer and input layer in Figure 2 belong to the residual branch. When compressing the down-sampling module, this solution first adopts the above-mentioned embodiment The described pruning method based on kernel set theory performs input channel pruning on the middle layer and the output layer of the residual branch, and reconstructs the corresponding convolution kernel, and then the input layer and downsampling of the residual branch The input channel of the layer is pruned according to the pruning method based on random sampling, and the convolution kernel is reconstructed; among them, the pruning method based on random sampling specifically refers to: according to the corresponding channel compression ratio, the original input channel is once Random sampling with equal probability to obtain the corresponding input channel screening results and perform pruning. For the convenience of description, this process will be referred to as the pruning method based on random sampling for short, and the process will not be described in detail.

Referring to FIG. 3 , it is a schematic diagram of the overall flow chart of the pruning of the down-sampling module provided in the embodiment of the present application. The overall flow chart specifically includes the following steps:

S201. The pruning method based on kernel set theory screens and prunes the original input channel of the middle layer of the residual branch, and reconstructs the convolution kernel of the output layer;

S202. The pruning method based on kernel set theory screens and prunes the original input channel of the output layer of the residual branch, and reconstructs the convolution kernel of the output layer;

S203. The pruning method based on random sampling screens the original input channel of the input layer of the residual branch, and obtains the first input channel screening result;

S204. Prune the input layer of the residual branch and the original input channel of the downsampling layer according to the screening result of the first input channel, and reconstruct the input layer of the residual branch and the convolution kernel of the downsampling layer.

It should be noted that since the input layer of the residual branch and the downsampling layer have the same input, after channel screening is performed on the input layer of the residual branch to obtain the channel kernel set, the channel kernel set can be directly applied to the downsampling layer. The downsampling layer of the module does not need to be recalculated.

2. If the N residual modules stacked in ResNet are pruned, when this solution executes S101 in the above embodiment, the middle layer and the output layer of each residual module in the N residual modules need to be pruned Serve as the target network layer in turn, and continue to execute S102 to S103, so as to prune the original input channel of the middle layer and the output layer, and reconstruct the convolution kernel of the middle layer and the output layer;

Correspondingly, due to the particularity of the residual module, in this embodiment, after pruning the original input channels of the intermediate layer and the output layer, and reconstructing the convolution kernels of the intermediate layer and the output layer, it also includes: using the residual The channel compression ratio of the input layer of the difference module is used to randomly sample the original input channel of the input layer of the residual module to obtain the second input channel screening result; for each residual module in the N residual modules, use the first The original input channel of the input layer of each residual module is pruned according to the screening result of the second input channel, and the input layer of the residual module is used as the target network layer for convolution kernel reconstruction; The original output channel of the output layer of the difference module is pruned, and the output layer of the residual module is used as the target network layer for convolution kernel reconstruction.

Referring to Figure 4, it is a schematic diagram of the structure of the residual module provided by the embodiment of the present application. The residual module includes: an input layer, an intermediate layer, an output layer and a shortcut; and, Figure 4 only records a residual module structure. For stacked residual The difference module is composed of multiple structures as shown in FIG. 4 . When compressing the stacked residual modules, the input layers of multiple residual modules can share the channel core set obtained by the pruning method based on random sampling, and keep the channels with the same number. When the difference module is compressed, the input channels of the intermediate layer and the output layer of the stacked residual modules are firstly pruned based on the kernel set theory and the convolution kernel is reconstructed, and then the pruning method is based on random sampling. A residual module obtains the filtering result of the input channel, and uses the filtering result of the input channel to prune and reconstruct the convolution kernel of the input layer and output layer of multiple stacked residual modules in turn, that is, the residual The output layer of the difference module performs two kernel reconstructions.

If the total number of residual modules is N, i and i' represent the i-th and i'th residual modules currently being pruned, and i and i' are 1 in the initial case. Referring to FIG. 5 , it is a schematic diagram of the overall process flow of the residual module pruning provided by the embodiment of the present application. The overall flow chart specifically includes the following steps:

S301. A pruning method based on kernel set theory, screening and pruning the original input channel of the middle layer of the i-th residual module, and reconstructing the convolution kernel of the middle layer;

S302. A pruning method based on kernel set theory, screening and pruning the original input channel of the output layer of the i-th residual module, and reconstructing the convolution kernel of the output layer;

S303. Determine whether i is less than or equal to N; if so, add 1 to i, and continue from S301 to S302; if not, continue to execute S304;

S304. Based on the pruning method of random sampling, the original input channel of the input layer of the i'th residual module is screened to obtain the screening result of the second input channel; the initial value of i' is 1;

S305. Prune the original input channel of the input layer of the i'th residual module according to the screening result of the second input channel, and reconstruct the convolution kernel of the input layer;

S306. Prune the original output channel of the output layer of the i'th residual module according to the screening result of the second input channel, and reconstruct the convolution kernel of the output layer.

S307. Determine whether i' is less than or equal to N; if so, add 1 to i', and continue to execute S305 to S306; if not, end the process.

It should be noted that since the residual module includes shortcuts, S306 is to prune the original output channels of the shortcut branch of the output layer. Since the output channels of the output layer change, the output layer is then pruned according to the number of channels after the change. The convolution kernel is reconstructed.

3. If the target Fire module of SqueezeNet is pruned, then this solution needs to use the Squeeze layer of the next Fire module of the target Fire module as the target network layer when executing S101 in the above embodiment, and continue to execute S102 to S103 , to use the original input channel of the Squeeze layer of the next Fire module for pruning, and reconstruct the convolution kernel of the Squeeze layer of the next Fire module;

Correspondingly, due to the particularity of the Fire module, in this embodiment, the pruning method also includes:

According to the channel core set of the Squeeze layer of the next Fire module, the original output channels of the convolution kernels of different sizes in the Expand layer of the target Fire module are pruned;

Using the channel compression ratio of the Expand layer of the target Fire module, the original input channel of the Expand layer of the target Fire module is randomly sampled to obtain the third input channel screening result;

Use the filtering results of the third input channel to prune the original input channels of the convolution kernels of different sizes in the Expand layer of the target Fire module, and use the Expand layer of the target Fire module as the target network layer, and perform convolutions of different sizes in the Expand layer The accumulation kernel is refactored.

Referring to FIG. 6 , it is a schematic structural diagram of the Fire module provided by the embodiment of the present application. The Fire module includes: a Squeeze layer and an Expand layer; and, the Expand layer has two convolution kernels of different sizes. The process of channel pruning for the Fire module in this solution is as follows: the pruning method based on kernel set theory prunes and reconstructs the original input channel of the Squeeze layer of the Fire module i+1, and then reconstructs the convolution kernel of the Fire module i. The output channel of the expand layer is pruned, and then the input channel of the expand layer of the Fire module i is pruned and the convolution kernel is reconstructed based on the random sampling pruning method. Moreover, in the input channel pruning of the squeeze layer, the input channel screening result of the squeeze layer of the i+1 Fire module is divided into two parts, corresponding to the 3×3 volumes of the expand layer of the i-th Fire module The output channel of the product and 1×1 convolution; for the input channel pruning of the i-th Fire module expand layer, the input channel screening results are shared between the 3×3 convolution and the 1×1 convolution.

Referring to FIG. 7 , it is a schematic diagram of the overall process flow of Fire module pruning provided by the embodiment of the present application. The overall flow chart specifically includes the following steps:

S401, the pruning method based on the core set theory, screening the original input channels of the Squeeze layer of the Fire module i+1, generating a channel core set, and performing channel pruning on the Squeeze layer of the Fire module i+1 according to the channel core set branch and convolution kernel reconstruction.

S402. Divide the channel core set into two parts, corresponding to the original output channels of convolution kernels of different sizes in the expand layer of the Fire module i, and pruning the original output channels.

It should be noted that although there is a corresponding relationship between the input channel of the Squeeze layer of Fire module i+1 and the output channel of the expand layer of Fire module i, in the pruning process, the two layers need to be respectively processed through S401 and S402. The respective convolution kernels are pruned.

S403. Based on the pruning method of random sampling, the original input channel of the expand layer of Fire module i is screened to generate a channel core set, and the input of convolution kernels of different sizes in the expand layer of Fire module i is input according to the channel core set. Channels are pruned and convolution kernels of different sizes are reconstructed.

To sum up, the channel pruning method described in this scheme implements filter-level pruning through input and output channel pruning in an asynchronous manner, and designs the pruning process according to the multi-branch structure characteristics of ResNet and SqueezeNet, which solves the problem of When compressing ResNe and SqueezeNett, the existing pruning method only prunes the input and output channels of the residual module, downsampling module and the middle layer of the Fire module, and does not compress the input and output channels of the entire module. A higher compression ratio can be achieved, the calculation amount of the network in the forward reasoning process can be reduced, and pruning with different sparsity granularity can be realized in different layers of the network. In addition, the channel selection rule based on kernel set theory designed by the method of the present application has data-independent characteristics, which is conducive to maintaining the robustness of the compressed image recognition network.

Furthermore, this solution can also be deployed in FPGA-based neural network acceleration applications or AI acceleration chip software platforms. This solution can prune the image recognition network with a high compression ratio, reducing the amount of calculation of the image recognition network in real-time image classification applications. This technical solution can also be extended and applied to the compression of the backbone network of the target detection network, such as YOLO or Faster RCNN.

The pruning device, equipment and medium provided in the embodiments of the present application are introduced below, and the pruning device, equipment and medium described below and the pruning method described above can be referred to each other.

Referring to FIG. 8 , a schematic structural diagram of a neural network pruning device provided in an embodiment of the present application. It can be seen from FIG. 8 that the device includes:

The network layer determination module 11 is used to determine the target network layer to be pruned in the neural network;

The channel kernel set determination module 12 is used to determine the channel kernel set of the target network layer by using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel; wherein, in the channel kernel set Records reserved input channels;

A channel pruning module 13, configured to prune the original input channel of the target network layer according to the channel core set;

The convolution kernel reconstruction module 14 is configured to reconstruct the convolution kernel of the target network layer.

Wherein, the channel core set determination module includes:

A first determining unit, configured to determine a second number of reserved input channels of the target network layer according to the first number of original input channels of the target network layer and the channel compression ratio;

The second determination unit is used to determine the importance value of each original input channel by using the first quantity, the second quantity and the Frobenius norm of the convolution kernel weight of each original input channel;

The third determination unit is used to determine the sampling probability of each original input channel according to the importance value of each original input channel;

The channel core set generating unit is used to perform R rounds of independent sampling on each original input channel by using the sampling probability of each original input channel, and generate a channel core set corresponding to the target network layer according to the sampling result.

Wherein, the second determination unit includes:

A weighting coefficient determining subunit, configured to determine the first number of weighting coefficients by using the first number and the second number;

The weighting coefficient allocation subunit is used to determine the weight of each original input channel according to the convolution kernel weight of each original input channel

Frobenius norm value, and assign corresponding weighting coefficients to each original input channel according to the Frobenius norm value of each original input channel; wherein, the weighting coefficient assigned to the original input channel with the larger Frobenius norm value is larger;

The importance value determining subunit is configured to determine the importance value of each original input channel according to the weighting coefficient of each original input channel and the initial importance function of each original input channel.

Wherein, the convolution kernel reconstruction module includes:

A function creation unit, configured to utilize the channel core set to create an optimization function; the optimization function is:

代表分别计算卷积核的K个输出通道的特征图重构误差并求和，

代表Frobenius范数，W _ik代表卷积核在输入通道i和输出通道k的权值，

为在通道核集

中每个保留输入通道的输入数据x _i经卷积核输出通道k的输出特征图之和，*代表卷积操作； Among them, Y _k is the output feature map of the original convolution kernel in the output channel k, K is the total number of convolution kernel output channels of the target network layer,

Represents the feature map reconstruction errors of the K output channels of the convolution kernel respectively calculated and summed,

Represents the Frobenius norm, Wi _ik represents the weight of the convolution kernel in the input channel i and output channel k,

for the channel kernel set

The sum of the output feature maps of the input data x _i of each reserved input channel through the convolution kernel output channel k, * represents the convolution operation;

A weight updating unit, configured to minimize the optimization function, and update the weight of the convolution kernel of the target network layer.

Wherein, if the down-sampling module of ResNet is pruned, the network layer determination module is specifically used to: use the middle layer and the output layer of the residual branch in the down-sampling module as the target network layer in turn, with Prune the original input channel of the middle layer and the output layer, and reconstruct the convolution kernel of the middle layer and the output layer;

Correspondingly, the device also includes:

The first screening module is configured to use the channel compression ratio of the input layer of the down-sampling module to randomly sample the original input channel of the input layer to obtain the first input channel screening result;

The channel pruning module is also used to: use the screening result of the first input channel to prune the input layer of the down sampling module and the original input channel of the down sampling layer;

The convolution kernel reconstruction module is also used for: reconstructing the convolution kernels of the input layer and the down-sampling layer.

Wherein, if the N residual modules stacked in ResNet are pruned, the network layer determination module is specifically used to: use the intermediate layer and the output layer of the N residual modules as the target network layer in turn, so as to The original input channels of the middle layer and the output layer are pruned, and the convolution kernels of the middle layer and the output layer are reconstructed;

Correspondingly, the device also includes:

The second screening module is used to pruning the original input channel of the middle layer and the output layer, and after reconstructing the convolution kernel of the middle layer and the output layer, using the channel compression ratio of the input layer of the residual module, the residual The original input channel of the input layer of the difference module is randomly sampled to obtain the screening result of the second input channel;

The channel pruning module is also used to: use the screening results of the second input channel to sequentially prune the original input channels of the input layer of the N residual modules, and prune the original output channels of the output layer of the N residual modules pruning;

The convolution kernel reconstruction module is also used to: reconstruct the convolution kernel of the input layer, and reconstruct the convolution kernel of the output layer.

Wherein, if the target Fire module of SqueezeNet is pruned, the network layer determination module is specifically used to: use the Squeeze layer of the next Fire module of the target Fire module as the target network layer to utilize the next Fire The original input channel of the Squeeze layer of the module is pruned, and the convolution kernel of the Squeeze layer of the next Fire module is reconstructed;

Correspondingly, the device also includes:

The second screening module is used to utilize the channel compression ratio of the Expand layer of the target Fire module to randomly sample the original input channel of the Expand layer of the target Fire module to obtain the third input channel screening result;

The channel pruning module is also used to: use the third input channel screening result to prune the original input channels of the convolution kernels of different sizes in the Expand layer of the target Fire module;

The convolution kernel reconstruction module is also used for: reconstructing convolution kernels of different sizes in the Expand layer of the target Fire module.

Referring to FIG. 9, it is a schematic structural diagram of an electronic device disclosed in the embodiment of the present application. It can be seen from FIG. 9 that the electronic device includes:

Memory 21, used to store computer programs;

The processor 22 is configured to implement the steps of the neural network pruning method described in any method embodiment above when executing the computer program.

In this embodiment, the device may be a PC (Personal Computer, personal computer), or may be a terminal device such as a smart phone, a tablet computer, a palmtop computer, or a portable computer.

The device may include a memory 21 , a processor 22 and a bus 23 .

Wherein, the memory 21 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card type memory (for example, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The storage 21 may be an internal storage unit of the device in some embodiments, such as a hard disk of the device. Memory 21 may also be an external storage device of the device in other embodiments, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card) etc. Further, the memory 21 may also include both an internal storage unit of the device and an external storage device. The memory 21 can not only be used to store application software and various data installed in the device, such as program codes for executing the pruning method, but also can be used to temporarily store data that has been output or will be output.

Processor 22 can be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor or other data processing chip in some embodiments, is used for running the program code stored in memory 21 or processing Data, such as the program code to execute the pruning method, etc.

The bus 23 may be a peripheral component interconnect standard (PCI for short) bus or an extended industry standard architecture (EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 9 , but it does not mean that there is only one bus or one type of bus.

Further, the device can also include a network interface 24, and the network interface 24 can optionally include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are usually used for communication between the device and other electronic devices Establish a communication connection.

Optionally, the device may further include a user interface 25, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 25 may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. Wherein, the display may also be properly referred to as a display screen or a display unit, and is used for displaying information processed in the device and for displaying a visualized user interface.

FIG. 9 only shows a device with components 21-25. Those skilled in the art can understand that the structure shown in FIG. 9 does not constitute a limitation on the device, and may include fewer or more components than shown in the figure. Or combine certain components, or different component arrangements.

The embodiment of the present application also discloses a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the clipping of the neural network described in any of the above-mentioned method embodiments is realized. The steps of the stick method.

Wherein, the storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc., which can store program codes. medium.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

A kind of pruning method of neural network, is characterized in that, comprises: Determine the target network layer to be pruned in the neural network; Using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel to determine the channel core set of the target network layer; wherein, the channel core set records the reserved input channel; pruning the original input channel of the target network layer according to the channel kernel set, and reconstructing the convolution kernel of the target network layer. The pruning method according to claim 1, wherein the channel compression ratio of the target network layer and the convolution kernel weights of each original input channel are used to determine the channel core set of the target network layer, include: determining a second number of reserved input channels of the target network layer according to the first number of original input channels of the target network layer and the channel compression ratio; Using the first quantity, the second quantity and the Frobenius norm of the convolution kernel weight of each original input channel to determine the importance value of each original input channel; Determine the sampling probability of each original input channel according to the importance value of each original input channel; R rounds of independent sampling are performed on each original input channel by using the sampling probability of each original input channel, and a channel core set corresponding to the target network layer is generated according to the sampling result. The pruning method according to claim 2, wherein the Frobenius norm of the convolution kernel weight of the first quantity, the second quantity and each original input channel is used to determine each original Enter the importance value of the channel, including: determining the first number of weighting coefficients by using the first number and the second number; Determine the Frobenius norm value of each original input channel according to the convolution kernel weight value of each original input channel, and assign corresponding weighting coefficients to each original input channel according to the Frobenius norm value of each original input channel; wherein, the Frobenius norm The weight coefficient assigned to the original input channel with a larger value is larger; The importance value of each original input channel is determined according to the weight coefficient of each original input channel and the initial importance function of each original input channel. The pruning method according to claim 1, wherein said reconstruction of the convolution kernel of said target network layer comprises: Create an optimization function using the channel core set; the optimization function is:

Among them, Y _k is the output feature map of the original convolution kernel in the output channel k, K is the total number of convolution kernel output channels of the target network layer,

Represents the feature map reconstruction errors of the K output channels of the convolution kernel respectively calculated and summed,

Represents the Frobenius norm, Wi _ik represents the weight of the convolution kernel in the input channel i and output channel k,

for the channel kernel set

The sum of the output feature maps of the input data x _i of each reserved input channel through the convolution kernel output channel k, * represents the convolution operation; The optimization function is minimized, and the weight of the convolution kernel of the target network layer is updated. A pruning method based on the ResNet down-sampling module, characterized in that it includes the pruning method described in any one of claims 1 to 4; wherein, the target network layer to be pruned in the determined neural network includes : The middle layer and the output layer of the residual branch in the down-sampling module are sequentially used as the target network layer to prune the original input channels of the middle layer and the output layer, and reconstruct the volume of the middle layer and the output layer. Accumulation; Correspondingly, the pruning method also includes: Using the channel compression ratio of the input layer of the down-sampling module, random sampling is performed on the original input channel of the input layer to obtain the screening result of the first input channel; Use the screening result of the first input channel to prune the input layer of the down sampling module and the original input channel of the down sampling layer, and use the input layer and the down sampling layer of the down sampling module as the target network layer respectively Convolution kernel reconstruction. A pruning method based on a ResNe residual module, characterized in that it comprises the pruning method described in any one of claims 1 to 4; wherein, the ResNet includes stacked N residual modules, so The above-mentioned determination of the target network layer to be pruned in the neural network includes: The intermediate layer and the output layer of each residual module in the N residual modules are sequentially used as the target network layer to prune the original input channels of the intermediate layer and the output layer, and reconstruct the intermediate layer and the output layer The convolution kernel; Correspondingly, after pruning the original input channels of the intermediate layer and the output layer, and reconstructing the convolution kernels of the intermediate layer and the output layer, the pruning method further includes: Using the channel compression ratio of the input layer of the residual module, random sampling is performed on the original input channel of the input layer of the residual module to obtain the screening result of the second input channel; For each residual module in the N residual modules, the original input channel of the input layer of each residual module is pruned by using the screening result of the second input channel, and the residual module The input layer of the input layer is used as the target network layer to perform convolution kernel reconstruction; the original output channel of the output layer of the residual module is pruned using the screening result of the second input channel, and the output layer of the residual module is pruned Convolution kernel reconstruction is performed as the target network layer. A kind of pruning method based on SqueezeNet, it is characterized in that, comprises the pruning method described in any one of above-mentioned claims 1 to 4; Wherein, if the target Fire module of described SqueezeNet is pruned, then described determination The target network layers to be pruned in the neural network include: The Squeeze layer of the next Fire module of the target Fire module is used as the target network layer to prune with the original input channel of the Squeeze layer of the next Fire module, and reconstruct the volume of the Squeeze layer of the next Fire module Accumulation; Correspondingly, the pruning method also includes: According to the channel core set of the Squeeze layer of the next Fire module, the original output channels of the convolution kernels of different sizes in the Expand layer of the target Fire module are pruned; Utilize the channel compression ratio of the Expand layer of described target Fire module, carry out random sampling to the original input channel of the Expand layer of described target Fire module, obtain the 3rd input channel screening result; Utilize the screening result of the third input channel to prune the original input channels of the convolution kernels of different sizes in the Expand layer of the target Fire module, and use the Expand layer of the target Fire module as the target network layer, to Expand The convolution kernels of different sizes in the layer are reconstructed. A neural network pruning device, characterized in that it comprises: The network layer determination module is used to determine the target network layer to be pruned in the neural network; A channel kernel set determination module, used to determine the channel kernel set of the target network layer by using the channel compression ratio of the target network layer and the convolution kernel weight of each original input channel; wherein, the channel kernel set records In order to reserve the input channel; A channel pruning module, configured to prune the original input channel of the target network layer according to the channel core set; The convolution kernel reconstruction module is used to reconstruct the convolution kernel of the target network layer. An electronic device, characterized in that it comprises: memory for storing computer programs; A processor, configured to implement the steps of the neural network pruning method according to any one of claims 1 to 7 when executing the computer program. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the neural network according to any one of claims 1 to 7 is realized. Steps of the pruning method.

Images