WO2024230358A1 - Model processing method, electronic device and medium - Google Patents

Abstract

The present application relates to the technical field of computers. Disclosed are a model processing method, an electronic device and a medium. The model processing method comprises: after weightings of a trained convergence model are pruned to obtain a pruned model, recovering among the pruned weightings some important weightings affecting model precision, and applying the recovered important weightings to the pruned model so as to retrain the model until a converged model is obtained. Thus, the precision of models can be ensured while the models are compressed.

Description

Model processing method, electronic device and medium

This application claims the priority of the Chinese patent application filed with the China Patent Office on May 8, 2023, with application number 202310514419.8 and application name “A model processing method, electronic device and medium”. The entire contents of the above application are incorporated into this application by reference.

Technical Field

The present application relates to the field of computer technology, and in particular to a model processing method, electronic equipment and medium.

Background Art

In recent years, with the development of Convolution Neural Network (CNN), the network depth has become deeper and deeper, and the number of parameters has increased, making it difficult to deploy large convolutional neural networks on small edge devices. Pruning, as a commonly used method for compressing models, is mainly divided into structured pruning and unstructured pruning. Among them, structured pruning uses the filter in the convolutional layer of the convolutional neural network as the pruning unit, changes the filter group and the number of feature channels in the network, and reduces the width of the convolutional neural network. Unstructured pruning uses a single weight in the convolutional neural network as the pruning unit and performs fine-grained pruning.

In traditional unstructured pruning methods, the model parameters are pruned to the greatest extent possible so that the pruned model can be deployed on small edge devices. However, the large-scale reduction of parameters will cause the model to become highly sparse, resulting in a reduction in the model's representation capabilities. For example, some features of the feature graph cannot be extracted, which in turn reduces the accuracy of the model.

Summary of the invention

In order to solve the problem mentioned above that the existing unstructured pruning method will cause the model to become highly sparse, resulting in a reduction in the model's representation ability, such as causing some features of the feature graph to be unable to be extracted, thereby resulting in a reduction in the accuracy of the model, the present application provides a model processing method, electronic device and medium.

In a first aspect, the present application provides a model processing method, which can be used in electronic devices, the model processing method comprising obtaining a first neural network model, wherein the first neural network model is obtained by pruning a neural network model to be pruned, and pruning comprises modifying a first weight set of the neural network model to be pruned to a second weight set of the first neural network model, wherein a non-zero first weight in the first weight set corresponds to a value of zero in the second weight set. A target weight set that meets the weight condition is selected from the second weight set. Parameter adjustment processing is performed on the target weight set in the second weight set to obtain a third weight set, wherein the value of the first weight in the third weight set is non-zero. The third weight set is used for the first neural network model.

Based on the above scheme, after obtaining the pruned model, the third weight set in the pruned weights of the converged model can be determined, that is, the important weights that can improve the model accuracy, and then the important weights are grown and reapplied to the pruned model, and the model is retrained based on the restored important weights and the unpruned weights in the pruned model until the model converges again, and the converged model is obtained. In this way, the model memory can be optimized to a certain extent while effectively ensuring the model accuracy.

It can be understood that modifying the first weight set of the neural network model to be pruned to the second weight set of the first neural network model may refer to setting the first weight set of the pruned neural network model to zero, that is, achieving pruning of the first weight set.

It can be understood that the target weight set in the second weight set may refer to the important weights screened out from the pruned weights mentioned in the present application.

It can be understood that growing the important weights may be restoring the important weights. For example, the weights that are set to zero may be adjusted to a positive number, or the weights that are set to zero may be adjusted to a negative number.

In some optional instances, a target weight set that meets the weight condition is selected from the second weight set, including: restoring any weight in the second weight set to obtain a candidate neural network model; obtaining the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model; and determining the target weight set based on the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model after restoring each weight in the second weight set.

It can be understood that the weight condition can be that when the weight is applied to the pruned model alone, the current loss function corresponding to the model has a larger gradient (i.e., a decrease) than the loss function corresponding to the pruned model before application.

In some optional examples, obtaining the loss function corresponding to the first neural network model includes: determining the first loss function based on the real information corresponding to the training data set and the predicted information output by the first neural network model; The candidate neural network model determines the second loss function; and the loss function corresponding to the first neural network model is determined based on the first loss function and the second loss function.

It can be understood that the first loss function can represent the accuracy of the first neural network model in achieving the preset task, and the second loss function can represent the degree of difference between the weight matrix in the first network model to be pruned and the target low-rank approximation matrix corresponding to the weight matrix.

It can be understood that when the preset network model to be pruned is a neural network model used to implement different tasks, the first loss function used to represent the accuracy of the first neural network model in implementing the preset task can be different. Commonly used first loss functions can include 0-1 loss function, absolute value loss function, logarithmic loss function, square loss function, exponential loss function, cost loss function, cross entropy loss function, etc.

In some optional instances, obtaining a first loss function corresponding to the first neural network model includes: inputting a training data set into the first neural network model to obtain prediction information output by the first neural network model; and obtaining the first loss function based on real information and prediction information corresponding to the training data set.

In some optional instances, obtaining a second loss function corresponding to the first neural network model includes: obtaining a set of weight matrices in the network model to be pruned; determining a target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set; and obtaining the second loss function corresponding to the first neural network model based on each weight matrix and the target low-rank approximation matrix corresponding to each weight matrix.

In some optional instances, the target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set is determined, including: normalizing each weight matrix to obtain a weight matrix to be decomposed corresponding to each weight matrix; performing low-rank decomposition processing on the weight matrix to be decomposed corresponding to each weight matrix to obtain a target low-rank approximation matrix corresponding to each weight matrix.

It can be understood that the target low-rank approximation matrix can be a low-rank decomposition of each weight matrix in the pruned network model, and a matrix composed of the multiplication of multiple sub-weight matrices with small parameters, simplicity and low rank can be obtained.

It can be understood that by determining the target low-rank approximation matrix corresponding to each weight matrix, the degree of difference between each weight matrix and the target low-rank approximation matrix in the first neural network model is determined to determine the important weights, which can improve the rank of the weight matrix, reduce the omission of feature extraction, and thus improve the accuracy of the model.

In some optional instances, pruning the neural network model to be pruned to obtain a first neural network model includes: pruning the network model to be pruned based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate to obtain the first neural network model.

In some optional instances, the network model to be pruned is pruned based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate to obtain a first neural network model, including: based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate, determining the weights corresponding to the preset sparsity rate in the network model to be pruned as pruning weights; setting the values corresponding to the pruning weights to zero to obtain the first neural network model.

For example, the network model to be pruned has m weights, and when the preset sparsity rate corresponding to the current processing is α, the m weights can be sorted from large to small based on the amplitude (absolute value) of the m weights, and the m×α weights in the sequence are determined as the weights in the first weight subset in order from large to small. The weights in the first weight set are pruned, that is, all the weights in the first weight set are set to zero, and the first neural network model after the network model to be pruned is obtained.

It can be understood that by pruning the network model to be pruned, the number of parameters of the network model to be pruned can be reduced. For example, reducing the parameter amount m to m-m×α can improve the model sparsity rate, so that the pruned model can be deployed on a small edge device.

In some optional instances, pruning the neural network model to be pruned to obtain a first neural network model includes: pruning the network model to be pruned based on the amplitude of each weight in the network model to be pruned, a preset sparsity rate, and a preset growth rate to obtain the first neural network model.

In some optional instances, a neural network model to be pruned is pruned to obtain a first neural network model, including: based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate, determining the weights corresponding to the preset sparsity rate in the network model to be pruned as a first weight subset; based on the amplitude of each weight other than the first weight subset in the network model to be pruned and a preset growth rate, determining the weights corresponding to the preset growth rate in the network model to be pruned other than the first weight subset as a second weight subset; setting the values corresponding to the weights in the first weight subset and the second weight subset to zero to obtain the first neural network model.

For example, the network model to be pruned has m weights, the preset sparsity rate corresponding to the next process is α, and the preset growth rate corresponding to the next process is β. The m weights can be sorted from large to small based on the amplitude (absolute value) of the m weights, and the m×α weights in the sequence are determined as the weights in the first weight subset in order from large to small. Then, the weights can be sorted based on the amplitude (absolute value) of the m×α non-zero weights. The unset-to-zero weights are sorted from large to small (value), and the m×β weights in the sequence are also determined as the weights in the second weight subset in order from large to small. Then, the union of the first weight subset and the second weight subset can be determined as the first weight set, and the weights in the first weight set can be pruned, that is, all the weights in the first weight set are set to zero, so that on the basis of pruning the first weight subset corresponding to the preset sparsity rate, the second weight subset is additionally pruned to obtain the first neural network model after the pruned network model is pruned.

It can be understood that by pruning the network model to be pruned, the number of parameters of the network model to be pruned can be reduced. For example, reducing the parameter amount m to m-m×α-m×β can further improve the model sparsity rate, so that the pruned model can be deployed on a small edge device.

In a second aspect, the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the model processing method mentioned in the present application.

In a third aspect, the present application provides a readable storage medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the model processing method mentioned in the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a schematic diagram of an application scenario according to some embodiments of the present application;

FIG2 is a schematic diagram of a framework of a pruning method for a model;

FIG3 is a flow chart of a pruning method that can be applied to model processing;

FIG4 is a schematic diagram of a model processing method provided in an embodiment of the present application;

FIG5 is a flow chart of a method for determining a target weight set provided in an embodiment of the present application;

FIG6 is a schematic diagram of a process for obtaining a second loss function of a network model to be pruned provided in an embodiment of the present application;

FIG7 is a curve diagram showing the computing power consumption and accuracy of the comprehensive model of the model processing method provided in an embodiment of the present application;

FIG. 8 shows a schematic diagram of the hardware structure of an electronic device.

DETAILED DESCRIPTION

The illustrative embodiments of the present application include but are not limited to a model-based processing method, an electronic device, and a medium.

It can be understood that the model processing method provided in the embodiments of the present application can be used in neural network models used in tasks such as image recognition, target detection, reinforcement learning, and semantic analysis. For example, the types of neural network models are not limited to convolutional neural networks, Transformer (a model based on a multi-head attention mechanism), recurrent neural networks (RNN), long short-term memory neural networks (LSTM), and other arbitrary neural network models.

As shown in FIG1 , in a specific implementation, the server 10 can prune the neural network model 11 for image recognition and then send it to the small edge device 20, so that the small edge device 20 can perform data processing based on the pruned neural network model. The small edge device can be any implementable electronic device such as a mobile phone, a camera, etc.

The following introduces the pruning methods of the models mentioned in some embodiments.

FIG2 is a schematic diagram of a framework of a pruning method for a model. As shown in FIG2 , the pruning method may include:

First, a feature map is obtained. For example, a sampled image can be input into a convolutional neural network, and feature extraction processing can be performed on the sampled image to obtain a feature map. A convolutional neural network can include a convolutional layer and a fully connected layer.

Then, a convolution layer is selected to perform pruning on it. For example, for the current convolution layer in the convolutional neural network, each filter performs filtering on the input first feature map and outputs a second feature map. Among them, the first feature map is the feature map output by the previous convolution layer of the current convolution layer. The filter, the first feature map and the second feature map are in one-to-one correspondence. The matrix rank corresponding to the second feature map is determined based on the parameters in the second feature map. It can be understood that when a matrix is an m×n matrix, the maximum rank of the matrix is the smaller of m and n, expressed as min(m,n). When the matrix rank is relatively large, it can be proved that there is a lot of redundant information in the feature map. The matrix rank corresponding to the second feature map output by each filter is sorted, and the convolutional neural network is pruned based on the sorting result, and the filter corresponding to the second feature map whose matrix rank is less than the preset matrix rank is cut off. Optionally, all the parameters in the filter can be set to zero. By cutting off the filter corresponding to the feature map with a small matrix rank, that is, setting all the parameters in the weight matrix corresponding to the filter that extracts redundant information to zero, the extraction of redundant information can be reduced in the subsequent process of extracting image features using the model with the cut filter.

Next, the pruned convolutional neural network is fine-tuned to obtain the final convolutional neural network for prediction.

Since this pruning scheme uses the feature graph as a whole to determine the matrix rank to prune the filter, it can only be applied to the structured pruning of the model. It cannot be applied to the unstructured pruning of the model, that is, it cannot achieve fine-grained pruning of the model and cannot optimize the memory optimization of the model.

Fig. 3 is a flow chart of a pruning method of a model. As shown in Fig. 3, the pruning method may include:

The weight of each filter in each convolution layer of the convolution neural network is obtained. For example, the weight W = {W _l ¹ ,...,W _n ¹ } of the filter in the l-th convolution layer can be obtained. The rank of the three-dimensional tensor of each filter weight is calculated to obtain the rank R = {R _l ¹ ,...,R _n ¹ } of the three-dimensional tensor of the filter weight. The weights of the filters are sorted according to the rank of the three-dimensional tensor of the filters to obtain the sorting result. For example, the weight W' = {W _l ¹ ,...,W _n ¹ } of the l-th filter is obtained. Based on the sorting result, the filters whose ranks are less than the preset ranks are pruned to obtain the pruned convolution neural network W = {W _m ¹ ,...,W _n ¹ }. After the pruning is completed, the convolution neural network is fine-tuned, and then the next convolution layer is pruned. For example, after the l-th convolution layer is pruned, the l+1-th convolution layer can be pruned. Since this pruning scheme uses the three-dimensional tensor of the filter weights as a whole to determine the rank to prune the filter, it cannot be applied to the unstructured pruning of the model, that is, it cannot achieve fine-grained pruning of the model and cannot achieve optimal memory optimization of the model.

In order to solve the problem that the above-mentioned pruning of filters by determining the rank based on the feature map as a whole and the rank based on the three-dimensional tensor of the filter weights as a whole cannot be applied to unstructured pruning, and thus cannot be applied to fine-grained unstructured pruning, in some embodiments, two unstructured pruning methods are provided, namely, a method based on a preset pruning process and a method based on a preset pruning standard.

Among them, the method based on the preset pruning process can achieve better pruning effects through the pruning process of preset weight pruning and preset growth weight. The preset pruning process includes gradual pruning, "from sparse to sparse" pruning that randomly initializes a sparse network, "compression/decompression" pruning that alternately trains the network in dense and sparse states, and so on. For example, gradual pruning performs pruning once every ΔT training iterations, slowly increases the sparsity rate of the model until the target sparsity rate is reached and pruning ends. The preset pruning standard selects redundant weights from the weight matrix for pruning by presetting the weight importance.

The method based on preset pruning standards implements pruning through preset pruning standards, wherein the pruning standards generally include those based on the absolute value of the weight, based on the weight gradient, based on the importance score of the weight obtained by self-learning, and the like.

However, both the method based on the preset pruning process and the method based on the preset pruning standard achieve pruning by pruning the model parameters to the greatest extent possible. A large-scale reduction in the number of parameters will cause the model to become highly sparse, resulting in a reduction in the model's representation ability. For example, some features of the feature map cannot be extracted, thereby reducing the accuracy of the model.

In order to solve the above problems, the present application provides a model processing method. After obtaining a converged model and pruning the weights in the converged model, the important weights in the pruned weights of the converged model that can improve the model accuracy can be determined. Then, the important weights are grown, that is, the important weights are restored and reapplied to the pruned model. The model is retrained based on the restored important weights and the unpruned weights in the pruned model until the model converges again, and the converged model is obtained. In this way, the model memory can be optimized to a certain extent while effectively ensuring the model accuracy.

Among them, each important weight can be a weight with a larger gradient (i.e., a decrease range) of the current loss function corresponding to the model and the loss function corresponding to the pruned model before the weight is applied alone when the weight is applied alone to the pruned model. It can be understood that the larger the gradient, the smaller the loss function value obtained after the weight is applied alone to the pruned model, that is, the higher the degree of model convergence, that is, the higher the accuracy of the model. For example, selecting a preset number of weights with larger corresponding gradients among the pruned weights as important weights can effectively ensure the accuracy of the model.

The important weights can be determined by restoring each of the pruned weights in turn. For example, first randomly select a weight A from the pruned weights, restore the weight A, and perform model training based on the currently restored weight A and the unpruned weights, obtain the current loss function corresponding to the model, and obtain the gradient (i.e., the degree of decrease) of the current loss function and the loss function corresponding to the aforementioned pruned model, and use the gradient as the gradient corresponding to weight A.

Then randomly select the next weight B for recovery, and perform model training based on the currently recovered weight B and the unpruned weights to obtain the current loss function corresponding to the model, and obtain the gradient of the current loss function and the loss function corresponding to the aforementioned pruned model, and obtain the gradient corresponding to weight B; that is, obtain the gradients corresponding to each weight in the pruned weights in turn based on the above method, sort the pruned weights based on the gradient size, and select the preset number of weights with larger corresponding gradients as important weights.

It can be understood that the larger the gradient is, the smaller the loss function value obtained by model training based on the currently restored weights and unpruned weights is, that is, the higher the degree of model convergence is, the higher the accuracy of the model is. In this way, selecting a preset number of weights corresponding to larger gradients as important weights can effectively ensure the accuracy of the model.

In some embodiments, during the above-mentioned model pruning process, a certain number of weights can be pruned additionally on the basis of the number of weight prunings that meet the preset sparsity rate of the model, and the number of important weights that are grown is ensured to be the same as the number of weights pruned additionally, so that It can ensure the accuracy of the model without changing the sparsity rate of the model.

In some embodiments, during the above-mentioned model pruning process, only a certain number of weights may be pruned so that the preset sparsity rate of the model is met after pruning, and no additional weights may be pruned. However, after growing some important weights, for example, when growing important weights for the first time to obtain a converged model, the model sparsity rate increases due to the growth of some important weights, and does not reach the preset sparsity rate. At this time, the above-mentioned pruning steps and weight growing steps may be re-executed until the model converges again and the sparsity rate of the model reaches the preset sparsity rate. In this way, the model sparsity rate can also be guaranteed.

For example, the preset model sparsity rate is 60%. Due to the growth of 20% of important weights, the model sparsity rate increases to 80%. At this time, after the model converges, a second pruning can be performed. For example, the number of weights pruned reduces the model sparsity rate to 40%. At this time, the model sparsity rate reaches 40%. Then, 20% of important weights are grown. When the model converges again, the preset model sparsity rate of 60% can be reached. Among them, the pruning ratio and important weight growth ratio of the model each time can be determined according to actual needs.

The model processing method mentioned in the embodiment of the present application is described in detail below. FIG4 is a schematic diagram of a model processing method provided in the embodiment of the present application. The model processing method shown in FIG4 can be executed by an electronic device. As shown in FIG4, the model processing method may include:

401: Obtain the network model to be pruned and the training data set.

In an embodiment of the present application, the network model to be pruned may include multiple convolutional layers, and each convolutional layer in the multiple convolutional layers may have layer identification information, such as identity identification information that characterizes the identity of the convolutional layer. The layer identification information of each convolutional layer may be unique, that is, the layer identification information of each convolutional layer is different. For example, the network model to be pruned may include n convolutional layers, and the layer identification information of the convolutional layers may be Conv1, Conv2...Conv n. Each convolutional layer may include multiple convolutional kernels, and each convolutional kernel in the multiple convolutional kernels may have kernel identification information, such as identity identification information that characterizes the identity of the convolutional kernel. The kernel identification information of each convolutional kernel may be unique, that is, the kernel identification information of each convolutional kernel is different. For example, the convolutional layer Conv1 may include m convolutional kernels, and the kernel identification information of the convolutional kernel may be W1, W2...Wm. Each convolutional kernel may include multiple weights (parameters), and the weights in each convolutional kernel may be all the same, all different, or partially the same. The multiple weights in each convolution kernel can be represented in the form of a matrix, that is, the multiple weights in each convolution kernel can be represented by a weight matrix, and each weight in the weight matrix can be used to extract and enhance the features of image data and audio data.

In some optional instances, the network model to be pruned may include but is not limited to any one of a convolutional neural network, a Transformer (a model based on a multi-head attention mechanism), a recurrent neural network (RNN), and a long short-term memory neural network (LSTM). It is understood that the network model to be pruned may be a neural network model used to implement tasks such as image recognition, target detection, reinforcement learning, and semantic analysis. In some optional implementations, the network model to be pruned may be a visual model for implementing various visual tasks.

In some optional instances, the training data set may be an image data set or an audio data set. The data in the training data set may have annotation information, and the annotation information may be information that matches the task of the network model to be pruned. For example, when the network model to be pruned is a neural network model used to implement the image recognition task, the annotation information may be the annotated category information. For another example, when the network model to be pruned is a neural network model used to implement the target detection task, the annotation information may be the annotated category information and location information.

402: Modify the first weight set of the network model to be pruned into a second weight set to obtain a first neural network model.

It can be understood that when the network model to be pruned has a large number of parameters and is difficult to deploy on a small edge device, it is necessary to minimize the number of parameters of the network model to be pruned (that is, reset the largest number of weights in the network model to be pruned to zero) to achieve maximum compression of the network model to be pruned, so that the pruned model can be deployed on a small edge device.

In some optional instances, a first weight set is determined from the network model to be pruned based on the amplitude (absolute value) of each weight in the network model to be pruned, the preset sparsity rate corresponding to the current training process, and the preset growth rate corresponding to the current training process. The weights in the first weight set may be located in the same convolution kernel, that is, belong to the same weight matrix, or may be located in different convolution kernels, that is, belong to different weight matrices. After determining the first weight set, the weights in the first weight set may be pruned, that is, the weights in the first weight set are reset to zero to obtain a second weight set.

In some optional examples, a first subset of weights may be determined from the network model to be pruned based on the amplitude (absolute value) of each weight in the network model to be pruned and a preset sparsity rate corresponding to the current training process, and a second subset of weights may be determined from the network model to be pruned based on the amplitude (absolute value) of each weight in the network model to be pruned and a preset growth rate corresponding to the current training process. Then, the union of the first subset of weights and the second subset of weights may be determined as the first set of weights, and the first set of weights may be modified to the second set of weights to obtain a first neural network model.

For example, the network model to be pruned has m weights, and the preset sparsity rate corresponding to the processing is α, and the preset growth rate corresponding to the processing is β. The m weights can be sorted from large to small based on the amplitude (absolute value) of the m weights, and the m×α weights in the sequence are determined as the weights in the first weight subset in order from large to small. Then, the non-zero weights can be sorted from large to small based on the amplitude (absolute value) of the m-m×α non-zero weights, and the m×β weights in the sequence are also determined as the weights in the second weight subset in order from large to small. Then, the union of the first weight subset and the second weight subset can be determined as the first weight set, and the weights in the first weight set can be pruned, that is, all the weights in the first weight set are set to zero, so that on the basis of pruning the first weight subset corresponding to the preset sparsity rate, the second weight subset is additionally pruned to obtain the first neural network model after the network model to be pruned is pruned.

403: Determine a target weight set that meets a preset condition from the second weight set.

It can be understood that in the process of pruning the network model to be pruned, it is necessary not only to reset the maximum number of weights to zero, but also to increase the rank of the weight matrix, reduce the omission of feature extraction, and achieve the accuracy of the network model to be pruned in achieving the preset task while compressing the network model to be pruned. Therefore, it is necessary to determine the target weight set from the second weight set, and adjust it in the subsequent weight adjustment process to ensure that important weights will not be pruned by mistake, so as to ensure the accuracy of the model in achieving the preset task.

In some optional instances, a target weight set that meets preset conditions can be determined from the second weight set based on the first loss function and the second loss function. The first loss function can be a loss function that represents the accuracy of the first network model to be pruned in achieving a preset task, and the second loss function can be a loss function that represents the different degrees of the weight matrix in the first network model to be pruned and the target low-rank approximation matrix corresponding to the weight matrix. For example, the degree of influence of the change of each weight in the second weight set on the change amplitude of the first loss function value and the second loss function value can be determined, and then the weight with a large degree of influence can be determined as the target weight to obtain the target weight set.

Among them, based on the first loss function and the second loss function, a specific method of determining a target weight set that meets the preset conditions from the second weight set is detailed in FIG5 .

404: Adjust the target weights in the target weight set to a third weight set to obtain a second neural network model.

It can be understood that adjusting the target weights in the target weight set to the third weight set can be to expand the absolute value of the target weights in the target weight set, that is, to adjust the zeroed weights to positive numbers, or to adjust the zeroed weights to negative numbers, thereby ensuring that important weights that have a significant impact on the accuracy of the pruned network model can be reactivated.

It can be understood that after obtaining the second neural network model, the training data set can be input into the second neural network model, and the second neural network model can be trained to obtain the target neural network model. For example, the training data set can be input into the second neural network model, and the training data set can be feature extracted based on the weights in the second neural network model, and prediction information can be output. Among them, the weights in the second neural network model include a third weight set. After the second neural network model outputs the prediction information of the training data set, the weights in the third weight set in the second neural network model are adjusted based on the annotation information and prediction information of the training data set until the number of iterations is greater than the preset number, and the target neural network model is obtained.

Below, the method for determining the target weight set mentioned in the embodiment of the present application is described in detail. Figure 5 is a flow chart of a method for determining the target weight set provided by the embodiment of the present application. As shown in Figure 5, the steps of determining the target weight set may include:

501: Obtain a first loss function of a first neural network model.

It can be understood that after obtaining the first neural network model, the accuracy of the first neural network model in achieving the preset task can be determined, that is, the accuracy of the prediction information output by the first neural network model for predicting the training data set can be determined, and the accuracy can be represented by the error between the labeling information of the training data set and the prediction information. Optionally, a first loss function can be used to represent the accuracy of the first neural network model in achieving the preset task. The smaller the value of the first loss function, that is, the smaller the error between the labeling information of the training data set and the prediction information, the higher the accuracy of the first neural network model in achieving the preset task.

It can be understood that when the network model to be pruned is preset as a neural network model used to implement different tasks, the first loss function used to represent the accuracy of the first neural network model in implementing the preset task may be different. Commonly used first loss functions may include 0-1 loss function, absolute value loss function, logarithmic loss function, square loss function, exponential loss function, cost loss function, cross entropy loss function, etc. For example, when the network model to be pruned is a neural network model used to implement an image recognition task, the cross entropy loss function can be obtained as the first loss function, and when the network model to be pruned is a neural network model used to implement a target detection task, the cost loss function can be obtained as the first loss function.

502: Obtain a second loss function of the first neural network model.

It can be understood that the network model to be pruned is a neural network model used to achieve the preset task, and the number of weights (parameters) There are more redundant weights, so the redundant weights in the network model to be pruned can be reduced (for example, the network model to be pruned is pruned, that is, some weights in the network model to be pruned are reset to zero) to reduce the number of parameters of the network model to be pruned and achieve compression of the network model to be pruned. When compressing the network model to be pruned, the redundant weights in the network model to be pruned are reduced, so the weight matrix in the network model to be pruned before and after compression changes, and the rank of the weight matrix after compression may be smaller than the rank of the weight matrix before compression (for example, the weights of a row of a weight matrix in the network model to be pruned are all set to zero). When the low-rank weight matrix is used to extract features from the data in the training data set, feature extraction omissions will be caused, which will lead to errors between the prediction information output by the network model to be pruned before and after compression to predict the training data set. The accuracy of the preset task achieved by the compressed network model to be pruned is lower than the accuracy of the preset task achieved by the network model to be pruned before compression.

In some optional instances, each weight matrix in the pruned network model can be subjected to low-rank decomposition, and a target low-rank approximation matrix can be obtained by multiplying multiple sub-weight matrices with small parameter amounts, simplicity and low rank. For example, for an m×n weight matrix A, it can be subjected to low-rank decomposition to obtain a target low-rank approximation matrix by multiplying sub-weight matrices U _m×k , sub-weight matrices ∑ _k×k and sub-weight matrices V ^T _k×n , where k＜＜n, the number of parameters of the weight matrix A _m×n is m×n, and the number of parameters of the target low-rank approximation matrix is k×(m+n+1). Although the number of parameters of the target low-rank approximation matrix corresponding to the weight matrix is much smaller than the corresponding weight matrix, due to the low rank of the sub-weight moments in the target low-rank approximation matrix, when the target low-rank approximation matrix corresponding to the weight matrix is used to extract features from the data in the training data set, feature extraction omissions will occur. Therefore, when compressing the network model to be pruned, the difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix can be increased, the rank of the compressed weight matrix can be increased, and the feature extraction omission can be reduced, thereby reducing the error between the prediction information output by predicting the training data set using the network model to be pruned before and after compression, and ensuring the accuracy of the compressed network model to be pruned in achieving the preset task. Optionally, a second loss function can be used to represent the difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix. The smaller the value of the second loss function, that is, the greater the difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix, the higher the rank of the weight matrix, the fewer feature extraction omissions, and the smaller the error between the prediction information output by predicting the training data set using the network model to be pruned before and after compression, the higher the accuracy of the compressed network model to be pruned in achieving the preset task.

503: Determine a target weight set from the second weight set based on the first loss function and the second loss function.

It can be understood that in the process of pruning the network model to be pruned, it is necessary not only to reduce the amount of parameters but also to reduce the omission of feature extraction. Therefore, in the process of pruning the network model to be pruned, it is necessary not only to reset the maximum number of weights to zero but also to increase the rank of the weight matrix to achieve the maximum compression of the network model to be pruned while ensuring the accuracy of the preset task of the network model to be pruned.

In some optional instances, by pruning the network model to be pruned to obtain the first neural network model, the error between the annotation information and the prediction information of the training data set can be adjusted, that is, the value of the first loss function is changed, and the degree of difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix can be adjusted to improve the rank of the weight matrix. Therefore, by pruning the network model to be pruned to obtain the first neural network model, the first loss function value and the second loss function value can be adjusted at the same time, so as to reduce the number of parameters of the network model to be pruned while improving the rank of the weight matrix in the network model to be pruned, reduce the omission of feature extraction, and ensure the accuracy of the network model to be pruned in achieving the preset task.

It can be understood that after the first weight set in the pruned network model is set to zero, if there are important weights in the first weight set that have a greater impact on the change amplitude of the first loss function value and the change amplitude of the second loss function value, that is, important weights that have a significant impact on the accuracy of the prediction information output by the first neural network model, if the important weights are kept at zero, in the subsequent process of adjusting the weights (weights that are not set to zero) of the first neural network model, the change amplitude of the first loss function value and the change amplitude of the second loss function value are small, and it is difficult to make the first neural network model converge by adjusting the weights that are not set to zero. Therefore, after pruning the first weight set in the network model to be pruned to obtain the second weight set, the target weight set (i.e., important weights) can be determined from the second weight set for growth processing to ensure that the important weights that have a greater impact on the variation range of the first loss function value and the variation range of the second loss value will not be pruned incorrectly, so that in the subsequent process of adjusting the weights of the first neural network model, the important weights can be adjusted to change the variation range of the first loss function value and the second loss function value, so that the first neural network model converges, so as to reduce the number of parameters of the network model to be pruned while improving the rank of the weight matrix in the network model to be pruned, reducing the omission of feature extraction, and ensuring the accuracy of the network model to be pruned in achieving the preset tasks.

In some optional examples, the target loss function can be determined according to the sum of the first loss function and the second loss function, and the important weights (i.e., the target weight set) that have a greater impact on the change range of the target loss function value can be determined, that is, the important weights that have a greater impact on the change range of the first loss function value and the change range of the second loss function value can be determined. When the weights in the pruned network model are subsequently adjusted, the important weights that have a greater impact on the change range of the first loss function value and the second loss function value can also be used. The target weights in the target weight set are adjusted so that the rank of the weight matrix in the network model to be pruned is improved while reducing the number of parameters of the network model to be pruned, reducing the omission of feature extraction, and ensuring the accuracy of the network model to be pruned in achieving the preset task.

In some optional instances, the objective loss function may take the form shown in Formula 1:
L＝L _task +λL _rank Formula 1

In Formula 1, L may represent the target loss function, L _task may represent the first loss function, L _rank may represent the second loss function, and λ may represent the introduced linear combination hyperparameter.

It can be understood that the target loss function can be a function with multiple weights in the network model to be pruned as independent variables. By chain-deriving each weight using the target loss function, the derivative corresponding to each weight can be obtained. The derivative can reflect the degree of influence of the change in weight on the change in the value of the target loss function. The larger the derivative of the weight, the greater the influence of the change in weight on the change in the value of the target loss function.

In some optional instances, after determining the derivative corresponding to each weight in the second weight set, the amplitudes (absolute values) of the derivatives corresponding to each weight in the second weight set can be sorted from large to small, and the weights corresponding to the m×β derivatives in the sequence can be determined as the target weight set in order from large to small.

Below, the method for determining the second loss function mentioned in the embodiment of the present application is described in detail. FIG6 is a schematic diagram of a process for obtaining the second loss function of the first neural network model provided by the embodiment of the present application. As shown in FIG6, the steps of obtaining the second loss function of the first neural network model may include:

601: Standardize each weight matrix in the network model to be pruned to obtain a weight matrix to be decomposed corresponding to each weight matrix.

It can be understood that for each weight matrix in the network model to be pruned, the Frobenius norm of each weight matrix in the network model to be pruned can be determined based on the Frobenius norm calculation formula, and then each weight matrix in the network model to be pruned can be divided by the Frobenius norm of each weight matrix to achieve L2 normalization of each weight matrix in the network model to be pruned, and obtain the weight matrix to be decomposed corresponding to each weight matrix in the network model to be pruned.

In some optional examples, the Frobenius Norm calculation formula may be in the form shown in Formula 2:

In Formula 2, ||W|| _F may represent the Frobeus norm of each weight matrix in the network model to be pruned, W may represent each weight matrix in the network model to be pruned, ^WT may represent the transpose of each weight matrix in the network model to be pruned, and tr(·) may represent the trace of ·.

In some optional examples, the weight matrix to be decomposed corresponding to each weight matrix in the network model to be pruned may be in the form shown in Formula 3:

Among them, in formula 3, ||W|| _F can represent the Frobeus norm of each weight matrix in the network model to be pruned, and W can represent each weight matrix in the network model to be pruned. It can represent the weight matrix to be decomposed.

602: Perform singular value decomposition processing on the weight matrices to be decomposed corresponding to each weight matrix in the network model to be pruned, and determine the target low-rank approximation matrix corresponding to each weight matrix in the network model to be pruned.

It can be understood that in order to improve the rank of the compressed weight matrix, the weight matrix to be decomposed corresponding to each weight matrix can be subjected to singular value decomposition to obtain the eigenvector set and singular value set corresponding to each weight matrix in the network model to be pruned. Then, the left singular matrix, singular value matrix and right singular matrix can be determined based on the eigenvector set and singular value set corresponding to each weight matrix in the network model to be pruned, and the left singular matrix, singular value matrix and right singular matrix are multiplied as the target low-rank approximation matrix corresponding to the weight matrix.

In some optional instances, for the weight matrix to be decomposed corresponding to each weight matrix in the network model to be pruned, the transpose of the weight matrix to be decomposed can be determined, and multiple eigenvalues and eigenvectors can be determined based on the weight matrix to be decomposed and the transpose of the weight matrix to be decomposed. Then, the left singular matrix and the right singular matrix corresponding to the weight matrix can be determined based on the eigenvectors, and the singular value matrix can be determined based on the eigenvalues. For example, multiple eigenvalues and eigenvectors can be determined based on the product of the weight matrix to be decomposed and the transpose of the weight matrix to be decomposed, and the left singular matrix can be determined based on the determined multiple eigenvectors. Multiple eigenvalues can be determined based on the transpose of the weight matrix to be decomposed and the product of the weight matrix to be decomposed. The right singular matrix is determined based on the determined multiple eigenvectors. The square root of all eigenvalues can be determined as the singular value corresponding to each weight matrix in the network model to be pruned, and the singular value set corresponding to each weight matrix in the network model to be pruned is obtained. Then, the singular value matrix can be determined based on the singular value set.

In some optional instances, after obtaining the singular value set corresponding to each weight matrix in the network model to be pruned, the candidate singular value set can be determined from the singular value set corresponding to each weight matrix of the network model to be pruned. For example, the largest k singular values can be selected from the singular value set corresponding to each weight matrix of the network model to be pruned according to the matrix low rank approximation principle (Eckart-Young) as the candidate singular value set corresponding to each weight matrix in the network model to be pruned. Then, the singular value matrix corresponding to each weight matrix in the network model to be pruned can be generated based on the candidate singular value set corresponding to each weight matrix in the network model to be pruned.

603: Determine a second loss function according to each weight matrix in the network model to be pruned and a target low-rank approximation matrix corresponding to each weight matrix.

Next, the second loss function is introduced by taking a weight matrix in the network model to be pruned and the target low-rank approximation matrix corresponding to the weight matrix as an example. Specifically, the calculation formula of the second loss function can be in the form shown in Formula 4:

U ^T U＝I
V ^T V＝I

In Formula 4, can represent the weight matrix to be decomposed obtained after the weight matrix is standardized, U can represent the left singular matrix, V can represent the transpose of the right singular matrix, ∑ can represent the singular value matrix, U and V can represent the unitary matrix concatenated by orthogonal basis vectors, Trun can represent the target low-rank approximation matrix, which can be specifically the product of the transpose of the left singular matrix, the singular value matrix, and the right singular matrix, ||·|| _F can represent the Frobeus norm of ·, and I can represent the identity matrix.

It can be understood that in practical applications, the model processing flow method can be implemented and executed in the form of software code.

In some optional implementations, the pseudo code (referred to as pseudo code M1) of the operations related to the model processing flow method is as follows:

Input: an n-layer dense network W; target sparsity rate _fs (iter) at iteration iter; growth weight ratio _fdeacy (iter) at iteration iter; update frequency Δ of pruning weights; total pruning time T = _Tprune + _Tfinetune ; learning rate α

Out：A Sparse Model W⊙M

:

1Initialize a matrix of all 1s;

2 M←1;

3 //Phase 1: weight update (pruning and growing);

4 for iter←1to T _prune do

5 Input the training data into the network for forward propagation and calculate the loss L;

8 if iter%Δ＝0then

7 //Update weight position mask M;

8

9 Prune to the target sparsity rate _fs (iter)% of iter iterations according to the weight amplitude |W|;

10 Again cut off f _deacy (iter)% of the remaining weights according to the weight amplitude |W|;

11 Gradient size generated by combined loss Grow the same amount of weights f _deacy (iter)%

12 end if

13 else

14 Training sparse networks using gradient descent

15 end if

18 end for

17 //Stage 2: Fine-tune the network without changing the position of weights;

18 for iter←T _prune +1to T do

19. Without changing the weight positions, train the sparse network until convergence;

20 end for

21 return sparse network W⊙M Next, we explain each line of code in pseudocode M1:

First, the fifth line of code in M1 is used to determine the task loss. The training data set can be input into the dense network W. In the dense network W, each processed weight matrix of each convolutional layer forward propagates the training data. The dense network W can output the prediction information of the data in the training data set, and then the task loss can be determined based on the labeling information and prediction information of the data in the training data set.

The 8th line in M1 is used to determine the adversarial loss function. The low-rank approximation corresponding to each weight matrix to be processed in the dense network W is determined, and the training data is convolved using each target low-rank approximation matrix. The dense network W can output the prediction information of the data in the training data set, and then the adversarial loss function can be determined based on the labeling information and prediction information of the data in the training data set.

Line 9 in M1 is used to prune a certain number of weights based on the preset weights that need to be pruned to improve the sparsity of the model and reduce the memory of the model.

Line 10 in M1 is used to prune a certain number of remaining unpruned weights based on the preset weights that need to be pruned.

Line 11 in M1 is used to sort the pruned weights according to the absolute value of the gradient and grow the weights with larger gradients, so as to ensure that important weights that were mistakenly pruned can be reactivated.

It can be understood that the above pseudocode, after pruning the weights, can determine the derivatives of all pruned weights based on the target loss function, and grow the weights with larger derivatives to ensure that important weights are not pruned incorrectly, thereby ensuring the accuracy of the model.

In order to verify the pruning effect of the model processing method provided in the embodiment of the present application, in some examples, the pruning effect is verified based on the small data set CIFAR-10. Table 1 shows the classification accuracy of the pruned model obtained based on different pruning algorithms at different sparsity rates. Based on Table 1, it can be seen that the pruning effect of the model processing method provided in the embodiment of the present application at a high sparsity rate is significantly better than other unstructured pruning methods.

Table 1

In other examples, the pruning effect is verified based on the large dataset ImageNet. Table 2 shows the classification accuracy of the pruned model based on different pruning algorithms at different sparsity rates. Based on Table 2, it can be seen that the pruning effect of the model processing method provided in the embodiment of the present application at a high sparsity rate is significantly better than other unstructured pruning methods.

Table 2

Figure 7 is a curve diagram of the computing power consumption and accuracy of the comprehensive model of the model processing method provided in the embodiment of the present application. Based on Figure 7, it can be seen that the model processing method provided in the embodiment of the present application is significantly better than other unstructured pruning methods in terms of the computing power consumption and accuracy of the comprehensive model.

In other examples, downstream visual tasks such as target detection and instance segmentation are verified on the COCO val2017 dataset by using the Mask R-CNN framework with ResNet-50 FPN as the backbone network. Table 3 shows the positioning accuracy and segmentation accuracy of the pruned target detection model obtained based on different pruning algorithms at different sparsity rates. Based on Table 3, it can be seen that the pruning effect of the model processing method provided in the embodiment of the present application at high sparsity rates is significantly better than other unstructured pruning methods.

Table 3

In other examples, unstructured pruning is performed on the DeiT-S model of the transformer structure, and then verified on the ImageNet large-scale image classification dataset. Table 4 shows the classification accuracy of the transformer model obtained based on different pruning algorithms at different sparsity rates. Based on Table 4, it can be seen that the model processing method provided in the embodiment of the present application is also more effective on the transformer architecture.

Table 4

The model processing method provided in the embodiment of the present application can efficiently compress the visual model, reduce the computing power of the visual model, and help deploy large models on small edge devices. For example, it can be deployed to a cloud service or to a terminal device. The following example illustrates the effect of deploying a pruned model on a small edge device.

Challenging visual tasks such as object detection often require the support of large visual models. In some cases, smartphones are small devices due to power and energy consumption constraints. By deploying the object detection model that completes unstructured pruning on the mobile phone side, it is possible to significantly reduce the energy consumption of the mobile phone without compromising the model performance, thereby extending the operating life of the mobile phone.

In other instances, in video surveillance, cameras are usually small end-side devices. For some remote areas where it is difficult to deploy power supply facilities for surveillance systems, they are often powered by solar energy. Therefore, the energy consumption of the target detection model deployed on the camera end is highly required. By deploying the target detection model that has completed unstructured pruning on the camera end, the energy consumption of video surveillance can be significantly reduced without compromising the performance of the model.

The electronic devices mentioned in this application are introduced below. It is understood that the electronic devices may include but are not limited to: mobile phones (including foldable screen mobile phones and straight-screen mobile phones), tablet computers, desktop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (personal digital assistants), tablet computers (portable android devices, PADs), personal digital assistants (personal digital assistants, PDAs), handheld devices with wireless communication functions, computing devices, vehicle-mounted devices or wearable devices, virtual reality (virtual reality) Electronic devices with data transmission synchronization requirements include mobile terminals or fixed terminals, such as virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical, wireless terminals in smart grid, wireless terminals in transportation safety, wireless terminals in smart city, wireless terminals in smart home, power banks, etc.

Fig. 8 shows a schematic diagram of the hardware structure of the electronic device. It is understandable that the electronic device of the present application can be a server, a desktop computer, a handheld computer, a notebook computer (laptop) and other electronic devices. The structure of the electronic device is introduced below by taking the electronic device as an example of a server.

FIG8 is a block diagram of a server provided in an embodiment of the present application, and FIG8 schematically shows an example server of multiple embodiments. In one embodiment, the server may include one or more processors 804, a system control logic 806 connected to at least one of the processors 804, a system memory 812 connected to the system control logic 808, a non-volatile memory (NVM) 816 connected to the system control logic 808, and a network interface 820 connected to the system control logic 808.

In some embodiments, the processor 804 may include one or more single-core or multi-core processors. In some embodiments, the processor 804 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In an embodiment where the server adopts an eNB (Evolved Node B, enhanced base station) 101 or a RAN (Radio Access Network, wireless access network) controller 102, the processor 804 may be configured to execute various compliant embodiments.

In some embodiments, system control logic 808 may include any suitable interface controller to provide any suitable interface to at least one of processors 804 and/or any suitable device or component in communication with system control logic 808 .

In some embodiments, the system control logic 808 may include one or more memory controllers to provide an interface to a system memory 812. The system memory 812 may be used to load and store data and/or instructions. In some embodiments, the server's memory 812 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

The non-volatile memory (NVM) 816 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. In some embodiments, the non-volatile memory (NVM) 816 may include any suitable non-volatile memory such as flash memory and/or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.

The non-volatile memory (NVM) 816 may include a portion of storage resources on the device where the server is installed, or it may be accessible by the device but not necessarily a portion of the device. For example, the non-volatile memory (NVM) 816 may be accessed over a network via the network interface 820 .

In particular, the system memory 812 and the non-volatile memory (NVM) 816 may include: a temporary copy and a permanent copy of the instruction 824, respectively. The instruction 824 may include: an instruction that causes the server to implement the model quantization method mentioned in the embodiment of the present application when executed by at least one of the processors 804. In some embodiments, the instruction 824, hardware, firmware and/or its software components may be additionally/alternatively placed in the system control logic 808, the network interface 820 and/or the processor 804.

The network interface 820 may include a transceiver for providing a radio interface for the server, and then communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 820 may be integrated with other components of the server. For example, the network interface 820 may be integrated with at least one of the processor 804, the system memory 812, the non-volatile memory (NVM) 816, and a firmware device (not shown) having instructions. When at least one of the processors 804 executes the instructions, the server implements the model quantization method mentioned in the embodiments of the present application.

The network interface 820 may further include any suitable hardware and/or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 820 may be a network adapter, a wireless network adapter, a telephone modem and/or a wireless modem.

In one embodiment, at least one of the processors 804 may be packaged together with logic for one or more controllers of the system control logic 808 to form a system in package (SiP). In one embodiment, at least one of the processors 804 may be integrated on the same die with logic for one or more controllers of the system control logic 808 to form a system on chip (SoC).

The server may further include: an input/output (I/O) device 832. The I/O device 832 may include a user interface that enables a user to interact with the server; the design of the peripheral component interface enables the peripheral component to also interact with the server. In some embodiments, the server also includes a sensor for determining at least one of an environmental condition and location information related to the server.

In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and/or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 820 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

It should be noted that the units/modules mentioned in the various device embodiments of the present application are all logical units/modules. Physically, a logical unit/module can be a physical unit/module, or a part of a physical unit/module, or can be implemented as a combination of multiple physical units/modules. The physical implementation method of these logical units/modules themselves is not the most important. The combination of functions implemented by these logical units/modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units/modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units/modules in the above-mentioned device embodiments.

It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims (12)

A model processing method, characterized in that it is used for electronic equipment, the method comprising: Obtaining a first neural network model, wherein the first neural network model is obtained by pruning the neural network model to be pruned, and the pruning includes modifying a first weight set of the neural network model to be pruned into a second weight set of the first neural network model, wherein a non-zero first weight in the first weight set has a corresponding value of zero in the second weight set; Selecting a target weight set that meets a weight condition from the second weight set; Performing parameter adjustment processing on the target weight set in the second weight set to obtain a third weight set, wherein the value of the first weight in the third weight set is non-zero; The third set of weights is used for the first neural network model. The method according to claim 1, characterized in that the step of selecting a target weight set satisfying a weight condition from the second weight set comprises: Performing recovery processing on any weight in the second weight set to obtain a candidate neural network model; Obtaining the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model; The target weight set is determined based on the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model after each weight in the second weight set is restored. The method according to claim 2, characterized in that obtaining the loss function corresponding to the first neural network model comprises: Determining a first loss function based on real information corresponding to the training data set and predicted information output by the first neural network model; Determine a second loss function based on the first neural network model and the candidate neural network model; A loss function corresponding to the first neural network model is determined based on the first loss function and the second loss function. The method according to claim 3, characterized in that obtaining a first loss function corresponding to the first neural network model comprises: Inputting the training data set into the first neural network model to obtain prediction information output by the first neural network model; The first loss function is obtained based on the real information corresponding to the training data set and the predicted information. The method according to claim 3, characterized in that obtaining the second loss function corresponding to the first neural network model comprises: Obtaining a weight matrix set in the network model to be pruned; Determine a target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set; Based on the weight matrices and the target low-rank approximation matrices corresponding to the weight matrices, the second loss function corresponding to the first neural network model is obtained. The method according to claim 5, characterized in that the step of determining the target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set comprises: Performing standardization processing on the weight matrices to obtain weight matrices to be decomposed corresponding to the weight matrices; Perform low-rank decomposition processing on the weight matrices to be decomposed corresponding to the weight matrices to obtain the target low-rank approximation matrices corresponding to the weight matrices. The method according to any one of claims 1 to 6, characterized in that the pruning of the neural network model to be pruned The first neural network model is obtained, including: The network model to be pruned is pruned based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate to obtain the first neural network model. The method according to claim 7, characterized in that the pruning of the network model to be pruned based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate to obtain the first neural network model comprises: Based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate, determining the weights of the number corresponding to the preset sparsity rate in the network model to be pruned as pruning weights; The value corresponding to the pruning weight is set to zero to obtain the first neural network model. The method according to any one of claims 1 to 6, characterized in that pruning the neural network model to be pruned to obtain a first neural network model comprises: Based on the amplitude of each weight in the network model to be pruned, a preset sparsity rate and a preset growth rate, the network model to be pruned is pruned to obtain the first neural network model. The method according to claim 9, characterized in that pruning the neural network model to be pruned to obtain the first neural network model comprises: Based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate, determining the weights of the number corresponding to the preset sparsity rate in the network model to be pruned as a first weight subset; Based on the amplitude and preset growth rate of each weight in the network model to be pruned except the first weight subset, the weights corresponding to the preset growth rate in the network model to be pruned except the first weight subset are determined as a second weight subset; The values corresponding to the weights in the first weight subset and the second weight subset are set to zero to obtain the first neural network model. An electronic device, characterized in that it comprises: a memory for storing instructions executed by one or more processors of the electronic device, and the processor, which is one of the one or more processors of the electronic device, is used to execute the model processing method described in any one of claims 1-10. A readable storage medium, characterized in that instructions are stored on the readable storage medium, and when the instructions are executed on an electronic device, the electronic device executes the model processing method according to any one of claims 1 to 10.