CN117787357A - Computer vision-oriented post-training model sparse method and device - Google Patents

Abstract

The invention discloses a post-training model sparse method and device for computer vision. The method includes the following steps: introducing control loss to achieve the target sparsity rate; using weighted reconstruction loss as supervision to reduce the difference between the sparse output and the original output; adding the control loss and reconstruction loss to obtain the overall optimized target loss; using The target loss is used to determine the reconstructed weight and sparsity rate that are most suitable for the target sparsity rate of each layer; the function that determines the reconstructed weight and sparsity rate is converted into a differentiable function to achieve differentiable optimization of sparse rate distribution. This method establishes a bridge between sparse threshold and sparse rate from the perspective of probability density, and further uses kernel density estimation technology to construct a differentiable representation between sparse threshold and sparse rate, which can preserve the original weight to the greatest extent without damaging the original weight. Learn the optimal sparsity rate allocation with global constraints in the case of visual features, and quickly obtain the required sparse neural network.

Description

A post-training model sparse method and device for computer vision

Technical field

The invention relates to a post-training model sparse method for computer vision, and also relates to a corresponding post-training model sparse device, belonging to the technical field of computer vision.

Background technique

Deep neural networks (DNNs) have achieved remarkable success in multiple fields such as computer vision, natural language processing, and information retrieval. However, when deploying deep neural networks on edge devices with limited resources, reducing the memory footprint of neural networks and improving energy efficiency become key issues, and therefore, various compression techniques have been proposed to improve the efficiency of the model. Among these compression techniques, sparse training (Sparse Training) is a method to reduce model parameters by setting sparsity constraints during the training process. Post-training sparsity (PTS) can also eliminate high training costs, but is usually caused by neglecting each Reasonable sparsity rate allocation of layers will lead to a significant decrease in model accuracy. The sparsity rate solution method in the existing technology is mainly aimed at training perception scenarios, and often cannot converge stably in post-training sparse scenarios where the amount of data is limited and the training cost is low. Most methods to restore the accuracy of sparse models require retraining the model under large amounts of data for a long time. When the training data set is huge, it usually takes hours or even days, which brings obstacles to large-scale applications in various industries.

In the existing technology, POT (Post-training via layer-wise calibration) proposes to sparse the neural network through post-training without training labels, and reconstructs sparse weights layer by layer to improve accuracy, which can be achieved Reduced cost of producing sparse models. However, in order to achieve comparable performance to the original neural network model, its maximum overall sparsity rate can only reach 50%, and when the degree of sparsity increases, the accuracy of the model will quickly collapse. In addition, some retraining-based methods, such as experience-based ERK ( -Rènyi-Kernel) or LAMP (Layer-Adaptive Sparsity for the Magnitude-based Pruning), based on learning STR (Soft ThresholdWeight Reparameterization for Learnable Sparsity) and LTP (Learnable Threshold Pruning). Although these methods generate a sparsity rate for each layer, However, problems will be encountered under the sparse setting of post-training. The experience-based ERK and LAMP methods rely heavily on manually designed prior knowledge and cannot guarantee the optimal solution. Learning-based STR and LTP methods either destroy the original weight distribution or require end-to-end training to converge. Therefore, none of them can achieve efficient and accurate distribution of sparsity after training, and even need to carefully adjust hyperparameters and repeat experiments multiple times to achieve the target sparsity rate, further increasing the cost of generating sparse models.

In the Chinese invention application with application number 202211358758.3, a method for detecting pedestrians based on channel pruning YOLOv5s is disclosed, including the following steps: unlocking the network fusion layer, sparse training model, model pruning, fine-tuning training, and target model, thereby reducing the parameters of the neural network while accelerating the reasoning of the neural network, so that the YOLOv5 algorithm can achieve accelerated reasoning effects on low computing power platforms.

Contents of the invention

The primary technical problem to be solved by the present invention is to provide a post-training model sparse method for computer vision.

Another technical problem to be solved by the present invention is to provide a post-training model sparse device for computer vision.

In order to achieve the above objects, the present invention adopts the following technical solutions:

According to the first aspect of the embodiment of the present invention, a post-training model sparse method for computer vision is provided, including the following steps:

(1) Introduce control loss to achieve the target sparsity rate. The control loss is:

Among them, r _i and N _i represent the sparsity rate and number of elements of the i-th layer weight respectively; r ₀ is the global sparsity rate target; L _c is the control loss;

(2) Use the reconstruction loss of weights as supervision to reduce the difference between the sparse output and the original output. The reconstruction loss is:

L _rec =D _KL (Y _dense ||Y _sparse )

Among them, L _rec is the reconstruction loss; D _KL (·) represents the Kullback-Leibler divergence function; Y _dense is the output of the original neural network model; Y _sparse is the output of the sparse neural network model;

(3) Add the control loss and the reconstruction loss to obtain the overall optimized target loss:

L＝L _rec +L _c

Among them, L is the target loss;

(4) Use the target loss to determine the reconstructed weights and sparsity rates of each layer that are most suitable for the target sparsity rate:

Among them, arg min is the arg min function; W is the reconstructed weight; r _i is the sparse rate of the i-th layer weight;

(5) Convert the function that determines the reconstructed weight and sparsity rate of each layer into a differentiable function to achieve differentiable optimization of sparse rate allocation.

Preferably, in step (2):

The sparse model output Y _sparse is calculated by the following formula:

Y _sparse =f(X,M⊙W)

Among them, f represents the neural network model, X is the input of the neural network model, M is the mask, and W is the weight of the neural network model.

Preferably, the generation of the mask M is calculated based on amplitude:

M＝0.5*sgn(|W|-t)+0.5

Among them, sgn represents the sign function. If it is a positive value, +1 is returned, otherwise -1 is returned; t is the sparse threshold; W is the weight of the neural network model.

Preferably, in step (5):

The derivative of the target loss L with respect to the sparsity rate r _l of layer l is:

Among them, L is the target loss, L _rec is the reconstruction loss, Lc is the control loss, and r _l is the sparse rate of layer l.

Preferably, the derivative of the control loss Lc with respect to the sparsity rate r _l of layer l is:

Among them, r _i and N _i respectively represent the sparse rate and the number of elements of the i-th layer weight, and r _l and N _l respectively represent the sparse rate and number of elements of the l-th layer weight.

Preferably, the derivative of the reconstruction loss L _rec with respect to the sparsity rate r _l of layer l is:

Among them, M _l is the mask of layer l, and t _l is the sparse threshold of layer l.

Preferably, in step (5):

The sparsity rate r _l of layer l is calculated to satisfy the formula:

Among them, p(w) is the probability density function of weight W _l , W _l is the weight distribution of layer l, w is the sampling weight from this distribution; g ^-1 is the inverse function of differentiable function g, t _l =g ( r _l ); t _l is the sparse threshold of layer l.

Preferably, the derivative of the sparsity rate r _l of the layer l with respect to the sparsity threshold t _l is:

Among them, p(t _l ) is the probability density function of t _l , p(-t _l ) is the probability density function of -t _l ; r _l is the sparse rate of layer l, and t _l is the sparse threshold of layer l.

Among them, the better method is to use the kernel density estimation method to estimate p(w) as follows:

Among them, p(w) is the probability density function of weight W _l ; K is a non-negative kernel function; n is the number of sampling points; h is a smoothing parameter called bandwidth, h>0; the kernel function with subscript h is called The scaling kernel function is defined as w is the sampling weight in the weight distribution W _l of the l-th layer; w _i is the weight of the i-th layer.

According to a second aspect of an embodiment of the present invention, there is provided a post-training model sparse device for computer vision, comprising a processor and a memory, wherein the processor and the memory are coupled; wherein:

The memory is used to store computer programs;

The processor is configured to run a computer program stored in the memory and execute the above-mentioned post-training model sparse method for computer vision.

Compared with the existing technology, the post-training model sparse method provided by the embodiment of the present invention reconstructs the target by defining a controllable sparsity rate, and establishes a bridge between the sparsity threshold and the sparsity rate from the perspective of probability density, and further uses kernel Density estimation technology is used to construct a differentiable representation between sparsity threshold and sparsity rate. It can learn the optimal sparsity rate allocation with global constraints without damaging the original weight and retaining visual features to the greatest extent, and quickly obtain the required Sparse neural networks. Therefore, the post-training model sparse method provided by the embodiment of the present invention has beneficial effects such as controllable sparsity rate distribution, high accuracy, and high efficiency.

Description of the drawings

Figure 1 is a flow chart of a post-training model sparse method for computer vision in an embodiment of the present invention;

Figure 2 is a logical framework diagram of a post-training model sparse method for computer vision in an embodiment of the present invention;

Figure 3 is a schematic diagram of 6 comparative test results of the accuracy of the sparsity rate that can be learned and the sparsity rate that cannot be learned in the embodiment of the present invention;

Figure 4 is a schematic structural diagram of a post-training model sparse device in an embodiment of the present invention.

Detailed ways

The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

As shown in Figure 1, the embodiment of the present invention provides a fast and controllable post-training sparse (abbreviated as FCPTS, the same below) method, which at least includes the following steps:

S1: Introduce control loss to achieve the target sparsity rate; the control loss Lc is:

Among them, r _i and N _i represent the sparsity rate and the number of elements of the i-th layer weight respectively; r ₀ is the global sparsity rate target; L _c is the loss to control the global sparsity rate.

The control loss L _c is used to achieve the target sparsity rate r ₀ without complex hyperparameter tuning.

S2: The reconstruction loss of the weight is used as supervision to reduce the difference between the sparse output and the original output to retain the image features learned in the original neural network model; the reconstruction loss L _rec is:

L _rec =D _KL (Y _dense ||Y _sparse ) (2)

Where D _KL (·) represents the Kullback-Leibler divergence function; Y _dense is the output of the original neural network model; and Y _sparse is the output of the sparse neural network model.

Under the supervision of the reconstruction loss L _rec , the weight optimization will be guided in a direction that helps the sparse output closely resemble the original output, leading to higher accuracy. Furthermore, the entire reconstruction is based on well-trained original weights, so the process is fast and easy to converge.

The sparse model output Y _sparse can be calculated as follows:

Y _sparse =f(X,M⊙W) (3)

Among them, f represents the neural network model, X is the input of the neural network model, which is generally a three-dimensional image in computer vision tasks, M is the mask, and W is the weight of the neural network model.

The generation of the mask M can be calculated in an amplitude-based manner:

M＝0.5*sgn(|W|-t)+0.5 (4)

Among them, sgn represents the sign function, if it is a positive value, it returns +1, otherwise it returns -1; t is the sparse threshold.

S3: Add the control loss L _c and the reconstruction loss L _rec to get the overall optimized target loss L:

L＝L _rec +L _c (5)

S4: Use the target loss L to determine the reconstructed weight W and sparsity rate r _i of each layer that are most suitable for the target sparsity rate r ₀ :

Among them, arg min is the arg min function.

S5: Convert the function that determines the reconstructed weight W and sparsity rate r _i of each layer into a differentiable function to achieve differentiable optimization of sparse rate allocation.

After the optimization objective is defined in step S4, since the argmin function itself that determines the reconstructed weight W and sparsity rate r _i of each layer is not differentiable, it needs to be converted into a differentiable function. In one embodiment of the present invention, a clever bridge function is proposed to realize the use of sparsity threshold t to calculate the sparsity rate r. The derivative of the bridge function can be obtained through kernel density estimation, which can transform the original more complex optimization problem into an optimization problem targeting the sparse threshold t. Since the transformed optimization function is differentiable, differentiable optimization of sparsity rate allocation is achieved, that is, the sparsity rate r of each layer can be obtained by learning the optimal sparsity threshold t of each layer. The conversion process is described in detail below.

Given a specific layer l, the derivative of the target loss L with respect to the sparsity rate r _l of the l-th layer is:

Among them, r _l is the sparsity rate of layer l, L is the target loss, L _rec is the reconstruction loss, and Lc is the control loss.

According to formula (1), it can be concluded that the latter term in formula (7) is:

Transform the first term of formula (7) as follows:

Among them, M _l is the mask of layer l, and t _l is the sparse threshold of layer l.

According to formula (2), formula (3) and formula (4), the first and second terms in formula (9) (i.e. and/> ) is easy to calculate and get the result. For the third term in formula (9) (i.e./> ), we need to find a differentiable function g that satisfies t _l =g(r _l ), then formula (9) can be calculated. Although the differentiable function g is difficult to construct, its inverse function g ^-1 can be formalized in a differentiable estimation manner. Then the learning of the sparsity rate r is transferred to the sparsity threshold t. The specific process is as follows:

Given the weight distribution W _l of layer l and its probability density function p(w), and sampling the weight w from this distribution, the sparsity rate r _l can be calculated by the following formula:

Among them, p(w) is the probability density function of weight W _l ; g ^-1 is the inverse function of the differentiable function g; t _l is the sparse threshold of layer l.

Therefore, the derivative of the sparsity rate r _l with respect to the sparsity threshold t _l can be written as:

Where p(t _l ) is the probability density function of t _l , p(-t _l ) is the probability density function of -t _l ; r _l is the sparsity rate of layer l, and t _l is the sparsity threshold of layer l.

It can be seen from formula (11) that the key step in calculating the derivative is to model the probability density function p(w). To this end, the embodiment of the present invention uses the kernel density estimation (KDE) method to estimate p(w) as follows:

Where K is a non-negative kernel function; n is the number of sampling points; h is a smoothing parameter called bandwidth, h＞0; the kernel function with subscript h is called the scaling kernel function, defined as w is the sampling weight in the weight distribution W _l of the lth layer; _wi is the weight of the ith layer.

Based on the trade-off between the bias of the estimator and its variance, it is usually set as follows: n = 100, h = 0.5, K(x) = Φ(x); where Φ is the standard normal density function. Using the kernel density estimation KDE technique, it is proved that the bridge function is differentiable.

With the help of the above bridge function and differentiable estimation, the sparse rate r can be learned and optimized through the intermediate variable, namely the sparse threshold t:

According to formula (8), formula (11) and formula (12), the derivative of the control loss L _c in formula (13) with respect to the sparse threshold t _l can be calculated.

For the reconstruction loss L _rec , the derivative of L _rec with respect to the sparsity threshold t _l can also be calculated directly:

Finally, through the learned sparse threshold t and bridge function (10), the accurate sparsity rate r can be calculated, which is helpful for flexible and controllable learning of sparsity distribution.

The post-training model sparse method for computer vision provided by the embodiment of the present invention has been introduced in detail above. The logical framework of the post-training sparse method is shown in Figure 2. First, a controllable sparsity rate reconstruction target is defined, and then a differentiable sparsity rate allocation method is designed that does not interfere with the training weights. Based on the sparsity rate reconstruction target and differentiable The sparsity rate allocation method can easily learn the optimal sparsity rate allocation. Moreover, thanks to the controllable and differentiable sparse allocation, this post-training sparse method performs well in post-training scenarios, requiring only one network reconstruction to reach the target sparsity rate, with high efficiency. Usually, only one NVIDIA RTX 3090 GPU can generate sparse neural networks in tens of minutes, achieving the reconstruction of sparse neural networks with high efficiency and high accuracy.

In actual operation, the post-training model sparse method for computer vision provided by the embodiment of the present invention can sparse a given neural network according to the algorithm steps shown in Table 1. Among them, the data sets in the embodiments of the present invention are all image data sets in computer vision tasks, such as ImageNet, etc.

Table 1

In order to verify the actual performance and effect of the post-training model sparse method for computer vision provided by the embodiment of the present invention, the inventor conducted the following series of experimental tests and evaluations in computer vision tasks such as image classification and target detection, and Relevant comparisons are made with existing technologies.

First. Experimental testing on image classification data set.

The test data set uses CIFAR-10/CIFAR-100 and ImageNet. The CIFAR-10 data set consists of 50K training images and 10K test images, with a size of 32×32 and contains 10 categories; the CIFAR-100 data set contains 100 categories. ImageNet ILSVRC12 contains approximately 1.2 million training images and 500,000 test images with 1000 classes.

In the comparative test, the only post-training sparse (PTS) method POT was selected as the existing technical solution, which reproduces some technologies originally designed for retraining, namely the heuristic non-uniform sparse method ERK in POT, and the PTS setting ProbMask.

The test network selected widely used network structures, including ResNet-32/ResNet-56 for CIFAR-10/CIFAR-100, ResNet-18/ResNet-50, MobileNetV2, RegNet-200M/RegNet- for ImageNet. 400M and Vit-Base/Vit-Large.

Table 2 shows the comparative test results of the Top1/Top5 accuracy (%) under different sparsity rates using the technical solution provided by the embodiment of the present invention and the prior art solution on the CIFAR-10 and CIFAR-100 datasets.

Table 2

It can be seen from Table 2 that as the sparsity increases, the existing technical solution quickly degrades at 80% sparsity and completely collapses at 90% or earlier. In contrast, the technical solution provided by the embodiment of the present invention (denoted by Ours in the table) can maintain very high accuracy stability even at 90% sparsity, especially on the CIFAR-100 dataset.

The accuracy (%) of the original network ResNet-32 without sparsity on the CIFAR-10/100 dataset is 93.53/99.77 and 70.16/90.89 respectively. The accuracy (%) of the original network ResNet-56 without sparsity on the CIFAR-10/100 dataset is 94.37/99.83 and 72.63/91.94 respectively. It can be seen from this that compared with the original network, the accuracy of the network after adopting the technical solution provided by the embodiment of the present invention does not decrease significantly.

Using the technical solution provided by the embodiment of the present invention and the existing technical solution respectively on the data set ImageNet, the Top1/Top5 accuracy comparison test results under different sparsity rates are shown in Table 3. The accuracy of the original neural network model without sparsification is listed below the name of the model architecture.

table 3

As can be seen from Table 3, with the increase of sparsity, the accuracy of the technical solution provided by the embodiment of the present invention in all neural network models is significantly better than that of the existing technical solution, especially when the sparsity rate exceeds 70%. It should be noted that at a sparsity rate of 70%, the technical solution provided by the embodiment of the present invention achieves an accuracy level almost similar to that corresponding to its original neural network model on the networks ResNet-18 and ResNet-50 (only a decrease of 2%), while the accuracy loss of the existing technical solution is nearly 10%. For lightweight architectures such as the networks RegNet and MobileNet, the current technical solution encounters an unacceptable decrease in accuracy at a sparsity of 60%, while the technical solution provided by the embodiment of the present invention can improve by 3% to 8%.

In addition, the comparative test results in the Vit-Base/Vit-Large model are shown in Table 4. The accuracy of the original neural network model without sparsification is listed below the name of the model architecture.

Table 4

It can be seen from Table 4 that the accuracy of the technical solution provided by the embodiment of the present invention in the ViT model is also better than the existing technical solution, which proves that the technical solution provided by the embodiment of the present invention is also applicable to the attention-based model architecture.

Second. Experimental testing on the target detection data set.

The test data set uses PASCAL VOC, which is a widely used object detection data set and contains 20 object categories. Each image has bounding box annotations and object class annotations. The test network structure uses MobileNetV1 SSD and MobileNetV2 SSD-lite. In the comparison test, POT was still selected as the existing technical solution.

The comparative test results of the mean average accuracy (mAP) when the sparsity rate is 90% using the technical solution provided by the embodiment of the present invention and the existing technical solution on the data set PASCAL VOC are shown in Table 5. Among them, the mAP of the original neural network model without sparse is listed under the name of the model architecture.

table 5

As can be seen from Table 5, compared with POT (existing technology) using L2-normalized amplitude and ERK sparsity rate allocation algorithm, the technical solution provided by the embodiment of the present invention under the extreme condition of 90% sparsity rate Has significant performance improvements. For the sparse MobileNetV1 SSD model, the mAP of the technical solution provided by the embodiment of the present invention only dropped by 2.6%; on the compressed MobileNetV2 SSD-Lite model, when the performance of the existing technical solution collapsed, the embodiment of the present invention The provided technical solution can still obtain 59.1% mAP.

Third. Comparative test of the accuracy of sparse rate learnable and sparse rate non-learnable.

The technical solution in which the sparsity rate can be learned is the technical solution provided by the embodiments of the present invention. As a technical solution in which the sparsity rate cannot be learned in the comparative test, it refers to a technical solution that only uses the ERK algorithm to initialize the sparsity of each layer of the model, and then reconstructs the entire network at once.

The comparison test was conducted on three models: ResNet-50, RegNetX-400M and MobileNetV2. First, the ERK algorithm is used to initialize the sparsity of each layer of the model, and then reconstruction is performed under two conditions: whether the sparsity rate can be learned or not. The comparative test results of the accuracy of the sparsity rate that can be learned and the sparsity rate that cannot be learned are shown in Figure 3.

As can be seen from Figure 3, the technical solution provided by the embodiment of the present invention can achieve higher accuracy than the non-learnable technical solution, and as the sparsity rate increases, the technical solution provided by the embodiment of the present invention The accuracy improvement is even more significant. For example, on the model RegNetX-400M, the improvement is about 10% when the sparsity rate is 80%; the improvement is about 25% when the sparsity rate is 90%. Therefore, it is proved that the technical solution provided by the embodiment of the present invention can dynamically optimize the sparse rate allocation due to the learnable sparse rate with global constraints, so it is more reasonable and can obtain unexpected model performance.

Fourth. Comparative test of reconstruction efficiency and inference efficiency.

The test data sets use CIFAR-100 and ImageNet. In the comparative test, POT was selected as the existing technical solution. ResNet-32 and ResNet-18 were selected as test network structures. In the comparative test, the reconstruction time cost of obtaining a sparse neural network was collected and the inference speed was evaluated on real hardware.

The technical solution provided by the embodiment of the present invention obeys the original weight distribution of the original neural network model, and therefore can make full use of historical training results. In addition, this algorithm directly performs network optimization on a smaller optimization space (i.e., threshold space) without any randomness, so it can converge quickly and stably. More importantly, the controllable sparsification rate allows us to achieve the target without repeated hyperparameter tuning. The comparative test results of the reconstruction efficiency using the technical solution provided by the embodiment of the present invention and the existing technical solution are as shown in Table 6.

Table 6

As can be seen from Table 6, the technical solution provided by the embodiment of the present invention can reduce the time to generate a ResNet-32 sparse model to 9 minutes, while existing technical solutions such as POT require more than one and a half hours. The reconstruction efficiency differs by about 10 times.

In addition, the inventor also tested the inference performance of the post-training sparse method on Ambarella CV22. Ambarella CV22 is an autonomous driving chip that supports unstructured sparse acceleration. The test results of this inference performance are shown in Table 7.

Table 7

As can be seen from Table 7, the post-training model sparse method provided by the embodiment of the present invention benefits from the sparsity, and the inference delay and memory usage are significantly reduced. This gain is especially significant for ResNet-18, achieving nearly 2.4x speedup and over 50% memory savings at 70% sparsity.

The above series of comparative experimental tests and evaluations prove that the post-training model sparsification method for computer vision provided by the embodiment of the present invention uses differentiable estimation to achieve a learnable and controllable sparsity rate. Thanks to the optimal sparse allocation, it has achieved state-of-the-art test results on four different data sets covering image classification and target detection tasks, pushing the limits of PTS accuracy and efficiency to a new level.

Based on the above post-training model sparse method for computer vision, embodiments of the present invention further provide a post-training model sparse device for computer vision. As shown in Figure 4, the post-training model sparse device includes one or more processors and memory. Wherein, the memory is coupled to the processor and is used to store one or more computer programs. When the one or more computer programs are executed by one or more processors, the one or more processors implement the computer-oriented implementation in the above embodiment. Post-training model sparse methods for vision.

Wherein, the processor is used to control the overall operation of the post-training model sparse device to complete all or part of the steps of the above-mentioned post-training model sparse method for computer vision. The processor module can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory is used to store various types of data to support operations of the post-training sparse device. These data may include, for example, instructions for any application program or method for post-training the operation of the sparse device, as well as application-related data. The memory module may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, etc.

In summary, compared with the prior art, the post-training model sparse method provided in the embodiment of the present invention establishes a bridge between the sparse threshold and the sparse rate from the perspective of probability density, and further uses the kernel density estimation technology to construct a differentiable representation between the sparse threshold and the sparse rate, which can learn the optimal sparse rate allocation with global constraints without damaging the original weights and retaining the visual features to the greatest extent, and quickly obtain the required sparse neural network. Therefore, the post-training model sparse method provided in the embodiment of the present invention has the beneficial effects of controllable sparse rate allocation, high precision, high efficiency, etc., and can be promoted on various neural network architectures (ResNet, MobileNet, RegNet) and tasks (CIFAR-10/100, ImageNet and PASCAL VOC).

The above is a detailed description of the post-training model sparse method and device for computer vision provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims (10)

1. A post-training model sparse method facing computer vision is characterized by comprising the following steps:

(1) Introducing a control loss to achieve the target sparsity ratio, the control loss being:

wherein r is _i And N _i The sparsity and the element number of the i layer weight are respectively represented; r is (r) ₀ Is a global sparsity target; l (L) _c To control losses;

(2) Taking the reconstruction loss of the weight as supervision to reduce the difference between the sparse output and the original output so as to preserve the image characteristics in the original neural network model, wherein the reconstruction loss is as follows:

L _rec ＝D _KL (Y _dense ||Y _sparse )

wherein L is _rec Loss for reconstruction; d (D) _KL (. Cndot.) represents the Kullback-Leibler divergence function; y is Y _dense Outputting an original neural network model; y is Y _sparse Outputting a sparse post neural network model;

(3) Adding the control loss and the reconstruction loss yields an overall optimized target loss:

L＝L _rec +L _c

wherein L is the target loss;

(4) Determining a reconstructed weight and a reconstructed sparsity rate of each layer most suitable for the target sparsity rate by using the target loss:

wherein arg min is an arg min function; w is the reconstructed weight; r is (r) _i The sparsity of the i-th layer weight;

(5) And converting the function for determining the weight and the sparsity of each layer after reconstruction into a micro-function so as to realize micro-optimization of sparsity distribution.

2. The computer vision oriented post-training model sparseness method of claim 1, wherein in step (2):

the sparse model outputs Y _sparse Calculated by the following formula:

Y _sparse ＝f(X,M⊙W)

where f represents the neural network model, X is a three-dimensional image of the input neural network model, M is a mask, and W is the weight of the neural network model.

3. The computer vision oriented post-training model sparseness method of claim 2, wherein:

the generation of the mask M is calculated by means of an amplitude-based approach:

M＝0.5*sgn(|W|-t)+0.5

wherein sgn represents a sign function, returns +1 if it is a positive value, otherwise returns-1; t is a sparse threshold; w is the weight of the neural network model.

4. The computer vision oriented post-training model sparseness method of claim 1, wherein in step (5):

sparsity r of the target loss L relative to the L layer _l The derivative of (2) is:

wherein L is target loss, L _rec For reconstruction loss, lc is control loss, r _l The sparseness of layer i.

5. The computer vision oriented post-training model sparseness method of claim 4, wherein:

sparseness r of control loss Lc relative to layer l _l The derivative of (2) is:

wherein r is _i And N _i The sparsity and the element number of the weight of the ith layer are respectively expressed, r _l And N _l The sparseness of the first layer weight and the number of elements are respectively represented.

6. The computer vision oriented post-training model sparseness method of claim 4, wherein:

the reconstruction loss L _rec Sparseness r relative to layer l _l The derivative of (2) is:

wherein M is _l Mask for layer l, t _l Is the sparseness threshold of layer i.

7. The computer vision oriented post-training model sparseness method of claim 1, wherein in step (5):

sparseness r of layer l _l The calculation satisfies the formula:

wherein p (W) is a weight W _l Probability density function, W _l A weight distribution for layer l, w being the sampling weights from the distribution; g ^-1 As an inverse function of the microcompatible function g, t _l ＝g(r _l )；t _l Is the sparseness threshold of layer i.

8. The computer vision oriented post-training model sparseness method of claim 7, wherein:

sparseness r of layer l _l For the sparse threshold t _l The derivative of (2) is:

wherein p (t) _l ) At t _l Probability density function, p (-t) _l ) Is-t _l Probability density functions of (2); r is (r) _l For the sparseness of layer l, t _l Is the sparseness threshold of layer i.

9. The computer vision oriented post-training model sparseness method of claim 7, wherein:

the estimation of p (w) using the nuclear density estimation method is as follows:

wherein p (W) is a weight W _l Probability density functions of (2); k is a non-negative kernel function; n is the number of sampling points; h is bandwidth, h>0; the kernel function with subscript h is a scaling kernel function, defined asW is the weight distribution W of the first layer _l Sampling weights of (a); w (w) _i Is the weight of the i-th layer.

10. A post-training model sparse device for computer vision tasks, comprising a processor and a memory, the processor and the memory being coupled; wherein,

the memory is used for storing a computer program;

the processor is configured to execute a computer program stored in the memory, and perform the computer vision oriented post-training model sparseness method of any of claims 1-9.