CN111126599B - A neural network weight initialization method based on transfer learning - Google Patents

Abstract

The invention belongs to the technical field of neural network models, and particularly relates to a neural network weight initialization method based on transfer learning. In the method, for a designated target task, a neural network model with higher complexity, namely a teacher model, is designed, the teacher model is trained, and after the training is finished, the weight initialization of the student model is guided by using the generated feature map; calculating the difference between the feature maps or mapping the feature maps into a regenerated kernel Hilbert space, calculating the difference of the feature maps in the regenerated kernel Hilbert space, and simplifying the calculation by adopting a kernel function method; the simple student model achieves a better weight initialization effect, and after the weight initialization is completed, the student model is generally trained, so that the student model achieves a better global convergence point, and the performance of the student model is more excellent. The invention can effectively improve the performance of the student model on the premise of not increasing the complexity of the student model.

Description

A neural network weight initialization method based on transfer learning

technical field

The invention belongs to the technical field of neural network models, and in particular relates to a neural network weight initialization method based on knowledge transfer learning.

Background technique

Neural networks have made great progress in recent years, especially in the fields of computer vision and natural language processing, and many of their performances have surpassed those of humans. application has caused great obstacles. Therefore, how to make a lightweight model perform better has become a hot issue that needs to be solved.

In the past few years, many researchers have proposed various schemes to help neural networks achieve a better convergence effect. It mainly includes the following categories. One is based on knowledge distillation and knowledge transfer. It tries to use a trained teacher model to help the student model perform better by adding some additional loss functions to the training process of the student model, thereby The performance of the model is improved without increasing the complexity of the student model. The second type is model-based quantitative pruning. By quantifying the weights of the neural network, the original 32-bit addition and subtraction is changed to 8-bit or even 1-bit addition and subtraction, thereby greatly reducing the neural network. weight complexity. Thereby reducing the amount of calculation. Pruning is to directly delete some connection edges connected by the neural network, and then evaluate whether the loss of the accuracy of the model caused by pruning is negligible, thereby achieving the effect of effectively reducing the complexity of the model.

Contents of the invention

The purpose of the present invention is to propose a neural network weight initialization method that can effectively improve the performance of the model without increasing the complexity of the model.

The neural network weight initialization method provided by the present invention is based on knowledge transfer learning technology, that is, for a neural network model with high complexity, it is called a teacher model (the teacher model with high complexity is difficult to apply in actual engineering. ); learn prior knowledge from the more complex teacher model, and thus help the less complex neural network model (called the student model, which has a better trade-off between complexity and performance in practical applications) Have a good initialization state, so as to get out of the local optimal point in the training process, so as to achieve better training effect.

In the present invention, firstly, for the specified task, a neural network model with high complexity is designed, namely the teacher model, and the teacher model is trained. After the training is completed, the generated feature map is used to guide the weight initialization of the student model; The difference between them, or map the feature map to the regenerating kernel Hilbert space, calculate the difference in the regenerating kernel Hilbert space, and use the kernel function method to simplify the calculation; make the simple student model better After the weight initialization is completed, the student model is generally trained, so that the student model reaches a better global convergence point, and its performance is more excellent.

The invention can effectively make the student model avoid the problem of converging to a local optimal solution caused by parameter dependence in the initial training.

The neural network weight initialization method that the present invention proposes, concrete steps are as follows:

(1) For a specific learning task, it often has a conventional loss function and model structure. First, design the teacher model for the target task, and use the conventional loss function to train the teacher model;

(2) Then export the output of the middle layer of the trained teacher model, and obtain the feature map through the mapping method; among them, the mapping method can have attention transfer [Sergey Zagoruyko and Nikos Komodakis, "Paying more attention to attention: Improving the performance of convolutional neuralnetworks via attention transfer,” arXiv preprint arXiv:1612.03928, 2016], or use kernel functions to map to the regenerated kernel Hilbert space [Zehao Huang and Naiyan Wang, “Like what you like:Knowledge distill via neuron selectivity transfer,” arXiv preprint arXiv:1707.01219,2017.]; specifically shown in formula (2) and formula (3);

(3) Design a student model with a relatively simple structure, which requires the same network structure as the teacher model; that is, the basic network layers that make up the network should be consistent, for example, the network structure is based on a serial connection network composed of convolutional layers , while the teacher model has more convolutional layers and more feature maps, and the student model has fewer convolutional layers and fewer feature maps;

(4) Use the mean square error between the feature map calculated in step (2) and the feature map obtained by mapping the student model in the same way as the loss function to train the student model; after the training, the obtained student model's The weight is not the initialization of normal distribution or uniform distribution in the traditional sense, but a kind of learning knowledge from the teacher model to adjust the weight of the student model, so that the weight of the student model can be initialized specifically, so that The student model has the ability to approach the performance of the teacher model;

(5) After the initialization is completed, the student model is finally trained using a conventional loss function to obtain a usable student model.

Wherein, the conventional loss function refers to the mean square error often used in the current task.

In the present invention, the complexity of the teacher model is higher than that of the student model, so that the student model can learn the characteristics of the teacher model well.

Description of drawings

Fig. 1 schematic diagram of the method of the present invention.

Figure 2. Training loss and test loss (QP=22) for experiments.

Figure 3. Training loss and test loss (QP=37) for experiments.

Detailed ways

The present invention will be further described below by taking the neural network-based loop filtering task in video coding as an example.

For the target task, it is necessary to add a neural network module to a traditional video encoder such as HEVC, and the function of this module is loop filtering. The performance of the video encoder is improved by a neural network-based in-loop filtering method. It can be understood as a noise reduction filtering problem, which removes artificial marks and noise brought by traditional video encoders. We first design a teacher model with relatively high complexity. Its complexity should be significantly higher than the complexity of the final target actual target application. For example, its computational complexity and computational resources consumed are more than twice that of the expected design model; use The conventional loss function trains the teacher model to obtain a trained teacher model. For the loop filtering task based on neural network, we designed a convolutional neural network, its structure is shown in Figure 1, the upper part is the designed teacher model structure, and the lower part is the designed student model structure. The conventional loss function here refers to the mean square error that is often used in the current task. This loss function is used to train the teacher and student models.

In terms of model structure, we use depthwise separable convolution and batch normalization as the main layer of the teacher model, with a feature layer of 64 and a convolution kernel size of 3x3. Among them, 24 depth-separable convolution layers are used as the backbone of the teacher model, which is divided into three parts. The first part consists of 10 depth-separable convolution layers, and the second part consists of 8 depth-separable convolution layers. , the third part consists of 6 depthwise separable convolutional layers. The last layer of the model is an ordinary convolutional layer, the number of feature layers is 1, and the convolution kernel size is 1x1. All depthwise separable convolutions have a ReLU activation function. The input of the model is connected to the final output through a direct connection, so that the neural network is in a state of residual learning, which makes the model converge faster.

The input and output of the model are the reconstructed pixel map and the filtered pixel map of the video encoder respectively, and the loss function L _T is the output of the neural network and raw pixels /> The mean square error between:

After the training of the teacher model is completed, the output of the middle layer of the three sub-parts of the teacher model is obtained from the training data set, and the output of the teacher model in these places is calculated from the neural network attention map or the mapping result of the regenerated kernel Hilbert space Result F _T . A new data set is formed by using the calculation result F _T of the middle layer of the teacher model and the data set input by the teacher model to train the student model.

The construction of the student model needs to be similar to the teacher model to ensure the success of knowledge transfer. So we also adopted a similar network structure, using 9 depth-separable convolutional layers as the backbone of the student model, and dividing it into three parts, each of which consists of 3 depth-separable convolutional layers. The number of feature layers is 32, and the convolution kernel size is 3x3. At the same time, because the design goals of the student model and the teacher model are the same, their input and output are also consistent, and the structure of directly connected edges is also used for the good learning residual of the student model.

Make the intermediate layer output of the teacher model and the intermediate layer output of the student model go through the same mapping, and calculate the mean square error between them, etc. Its loss function is specifically expressed as formulas (2) and (3). Computing its attention map or using a linear kernel function k(x, y) = x ^T y to approximate the result of mapping in the regenerating kernel Hilbert space. Its calculation formula is as follows:

Here, F _T , F _S denote the attention maps of the teacher and student models respectively, and /> Then specifically represent the i-th feature map, C _T and C _S respectively represent the number of feature layers of the teacher model and the student model. For practical applications, p takes a positive integer. Thus, an initialized student model is obtained.

Then, use the standard mean square error _LS as shown in formula (4) to carry out final training on the student model, and the trained student model can be obtained. corresponding Represents the output of the student model.

After such initialization, lightweight models can often perform better than without initialization. After obtaining the student model, the student model can be imported back to the video encoder to realize the loop filtering based on the neural network in the video coding. As shown in Figure 3, the training and test losses during the experiment are shown. It can be clearly seen that after initialization is used, the loss function decreases faster.

Claims (1)

1. A neural network-based loop filter optimization method in video coding, at first adding a neural network module in a traditional video encoder, the function of this module is a loop filter; for loop filter target tasks, the neural network The network module includes a more complex neural network model, the teacher model, and a simpler student model; Train the teacher model. After the training is completed, use the generated feature map to guide the weight initialization of the student model; by calculating the difference between the feature maps, or mapping the feature map to the regenerating kernel Hilbert space, calculate its regeneration. The difference in the kernel Hilbert space, using the kernel function method to simplify the calculation; to make the simple student model achieve better weight initialization effect, after the weight initialization is completed, the student model is generally trained, so that the student model To achieve a better global convergence point, its performance is more excellent; the specific steps are as follows: (1) For a specific learning task, it has a conventional loss function and model structure. First, design a teacher model for the target task, and use the conventional loss function to train the teacher model; (2) Then export the output of the middle layer of the trained teacher model, and obtain the feature map through the mapping method; among them, the mapping method has attention transfer, or uses the kernel function to map to the regenerated kernel Hilbert space; (3) Design a student model with a relatively simple structure, which requires the same network structure as the teacher model, that is, the basic network layer that constitutes the network is consistent; when the network structure is a serial connection network based on convolutional layers, The teacher model has more convolutional layers and more feature maps, while the student model has fewer convolutional layers and fewer feature maps; (4) Use the mean square error between the feature map calculated in step (2) and the feature map obtained by mapping the student model in the same way as the loss function to train the student model; after the training, the obtained student model The weight of is to adjust the weight of the student model by learning knowledge from the teacher model, so that the weight of the student model can be initialized specifically, so that the student model has the ability to approach the performance of the teacher model; (5) After the initialization is completed, the student model is trained using a conventional loss function to obtain a usable student model; Using depth-separable convolution and batch normalization as the main layer of the teacher model, its feature layer number is 64, and the convolution kernel size is 3x3; among them, 24 depth-separable convolution layers are used as the backbone of the teacher model, and it is divided into For three parts, the first part consists of 10 depthwise separable convolutional layers, the second part consists of 8 depthwise separable convolutional layers, and the third part consists of 6 depthwise separable convolutional layers; the last layer of the model The ordinary convolution layer is selected, the number of feature layers is 1, and the size of the convolution kernel is 1x1; all depth-separable convolutions have a ReLU activation function; the input of the model is connected to the final output through a direct connection, so that the neural The network is in a state of residual learning, resulting in faster convergence; The input and output of the model are the reconstructed pixel map and the filtered pixel map of the video encoder respectively, and the loss function L _T selects the output of the neural network and raw pixels /> The mean square error between: After the training of the teacher model is completed, the output of the middle layer of the three sub-parts of the teacher model is obtained from the training data set, and the output of the teacher model in these places is calculated from the neural network attention map or the mapping result of the regenerated kernel Hilbert space Result F _T ; use the middle layer calculation result F _T of the teacher model and the data set input by the teacher model to form a new data set to train the student model; The student model adopts a network structure similar to the teacher model, using 9 depth-separable convolutional layers as the backbone of the student model, and dividing it into three parts, each of which is composed of 3 depth-separable convolutional layers, and its feature layer The number is 32, and the size of the convolution kernel is 3x3; the goals designed by the student model and the teacher model are consistent, and their input and output are also consistent, and the structure of straight edges is also used for the learning residual of the student model; Make the intermediate layer output of the teacher model and the intermediate layer output of the student model go through the same mapping, and calculate the mean square error between them; the loss function is specifically expressed as formulas (2) and (3); calculate its attention map or use The linear kernel function k(x, y)=x ^T y is used to approximate the result of mapping in the regenerating kernel Hilbert space; its calculation formula is as follows: Here, F _T and F _S represent the attention maps of the teacher and student models respectively, f _T ⁱ and f _S ⁱ specifically represent the i-th feature map, C _T and C _S represent the number of feature layers of the teacher model and the student model, respectively, p takes a positive integer; thus, the initialized student model is obtained; Use the standard mean square error _LS shown in formula (4) to perform final training on the initialized student model, that is, to obtain the trained student model; the corresponding Y _S ⁱ represents the output of the student model; Import the student model back into the video encoder, that is, implement neural network-based loop filtering in video encoding.