CN118036672A - A Neural Network Optimization Method Based on Taylor Expansion Momentum Correction - Google Patents

Abstract

The invention discloses a neural network optimization method based on Taylor expansion momentum correction. Aiming at the defects that the Adam algorithm is high in convergence speed and is inferior to SGDM algorithm in generalization effect, the method combines the second-order Taylor expansion of the activation function with the parameters softplus with the second-order momentum, realizes gradual smooth transition from Adam to SGDM, and specifically comprises the following steps: s1, data preprocessing and batch division; s2, initializing a neural network model and super parameters; s3, calculating the gradient, the first-order momentum and the second-order momentum of the model parameters, correcting the second-order momentum by using a second-order Taylor expansion of the function activated by the band parameters softplus, and updating the model parameters; s4, accumulating the iteration round number until the specified maximum round number is reached. Compared with a plurality of currently popular neural network optimization algorithms, the method has the advantages of faster convergence speed and better generalization.

Description

A Neural Network Optimization Method Based on Taylor Expansion Momentum Correction

Technical Field

The present invention belongs to the technical field of deep neural networks, and in particular relates to a neural network optimization method based on Taylor expansion momentum correction, which can successfully achieve a gradual and smooth transition from Adam to SGDM.

Background technique

A deep neural network (DNN) is a type of computational model composed of multiple layers of neurons. It has a deep structure and usually includes an input layer, multiple hidden layers, and an output layer. By stacking multiple hidden layers, a deep neural network can extract and express the abstract features of data layer by layer. Deep neural networks are widely used in the fields of machine learning and artificial intelligence, including tasks such as image recognition, speech recognition, and natural language processing. Some common deep neural network structures include convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM).

The process of training a deep neural network usually involves a large amount of data and parameter adjustments, in which optimization algorithms play a vital role. The optimization of a deep neural network refers to adjusting network parameters and updating algorithms so that the network can better fit the training data and improve its generalization ability on unseen data.

Gradient descent (GD) is one of the most commonly used optimization algorithms, but there are some problems, such as local minima and the choice of learning rate. In order to solve these problems, researchers have proposed a series of improved optimization algorithms, such as stochastic gradient descent (SGD), momentum method (SGDM) and adaptive learning rate methods (such as Adam, Adagrad, etc.). Among them, the Adam algorithm has achieved remarkable success in fields such as image processing and natural language processing due to its fast, robust and easy implementation, and has become one of the most popular optimization algorithms in the field of deep learning.

However, in practical applications, although these algorithms have shown excellent performance in different situations, there are still some common problems. Specifically, when optimizing deep neural networks, traditional adaptive learning methods (such as Adam) may not be as good as the stochastic gradient descent method with momentum (SGDM) in terms of generalization performance, despite their fast convergence speed. Therefore, how to overcome the disadvantages of traditional adaptive methods in generalization performance while maintaining a fast convergence speed has become a problem that researchers are eager to solve.

Summary of the invention

In view of the problem that existing neural network optimization algorithms cannot balance generalization and convergence speed, the present invention proposes a neural network optimization method based on Taylor expansion momentum correction. The optimization method combines the second-order momentum term and the second-order Taylor expansion with parameter softplus activation function, so that the algorithm has both good generalization and fast convergence speed, which solves the defects of existing optimization algorithms to a certain extent.

The present invention provides a neural network optimization method based on Taylor expansion momentum correction, comprising the following steps:

S1: After preprocessing the data set, divide it into training set and test set to determine the size of the data batch during optimization;

S2: Randomly initialize the parameters of the neural network model and assign values to the hyperparameters and process control parameters of the optimization process;

S3: Input the training set data into the neural network model batch by batch according to the batch size determined in step S1; when each batch of data is input, calculate the gradient, first-order momentum and second-order momentum of the loss function in the neural network model under the current batch of training data, and correct the second-order momentum to obtain the corrected second-order momentum, and then iteratively update the neural network model parameters based on the first-order momentum and the corrected second-order momentum; each time the neural network model parameters are updated, the number of iterations is accumulated by 1 until the current data batch is the last batch of all data in the training set, then the number of iterations is accumulated by 1;

S4: If the number of iterations reaches the set maximum number of iterations epoch, output the final neural network model parameters, otherwise return to step S3.

Furthermore, the step S1 includes the following sub-steps:

S11: randomly crop, horizontally flip, and standardize each original image of the original data set in sequence;

S12: Divide the data set processed in step S11 into a training set and a test set in proportion, and convert both the training set and the test set into data tensors; determine the size of the data batch during optimization.

Furthermore, the step S2 includes the following sub-steps:

S21: Set the input dimension and output dimension of each layer and module of the neural network model and connect them, where the initial convolution layer includes a convolution kernel and output channels with the number of output dimensions, and a batch normalization layer is added after the output channel; the initial convolution layer is followed by a residual block and a fully connected layer; the activation function uses ReLU.

S22: The batch normalization layer, the convolution layer in the residual block, and the fully connected layer parameters are initialized using uniformly distributed random numbers. The range of the initialization distribution of the convolution layer and the fully connected layer parameters in the residual block is determined by the dimension of the neural network model parameters.

S23: Assign values to the hyperparameters and process control parameters of the optimization process.

Furthermore, the first-order momentum m _t =β ¹ _t m _t-1 +(1-β ¹ _t )g _t , and the second-order momentum Among them,/> g _t is the gradient of the neural network model parameters, β ¹ _t is the first-order attenuation coefficient, β ² _t is the second-order attenuation coefficient, the initial value of the first-order momentum m _t is 0, and the initial value of the second-order momentum v _t is 0.

Furthermore, the correction of the second-order momentum comprises the following sub-steps:

①: Substitute the smoothing parameter into the softplus activation function with parameters;

②: Perform a second-order Taylor expansion at zero for the softplus activation function with parameters obtained in step ①, and discard the remainder;

③: Substitute the second-order momentum into the second-order Taylor expansion of step ② to obtain the corrected second-order momentum.

Furthermore, the iterative updating of the neural network model parameters based on the first-order momentum and the modified second-order momentum includes the following sub-steps:

①: Update the first-order attenuation coefficient and the second-order attenuation coefficient, and calculate the learning rate based on the new first-order attenuation coefficient and the second-order attenuation coefficient;

②: Multiply the learning rate, the first-order momentum, and the reciprocal of the arithmetic square root of the modified second-order momentum to obtain the update amount of the neural network model parameters;

③: Subtract the neural network model parameter value from the update amount of the neural network model parameter to obtain the new neural network model parameter value.

Furthermore, the maximum number of iterations epoch in step S4 is the number of times all the data in the training set are traversed.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention realizes an improvement from the Adam type optimization algorithm and the SGDM algorithm, smoothes the second-order momentum through the second-order Taylor expansion with a parameter softplus function, and constrains the extreme values of the adaptive learning rate, thereby improving the stability and robustness of the adaptive learning rate, which helps to avoid the adverse effects of excessive or too small learning rates on training.

2. The present invention eliminates the need for sensitive parameters ∈ in the original implementation of the Adam algorithm: the second-order Taylor expansion with parameter softplus function has a non-zero numerical lower bound when used on non-negative variables, avoiding dependence on ∈, and the parameters in the expansion can be flexibly set as needed, greatly simplifying the optimization process.

3. The hyperparameter of the second-order Taylor expansion in the present invention is a controller for smoothing the anisotropy of the learning rate, which realizes the gradual transition from Adam to SGDM, so that the present invention has the advantages of fast convergence speed of the Adam algorithm and strong generalization performance of the SGDM algorithm: when the hyperparameter is small, the performance of the present invention is similar to SGDM, and when the hyperparameter is large enough, the present invention degenerates into the original Adam optimization method.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a neural network optimization method based on Taylor expansion momentum correction provided by the present invention.

Figure 2 is a performance comparison of different optimization methods for training the MNIST dataset on a multi-layer perceptron (MLP).

Figure 3 is a performance comparison of different optimization methods for training the MNIST dataset on the residual network ResNet-18.

Figure 4 is a performance comparison of different optimization methods for training the CIFAR-10 dataset on the convolutional network LeNet-5.

Figure 5 is a performance comparison of different optimization methods for training the CIFAR-10 dataset on the residual network ResNet-20.

Detailed ways

The following is a further detailed description of the embodiments of the present invention in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention. Those skilled in the art can combine the technical features in the embodiments to form new embodiments without creative work, and these new embodiments are also covered within the scope of protection of the present invention.

The present invention is described in detail below by taking the training of CIFAR-10 dataset on ResNet-20 as an example and combining with the accompanying drawings.

The present invention provides a neural network optimization method based on Taylor expansion momentum correction. The basic idea is to combine Adam and SGDM, two optimization algorithms with their own advantages, and correct the second-order momentum term with the second-order Taylor expansion of the softplus activation function to achieve a smooth transition from Adam to SGDM. By controlling parameters, the algorithm exhibits the fast convergence characteristics of Adam in the early stage of training and the strong generalization performance characteristics of SGDM in the later stage of training, thereby improving the accuracy and convergence speed of the model.

As shown in FIG1 , the neural network optimization method based on Taylor expansion momentum correction provided by the present invention specifically comprises the following steps:

Step 1: Preprocess the original data set (step 1.1 to step 1.3) and segment it, taking the CIFAR-10 data set as an example.

Step 1.1: Randomly crop each original image in the original dataset to a 32×32 pixel area. If the original image size is smaller than 32×32, a 4-pixel border will be padded around the image before cropping. This operation helps increase the diversity of the data and simulates the image size changes that may occur in actual applications.

Step 1.2: Flip the cropped image horizontally with a probability of 50%. This data augmentation method can improve the model's robustness to left-right symmetry.

Step 1.3: Normalize the image processed in step 1.2. Use formula (1) to scale the pixel value range to [-1, 1].

oldValuei represents the value of a single pixel of the i-th channel in the image before normalization, NewValuei represents the result of normalization, meani is the mean value of the pixels of the i-th channel, stdi is the standard deviation of the pixels of the i-th channel, the three RGB channels are respectively recorded as the first channel, the second channel, and the third channel, the meani of each channel is recorded in mean, the stdi of each channel is recorded in std, mean = [mean ₁ , mean ₂ , mean ₃ ] and std = [std ₁ , std ₂ , std ₃ ]. It should be noted that, in this embodiment, mean = [0.4914, 0.4822, 0.4465] and std = [0.2023, 0.1994, 0.2010].

Step 1.4, segmentation: divide the data set and convert the data into computable tensors. The training set consists of 50,000 images, which are randomly shuffled and spliced into a data tensor of 32×32×3×50,000. The test set consists of 10,000 images, which are not shuffled and directly spliced into a data tensor of 32×32×3×10,000. Set the data batch size batch_size to 128, and finally treat it as a batch if it is less than a complete batch.

Step 2: Randomly initialize the parameters of the ResNet-20 model, giving specific values of the hyperparameters and control parameters involved in the training process.

Step 2.1, set ResNet-20 to 3-channel input, the initial convolution layer includes a 3×3 convolution kernel and 16 output channels, and a batch normalization layer is added after the 16 output channels; then it is divided into 3 stages, each stage contains 3 residual blocks, each residual block has two 3×3 convolution layers, and the global average pooling layer is used to reduce the dimension of the feature map. The final fully connected layer outputs the category probability, and the number of output categories is 10; the activation function uses ReLU.

Step 2.2: The parameters of the batch normalization layer are initialized using uniformly distributed random numbers in the range [0, 1]. The parameters of the convolutional layer and the fully connected layer in the residual block are initialized using uniformly distributed random numbers, and the distribution range is taken as the inverse of the arithmetic square root of the dimension of the parameter θ, that is, formula (2).

Where θ represents all parameters of the model, the subscript represents the number of iterative updates, θ ₀ is the 0th iterative update, i.e. initialization; size(θ) represents the dimension of parameter θ, and rand(·,·) represents taking a uniformly distributed random number in the corresponding interval.

Step 2.3, assign values to the hyperparameters and process control parameters of the optimization process, including: maximum number of iterations epoch = 101, current iteration number epoch_temp = 0, initial value of learning rate α _t = 0.01, first-order attenuation coefficient β ¹ _t = 0.9, second-order attenuation coefficient β ² _t = 0.999, smoothing parameter β = 50; number of iteration updates t = 1, initial value of first-order momentum m _t = 0, initial value of second-order momentum v _t = 0.

Step 3: According to batch_size=128, the concatenated data tensors of the training set are divided into batches and sent to the ResNet-20 model for training. Specifically, the predicted value of the ResNet-20 model for the current data batch and the true value of the current data batch are substituted into the loss function f(θ), and the gradient of the loss function to the ResNet-20 model parameters is calculated using the chain rule. f(θ _t ) is the updated loss function f(θ) value at the tth iteration,/> is the gradient operator.

Compute the first-order momentum:

m _t = β ¹ _t m _t-1 + (1-β ¹ _t ) g _t (3)

Second-order momentum:

Next, the second-order momentum is corrected as follows:

The softplus activation function with parameters, that is, formula (5), is subjected to a second-order Taylor expansion at x=0, and the remainder is discarded to obtain formula (6).

Substituting the second-order momentum _vt into x in formula (6), we get the modified second-order momentum

Finally, update the first-order attenuation coefficient and the second-order attenuation coefficient, calculate the learning rate, and update the model parameter θ once, specifically:

According to formula (7), the new first-order attenuation coefficient and second-order attenuation coefficient are calculated:

Calculate the new learning rate according to formula (8)

Update the model parameters once according to formula (9).

The number of iterations t is accumulated by 1 until the current data batch is the last batch, then the current iteration round number epoch_temp is accumulated by 1.

Step 4: If epoch_temp<epoch, return to step 3, otherwise terminate the training and output the final model parameters θ.

In order to verify the effectiveness of the neural network optimization method based on Taylor expansion momentum correction (denoted as MyAdam) proposed in the present invention, the deep learning framework Pytorch was used to test it on two classic image classification datasets MNIST and CIFAR-10, and the loss function value (Loss) of the training process, the test set prediction accuracy (Accuracy) and other indicators were recorded for comparison with several classic and effective optimization algorithms such as SGDM, Adam, Adabound, and Sadam.

In order to demonstrate the effectiveness and high generalization of the algorithm of the present invention, the experiment used four different neural network models, namely, multi-layer neural network MLP, residual network ResNet-18, convolutional network LeNet-5, and residual network ResNet-20. The number of training rounds was 101, and the experimental results are shown in Figures 2, 3, 4, and 5. Except for the multi-layer neural network MLP, the structures of the other three networks are fixed structures. The following is the network and parameter settings of the multi-layer neural network MLP:

The input layer of the network has a dimension of 28×28 (784 dimensions). The first hidden layer has 1000 neurons, and the activation function uses ReLU, followed by 50% Dropout (drop rate is 0.5). The second hidden layer also has 1000 neurons, and the activation function is also ReLU, and 50% Dropout is used again. The final output layer has 10 neurons, where the output layer uses the softmax function for multi-category classification, and the default initialization method of Pytorch is used for initialization.

As shown in Figures 2, 3, 4 and 5, for two data sets with different classification difficulties and different model structures, the neural network optimization method based on Taylor expansion momentum correction of the present invention performs better in convergence speed and generalization accuracy than SGDM, Adam, Adabound, Sadam and other methods. It can also be seen that the method of the present invention makes up for the deficiency that although the convergence speed of the Adam method is faster than SGDM, the generalization performance is weaker than SGDM.

In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. A neural network optimization method based on Taylor expansion momentum correction is characterized by comprising the following steps of: the method comprises the following steps:

S1: preprocessing a data set, dividing the preprocessed data set into a training set and a testing set, and determining the size of a data batch in optimization;

S2: randomly initializing parameters of the neural network model, and assigning super parameters and flow control parameters of the optimization process;

S3: inputting the training set data into a neural network model batch by batch according to the batch size determined in the step S1; when each batch of data is input, calculating the gradient, the first-order momentum and the second-order momentum of the neural network model parameters by a loss function in the neural network model under the current batch of training data, correcting the second-order momentum to obtain corrected second-order momentum, and carrying out iterative update on the neural network model parameters based on the first-order momentum and the corrected second-order momentum; accumulating the iteration times by 1 when the neural network model parameters are updated once, and accumulating the iteration times by 1 when the current data batch is the last batch in all data of the training set;

S4: and outputting final neural network model parameters if the iteration round number reaches the set maximum iteration round number epoch, otherwise, returning to the step S3.

2. The neural network optimization method based on taylor expansion momentum correction according to claim 1, wherein the method comprises the following steps of: said step S1 comprises the sub-steps of:

S11: sequentially carrying out random cutting, horizontal overturning and standardization on each original image of an original data set;

S12: dividing the data set processed in the step S11 into a training set and a testing set according to a proportion, and converting the training set and the testing set into data tensors; the size of the data batch at the time of optimization is determined.

3. The neural network optimization method based on taylor expansion momentum correction according to claim 1, wherein the method comprises the following steps of: said step S2 comprises the sub-steps of:

S21: setting input dimensions and output dimensions of each layer and module of the neural network model and connecting the layers, wherein an initial convolution layer comprises a convolution kernel and output channels with the number of the output dimensions, and adding a batch normalization layer after the output channels; the initial convolution layer is followed by a residual block and a full connection layer in sequence; the activation function adopts a ReLU;

S22: uniformly distributed random numbers are used for initializing parameters of the batch normalization layer, the residual intra-block convolution layer and the full connection layer, and the range of the parameter initialization distribution of the residual intra-block convolution layer and the full connection layer is determined by the dimension of the neural network model parameters;

s23: and assigning values to the super-parameters and the flow control parameters of the optimization process.

4. The neural network optimization method based on taylor expansion momentum correction according to claim 1, wherein the method comprises the following steps of: the first order momentum mt=beta ¹tmt-₁+(1-β¹ t) gt, the second order momentumWherein,Gt is the gradient of the neural network model parameters, beta ¹ _t is a first-order attenuation coefficient, beta ² _t is a second-order attenuation coefficient, the initial value of the first-order momentum m _t is 0, and the initial value of the second-order momentum v _t is 0.

5. The neural network optimization method based on taylor expansion momentum correction according to claim 1, wherein the method comprises the following steps of: the second order momentum is corrected, which comprises the following substeps:

① : substituting the smoothing parameters into the band parameter softplus to activate the function;

② : performing second-order Taylor expansion on the activation function of the band parameter softplus obtained in the step ① at zero, and discarding the remainder;

③ : substituting the second order momentum into the second order taylor expansion of step ② results in a modified second order momentum.

6. The neural network optimization method based on taylor expansion momentum correction according to claim 4, wherein the method comprises the following steps: the iterative updating of the neural network model parameters based on the first-order momentum and the corrected second-order momentum comprises the following sub-steps:

① : updating the first-order attenuation coefficient and the second-order attenuation coefficient, and calculating the learning rate according to the new first-order attenuation coefficient and the new second-order attenuation coefficient;

② : multiplying the learning rate, the first-order momentum and the inverse of the arithmetic square root of the corrected second-order momentum to obtain the updating quantity of the neural network model parameters;

③ : and obtaining a new neural network model parameter value by making a difference between the neural network model parameter value and the update quantity of the neural network model parameter.

7. The neural network optimization method based on taylor expansion momentum correction according to claim 1, wherein the method comprises the following steps of: and in the step S4, the maximum iteration round number epoch is the number of times of traversing all data of the training set.