WO2019237357A1 - Method and device for determining weight parameters of neural network model - Google Patents

Abstract

A method for determining weight parameters of a neural network model, comprising: processing sample data on the basis of to-be-determined weight parameters of a neural network model, to obtain an output result; calculating an original error value between the output result and a preset expected result, the original error value being a numerical representation of the difference between the output result and the expected result; correcting the original error value on the basis of a correction value, to obtain a corrected error value; determining, on the basis of the corrected error value and the to-be-determined weight parameters, model weight parameters of the neural network model; wherein the correction value is obtained according to the following formula: R＝(w k-Q(w k))×Q(w k)，R representing the correction value, w k representing the k-th to-be-determined weight parameter of the neural network model,Q(w k) representing a quantization value of the k-th to-be-determined weight parameter, k being a non-negative integer.

Description

Method and equipment for determining weight parameters of neural network model Technical field

The present application relates to the field of data processing technology, and in particular, to a method and device for determining weight parameters of a neural network model.

Background technique

In recent years, neural network models have shown extremely superior performance in applications such as computer vision and speech processing, and have received widespread attention. The cost of the success of the neural network model is the introduction of a large number of parameters and calculations, and the quantization technology of the relevant model parameters of the neural network model can reduce the redundancy of the accuracy of the relevant model parameters, and achieve the premise of reducing the adverse impact on the accuracy of the model Purpose of model compression.

Model compression can not only reduce the consumption of memory bandwidth and energy consumption for data access, but low-precision operations often also bring lower operation energy consumption. For some calculation units that support multiple precision calculations, the number of low-precision calculations per unit time is higher than the number of times that high-precision calculations can be completed.

Summary of the Invention

The embodiments of the present application provide a method and a device for determining a weight parameter of a neural network model. In various data processing scenarios applied to the neural network model, such as image recognition, speech recognition, image super-resolution processing, etc., the model can be trained by The error between the output result and the expected result introduces a proper correction value, which reduces the quantization error and avoids the problem of over-fitting caused by some weight parameters with larger numerical values leading the reasoning results of the neural network.

To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

In a first aspect, an embodiment of the present application provides a method for determining a weight parameter of a neural network model, including: processing sample data based on a neural network model's pending weight parameters to obtain an output result; calculating the output result and a preset The original error value of the expected result, the original error value being a numerical representation of the difference between the output result and the expected result; based on the correction value, correcting the original error value to obtain a corrected error value; based on The correction error value and the pending weight parameter determine a model weight parameter of the neural network model; wherein the correction value is obtained according to the following formula:

R = (w _k -Q (w _k )) × Q (w _k )

R represents the correction value, w _k represents the k-th pending weight parameter of the neural network model, Q (w _k ) represents a quantized value of the k-th pending weight parameter, and k is a non-negative integer.

The embodiment of the present application introduces a proper correction value to the error between the output result of the model training and the expected result, thereby reducing the quantization error and avoiding the overfitting caused by some of the larger weight parameter leading the reasoning results of the neural network. problem.

In a first feasible implementation manner of the first aspect, the correction error value is obtained according to the following formula:

Among them, E1 represents the correction error value, E0 represents the original error value, α is a constant, m is the total number of pending weight parameters used to process the sample data, and F ((w _k -Q (w _k ) ) × Q (w _k )) represents a function in which the correction value is an independent variable, and m is a positive integer.

In a second feasible implementation manner of the first aspect, the absolute value of the correction value is calculated by using the function of the correction value as an independent variable; correspondingly, the correction error value is obtained according to the following formula:

Among them, | (w _k -Q (w _k )) × Q (w _k ) | means calculating the absolute value of (w _k -Q (w _k )) × Q (w _k ).

In a third feasible implementation manner of the first aspect, the neural network model includes p network layers, each of the network layers includes q the pending weight parameters, and the k-th pending weight parameter is The j-th pending weight parameter of the i-th network layer in the neural network model is described; correspondingly, the correction error value is obtained according to the following formula:

Among them, p and q are positive integers, and i and j are non-negative integers.

The foregoing feasible implementation manners of the embodiments of the present application exemplarily give different calculation methods and forms of correction values to correct the error values of the training results, thereby achieving the right of reducing quantization errors and avoiding large values. The value parameter dominates the reasoning results of the neural network and leads to the effect of overfitting problems.

It should be understood that in some neural network models, some network layers do not have undetermined weight parameters. Obviously, network layers without undetermined weight parameters cannot participate in the calculation of the correction error value in the above formula.

In a fourth feasible implementation manner of the first aspect, processing the sample data based on the neural network model's pending weight parameters includes: obtaining the pending weight parameters; quantizing the obtained pending weight parameters, and A quantized weight parameter is obtained, where the quantized weight parameter is a quantized value of the pending weight parameter; the quantized weight parameter is used as a model weight parameter of the neural network model, and a forward propagation algorithm is used to process the sample data Obtaining the output result from an output layer of the neural network model.

It should be understood that there may be multiple forward propagation algorithms mentioned in the embodiments of the present application, which are not limited in the embodiments of the present invention. Through the forward propagation algorithm, the input sample data is deduced based on the neural network model to obtain the output result.

In a fifth feasible implementation manner of the first aspect, the model weight parameters of the neural network model are obtained in an iterative training manner, and when the iterative training meets an end condition, the based on the modified error value and the Said to-be-determined weight parameters and determining model weight parameters of the neural network model includes: using the quantized weight parameters as model weight parameters of the neural network model.

The embodiment of the present application realizes compression of the neural network model by quantizing the weight parameters, reduces the occupation of memory bandwidth and energy consumption, and improves the computing efficiency of the processor.

In a sixth feasible implementation manner of the first aspect, when the iterative training does not satisfy the end condition, the model of the neural network model is determined based on the modified error value and the pending weight parameter. The weight parameter includes: adjusting the pending weight parameter layer by layer to the network layer of the neural network model by using a back-propagation algorithm according to the modified error value until the input layer of the neural network model to obtain the The adjusted weight parameters of the neural network model are described.

In a seventh feasible implementation manner of the first aspect, the pending weight parameters of the neural network model are adjusted according to the following formula:

Wherein, w0 _k represents the k-th pending weight parameter, w1 _k represents the k-th adjusted weight parameter, and β is a normal number.

It should be understood that there may be multiple types of back propagation algorithms, also referred to as back propagation algorithms, mentioned in the embodiments of the present application, which are not limited in the embodiments of the present invention. Through the back propagation algorithm, the weight parameters are trained, and the updated weight parameters make the neural network model further optimized.

In an eighth feasible implementation manner of the first aspect, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition includes the following conditions: A combination of one or more of: the original error value in the Nth training cycle is less than a preset first threshold; the modified error value in the Nth training cycle is less than a preset second threshold; The difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; the modified error value in the Nth training cycle and the first The difference between the correction error values in the NM training cycles is less than a preset fourth threshold; the difference between the pending weight parameters in the Nth training cycle and the pending weight parameters in the NM training cycle is less than a preset A fifth threshold; and N is greater than a preset sixth threshold.

In a ninth feasible implementation manner of the first aspect, when the Nth training cycle does not satisfy the end condition, one or more combinations of the following physical quantities are stored: the Nth training cycle The original error value in; the modified error value in the N-th training cycle; the pending weight parameter in the N-th training cycle; and the number of cycles N of the N-th training cycle.

In the embodiment of the present application, by properly setting the training end conditions, the training efficiency is improved, and the training effect and the balance of the resources consumed by training are achieved.

In a tenth feasible implementation manner of the first aspect, the obtaining the pending weight parameter includes: using a preset initial weight parameter as the pending weight during a first training cycle of the iterative training. Parameters; in a non-first training cycle of the iterative training, using the adjusted weight parameters of the neural network model as the pending weight parameters.

In an eleventh feasible implementation manner of the first aspect, the neural network model is used for image recognition; correspondingly, the sample data includes image samples; and correspondingly, the output result includes all characteristics characterized as a probability form. The recognition result of image recognition is described.

In a twelfth feasible implementation manner of the first aspect, the neural network model is used for voice recognition; correspondingly, the sample data includes sound samples; and correspondingly, the output result includes all features characterized as a probability form. The recognition result of voice recognition will be described.

In a thirteenth feasible implementation manner of the first aspect, the neural network model is used for obtaining a super-resolution image; correspondingly, the sample data includes image samples; correspondingly, the output result includes super-resolution The pixel value of the processed image.

The above feasible implementation manners of the examples of the present application exemplarily provide specific application scenarios of the neural network model in the examples of the present application. Through the application of the neural network model, the recognition rate of image recognition and sound recognition can be improved, and the image can be improved Super-resolution processed image quality can also achieve significant beneficial effects in other application areas.

In a second aspect, an embodiment of the present application provides a device for determining a weight parameter of a neural network model, which is characterized by including: a forward propagation module for processing sample data based on a pending weight parameter of the neural network model to obtain Output result; a comparison module for calculating an original error value of the output result and a preset expected result, the original error value being a numerical representation of a difference between the output result and the expected result; a correction module, which uses Based on the correction value, the original error value is modified to obtain a correction error value; a determination module is configured to determine a model weight parameter of the neural network model based on the correction error value and the pending weight parameter; , The correction value is obtained according to the following formula:

R = (w _k -Q (w _k )) × Q (w _k )

R represents the correction value, w _k represents the k-th pending weight parameter of the neural network model, Q (w _k ) represents a quantized value of the k-th pending weight parameter, and k is a non-negative integer.

In a first feasible implementation manner of the second aspect, the correction error value is obtained according to the following formula:

Among them, E1 represents the correction error value, E0 represents the original error value, α is a constant, m is the total number of pending weight parameters used to process the sample data, and F ((w _k -Q (w _k ) ) × Q (w _k )) represents a function in which the correction value is an independent variable, and m is a positive integer.

In a second feasible implementation manner of the second aspect, the absolute value of the correction value is calculated by using the function of the correction value as an independent variable; correspondingly, the correction error value is obtained according to the following formula:

Among them, | (w _k -Q (w _k )) × Q (w _k ) | means calculating the absolute value of (w _k -Q (w _k )) × Q (w _k ).

In a third feasible implementation manner of the second aspect, the neural network model includes p network layers, each of the network layers includes q the pending weight parameters, and the k-th pending weight parameter is The j-th pending weight parameter of the i-th network layer in the neural network model is described; correspondingly, the correction error value is obtained according to the following formula:

Among them, p and q are positive integers, and i and j are non-negative integers.

In a fourth feasible implementation manner of the second aspect, the forward propagation module is specifically configured to: obtain the pending weight parameter; quantize the obtained pending weight parameter to obtain a quantized weight parameter, and the quantized weight The parameter is a quantized value of the pending weight parameter; the quantized weight parameter is used as a model weight parameter of the neural network model, and the sample data is processed using a forward propagation algorithm; the output from the neural network model The layer obtains the output result.

In a fifth feasible implementation manner of the second aspect, the model weight parameters of the neural network model are obtained by an iterative training method, and when the iterative training meets an end condition, the determining module is specifically configured to: The quantized weight parameter is used as a model weight parameter of the neural network model.

In a sixth feasible implementation manner of the second aspect, further including a back propagation module, when the iterative training does not satisfy the end condition, the back propagation module is specifically configured to: according to the correction error value Using a back-propagation algorithm to adjust the pending weight parameters layer by layer for the network layer of the neural network model until the input layer of the neural network model to obtain the adjusted weight parameters of the neural network model.

In a seventh feasible implementation manner of the second aspect, the pending weight parameters of the neural network model are adjusted according to the following formula:

Wherein, w0 _k represents the k-th pending weight parameter, w1 _k represents the k-th adjusted weight parameter, and β is a normal number.

In an eighth feasible implementation manner of the second aspect, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition includes the following conditions: A combination of one or more of: the original error value in the Nth training cycle is less than a preset first threshold; the modified error value in the Nth training cycle is less than a preset second threshold; The difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; the modified error value in the Nth training cycle and the first The difference between the correction error values in the NM training cycles is less than a preset fourth threshold; the difference between the pending weight parameters in the Nth training cycle and the pending weight parameters in the NM training cycle is less than a preset A fifth threshold; and N is greater than a preset sixth threshold.

In a ninth feasible implementation manner of the second aspect, when the Nth training cycle does not satisfy the end condition, one or more combinations of the following physical quantities are stored: the Nth training cycle The original error value in; the modified error value in the N-th training cycle; the pending weight parameter in the N-th training cycle; and the number of cycles N of the N-th training cycle.

In a tenth feasible implementation manner of the second aspect, the forward propagation module is specifically configured to: during a first training cycle of the iterative training, use a preset initial weight parameter as the pending weight parameter In a non-first training cycle of the iterative training, using the adjusted weight parameter of the neural network model as the pending weight parameter.

In an eleventh feasible implementation manner of the second aspect, the neural network model is used for image recognition; correspondingly, the sample data includes image samples; and correspondingly, the output result includes all characteristics characterized in a probabilistic form. The recognition result of image recognition is described.

In a twelfth feasible implementation manner of the second aspect, the neural network model is used for voice recognition; correspondingly, the sample data includes sound samples; and correspondingly, the output result includes all features characterized as a probability form. The recognition result of voice recognition will be described.

In a thirteenth feasible implementation manner of the second aspect, the neural network model is used for obtaining a super-resolution image; correspondingly, the sample data includes image samples; correspondingly, the output result includes super-resolution The pixel value of the processed image.

In a third aspect, an embodiment of the present application provides an electronic device including: one or more processors and one or more memories. The one or more memories are coupled with one or more processors, and the one or more memories are used to store computer program code. The computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs On the one hand, a method for determining weight parameters of a neural network model.

In a fourth aspect, an embodiment of the present application provides a computer storage medium including computer instructions, and when the computer instructions are run on an electronic device, the electronic device is caused to execute the data processing method according to any one of the first aspects.

In a fifth aspect, an embodiment of the present application provides a computer program product that, when the computer program product runs on a computer, causes the computer to execute the method for determining a weight parameter of a neural network model according to any one of the first aspect.

According to a sixth aspect, an embodiment of the present application provides a chip including a processor and a memory. The memory is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes any of the first A method for determining weight parameters of a neural network model.

For the beneficial effects of the second aspect to the sixth aspect, reference may be made to the description in the first aspect, and details are not described herein again.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary schematic diagram of a neural network structure;

2 is a schematic structural diagram of an exemplary neuron;

3 is an exemplary schematic diagram of another neural network structure;

4 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

5 is an exemplary flowchart of a method for determining weight parameters of a neural network model according to an embodiment of the present application;

6 is an exemplary structural block diagram of a device for determining a weight parameter of a neural network model according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of another electronic device according to an embodiment of the present application.

detailed description

Neural networks (also known as deep neural networks) can be used to process various data, such as image data, audio data, and so on. A neural network may include one or more network layers (also called neural network layers). The network layer may be a convolutional layer, a fully connected layer, a deconvolutional layer, or a recurrent layer. A typical neural network model is shown in Figure 1.

In order to facilitate understanding of the embodiments of the present invention, some concepts related to the embodiments of the present application are given for reference.

(1) Neural network model and forward propagation: For the training sample set (x ( ⁱ ), y ( ⁱ )), the neural network algorithm can provide a complex and nonlinear hypothetical model h _{W, b} ( x), which has parameters W, b, which can be used to fit the data. In order to describe the neural network, let's start with the simplest neural network model. The neural network model is also referred to as the neural network in this article. This neural network consists of only one "neuron". Figure 2 is the "neuron" Icon.

其中函数

被称为“激活函数”。 This "neuron" is an arithmetic unit that takes x ₁ , x ₂ , x ₃ and intercept +1 as input values, and its output is

Where function

Called the "activation function".

The so-called neural network is to connect many single "neurons" together, so that the output of one "neuron" can be the input of another "neuron". Figure 3 is a simple neural network.

The circle is used to represent the input of the neural network. The circle marked "+1" is called the bias node, which is the intercept term. The leftmost layer of the neural network is called the input layer, and the rightmost layer is called the output layer (in this example, the output layer has only one node). A layer composed of all nodes in the middle is called a hidden layer (in other embodiments, the hidden layer may not exist, or there may be multiple layers). At the same time, it can be seen that in the above neural network example, there are 3 input units (excluding the bias unit), 3 hidden units and an output unit.

本例神经网络有参数(W，b)＝(W ⁽¹⁾，b ⁽¹⁾，W ⁽²⁾，b ⁽²⁾)，其中

是第l层第j单元与第l+1层第i单元之间的联接参数(其实就是连接线上的权重)，

是第l+1层第i单元的偏置项。没有其他单元连向偏置单元(即偏置单元没有输入)，因为它们总是输出+1。同时，用s _l表示第l层的节点数(偏置单元不计在内)。用

表示第l层第i单元的激活值(输出值)。当l＝1时，

也就是第i个输入值(输入值的第i个特征)。对于给定参数集合W，b，神经网络就可以按照函数h _W，b(x)来计算输出结果。本例神经网络的计算步骤如下： It may be useful to use n _l to represent the number of layers in the network. In this example, n _l = 3, we label the first layer as L _l , so L ₁ is the input layer and the output layer is

The neural network in this example has parameters (W, b) = (W ⁽¹⁾ , b ⁽¹⁾ , W ⁽²⁾ , b ⁽²⁾ ), where

Is the connection parameter (in fact, the weight on the connection line) between the jth unit in the lth layer and the ith unit in the l + 1th layer,

Is the bias term for the i + 1th cell in the l + 1th layer. No other unit is connected to the bias unit (ie the bias unit has no input) because they always output +1. At the same time, by S _l l represents the number of nodes of layer (not counting the bias unit). use

Represents the activation value (output value) of the i-th unit of the l-th layer. When l = 1,

That is, the i-th input value (i-th feature of the input value). For a given parameter set W, b, the neural network can calculate the output according to the function h _{W, b} (x). The calculation steps of the neural network in this example are as follows:

表示第l层第i单元输入加权和(包括偏置单元)，比如，

则

May wish to use

Represents the weighted sum (including the bias unit) of the l-th and i-th units

then

The above calculation step is called forward propagation.

(2) Backpropagation algorithm (also known as Backpropagation) and error backpropagation algorithm: Backpropagation algorithm is mainly repeated and iterated by two links (incentive propagation and weight update) until the network The input response reaches the predetermined target range.

The learning process of the error back-propagation algorithm consists of a forward-propagation process and a back-propagation process. In the forward propagation process, input information passes through the input layer through the hidden layer, is processed layer by layer, and is transmitted to the output layer. If the desired output value cannot be obtained at the output layer, take various representation forms (such as the sum of squares) of the output and the expected error as the objective function, transfer to back propagation, and find the objective function for each neuron layer by layer The partial derivatives of the weights constitute the ladder of the objective function to the weight vector. As a basis for modifying the weights, the learning of the network is completed in the process of the weights modification. When the error reaches the expected value, the network learning ends.

In the incentive propagation step, the propagation link in each iteration includes two steps: sending the training input to the network to obtain the incentive response (forward propagation phase); and differentiating the incentive response with the target output corresponding to the training input to obtain the hidden Layer and output layer response error (back propagation phase).

In the weight update step, the weight on each neuron is updated according to the following steps: multiply the input excitation and response errors to obtain the gradient of the weights; multiply this gradient by a ratio and invert and add to the weights on. This ratio will affect the speed and effectiveness of the training process, so it is called "training factor". The direction of the gradient indicates the direction of the error expansion, so when updating the weights, it needs to be inverted to reduce the errors caused by the weights.

Matt Mazur's "Step by Step Back Propagation Example" ("Step Breakdown of a Backward Propagation Example" is available from https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation- Example / Download, the full text cited in this article) exemplarily introduces an implementation of an error back propagation algorithm, which can be applied to the embodiments of this application, and will not be described in detail.

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Wherein, in the description of the embodiments of the present application, unless otherwise stated, "/" represents or means, for example, A / B may represent A or B; "and / or" herein is only a description of an associated object The association relationship indicates that there can be three kinds of relationships, for example, A and / or B can be expressed as: there are three cases where A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” means two or more than two.

The data processing device involved in this embodiment of the present application is an electronic device that processes data such as images and voice by using a convolutional neural network, and may be, for example, a server or a terminal. Exemplarily, when the electronic device is a terminal, the electronic device may specifically be a desktop computer, a portable computer, a personal digital assistant (PDA), a tablet computer, an embedded device, a mobile phone, or a smart peripheral (such as a smart watch, a handheld Ring, glasses, etc.), TV set-top boxes, surveillance cameras, etc. The embodiment of the present application does not limit the specific type of the electronic device.

For example, FIG. 4 shows a schematic diagram of a hardware structure of an electronic device 400 according to an embodiment of the present application. The electronic device 400 may include at least one processor 401, a communication bus 402, and a memory 403. The electronic device 400 may further include at least one communication interface 404.

The processor 401 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), and a graphics processor.

(graphics processing unit, GPU), field programmable gate array (FPGA), or one or more integrated circuits for controlling the execution of the program procedures of the present application.

The communication bus 402 may include a path for transmitting information between the aforementioned components.

The communication interface 404 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. .

The memory 403 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type that can store information and instructions Dynamic storage device, can also be electrically erasable programmable read-only memory (electrically erasable programmable read-only memory (EEPROM)), read-only compact disc (compact disc-read-only memory (CD-ROM) or other optical disc storage, optical disc storage (Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be used by a computer Any other media accessed, but not limited to this. The memory may exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

The memory 403 is configured to store application program code that executes the solution provided by the embodiment of the present application, and a neural network model structure, weights, and intermediate results of the processor 401 operation using the solution provided by the embodiment of the present application. 401 to control execution. The processor 401 is configured to execute application program code stored in the memory 203, so as to implement a data processing method provided in the following embodiments of the present application.

In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 4.

In a specific implementation, as an embodiment, the electronic device 400 may include multiple processors, such as the processor 401 and the processor 407 in FIG. 4. Each of these processors can be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

In specific implementation, as an embodiment, the electronic device 400 may further include an output device 405 and an input device 406. The output device 405 communicates with the processor 401 and can display information in a variety of ways. For example, the output device 405 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. Wait. The input device 406 is in communication with the processor 401 and can accept user input in a variety of ways. For example, the input device 406 may be a mouse, a keyboard, a camera, a microphone, a touch screen device, or a sensing device.

As mentioned above, neural networks are composed of network layers, and each network layer processes its input data and passes it to the next network layer. In the network layer, according to different attributes of the network layer (such as a convolution layer, a fully connected layer, etc.), different input weights are used to perform convolution, multiplication and addition operations on the input data. The methods and methods of these operations are determined by the properties of the network layer, but the values of the weights used by different operations are obtained through training. Different data processing results can be obtained by adjusting the weight values.

In the training process of model weight parameters, too much focus on the data fitting of the training data set may lead to poor generalization. Its performance is: the model fits the training data set very well, and the accuracy is very high. But on the data set outside the training data set, it can not fit the data well, the effect is not good, and the accuracy is severely reduced.

The neural network weight parameter precision is redundant. Low-precision data formats (such as INT8, binary, etc.) are used to record the weight parameters. Instead of high-precision data formats (such as FP32, FP64), weight parameter compression can be achieved.

Quantifying weight parameters is a feasible implementation method for weight parameter compression. Compressing the accuracy of neural network parameters can obtain a balance between storage, energy consumption, and accuracy. Data that can be quantified include: weights, Feature tensors (activations), gradients and other parameters of the neural network model are not limited.

In a feasible implementation manner, the weight coefficients of the high-precision (non-quantized, generally FP32 or FP64-accurate) neural network are first obtained through training, and then the weight coefficients expressed in the high-precision data format are used. Low-precision data format for expression. At this time, because the low-precision data format cannot express the details of high-precision data, it will cause a difference in the value. This difference is accumulated multiple times in the operation, and eventually the accuracy of the final calculation result of the quantized model is compared with that of the quantized model The original model has declined. In order to solve the above problems, the quantized model parameters are generally trained again under the premise of ensuring a low-precision data format, and the weight values are adjusted through retraining to finally achieve a model accuracy close to high accuracy at low accuracy.

Specifically, the sample data is input to the neural network during the training process, and the difference between the expected data and the expected data is calculated, and then the difference is used to calculate the gradient of the weight of the neural network to be adjusted (that is, the trend of the weights to be adjusted), and adjust The weight in the neural network achieves the purpose of reducing the difference, that is, achieving higher accuracy. However, in the quantized neural network, because the gradient value of the weight adjustment during training is extremely small, the probability is smaller than the minimum interval value that can be expressed under the weight accuracy, which makes the gradient unable to actually adjust the weight value. For example, in the data format of INT4, the value range is defined as 0 to 1, and the minimum value of the expressible interval is 2 ^-4 . However, the gradient is likely to be much smaller than this value, assuming ^2-6 . Any value A under INT4, after summing with the gradient, the result should be A + 2 ^-6 , but in the data format of INT4, because its minimum interval cannot express 2 ^-6 , the actual result is still A. Therefore, you cannot directly use quantized weights for training.

In order to solve the above problem, a common training method is to use a high-precision reference weight to carry unquantized weight information. The quantized weight is obtained from the reference weight. The error calculation is performed by the quantized weight. The gradient of the error obtained is applied to the unquantized Weight (also called reference weight). After the reference weights are adjusted beyond a certain amount, they can be expressed in quantized weights. (For example, the weight after quantization can only express an integer from 0 to 255. The reference weight is initially 1. The value has been slightly floating during the adjustment process. Until the reference weight is greater than 1.5, the quantization weight becomes 2 until the reference weight is less than 0.5 Only when the quantization weight becomes 0, otherwise the quantization weight is 1).

The current technical problems brought by the above methods include: Unlike neural networks without quantization, quantized neural network calculation uses quantized weights instead of reference weights. Differences between reference weights and quantized weight values accumulate in neural network calculations, and ultimately affect The result of the calculation is a quantization error. The gradient used to adjust the reference weight is derived from the above-mentioned quantized weight to calculate the derivation result and the inference error obtained, which is not accurate enough and there is a gradient error. The above problems will make it difficult for the model training to reach the global optimal solution.

An embodiment of the present application provides a method for determining a weight parameter of a neural network model, as shown in FIG. 5, which specifically includes:

S501. Process the sample data based on the pending weight parameters of the neural network model to obtain an output result.

Specifically, in a feasible implementation manner, this step includes:

S5011. Obtain the pending weight parameters.

It may be assumed that, in the embodiment of the present application, the model weight parameters of the neural network model are obtained in an iterative training manner.

When entering training for the first time, that is, during the first training cycle of the iterative training, the execution of step S5011 includes obtaining a default initial value, such as assigning constant parameters such as 0, 1 to a pending weight parameter. The determined value is given to the weight parameters to be determined, such as the stored weight parameters of the pre-trained neural network model.

When in the iterative training process, that is, during the non-first training cycle of the iterative training, the execution of step S5011 includes obtaining the pending weight parameters (that is, the adjusted weight parameters) updated by the back propagation algorithm in the previous training cycle. ) As the pending weight parameter obtained in this step. The specific implementation manner will be detailed in the subsequent steps, and this step will not be repeated here.

S5012: Quantify the obtained pending weight parameters to obtain a quantized weight parameter, where the quantized weight parameter is a quantized value of the pending weight parameter.

To quantify the weight parameter to be determined, different preset quantization schemes can be adopted, for example, the high-precision expression mode of the weight parameter is transformed into a low-precision expression mode introduced in the foregoing, which is not limited in this step.

S5013. Use the quantized weight parameter as a model weight parameter of the neural network model, and use a forward propagation algorithm to process the sample data.

The forward propagation algorithm was introduced in the foregoing. In this step, the quantified pending weight parameters are used as the model parameters of the neural network model, taking the sample data as input, and the forward propagation algorithm as the calculation criterion for calculation. It should be understood that this step does not limit the specific forward propagation algorithm.

S5014. Obtain the output result from an output layer of the neural network model.

A calculation result for the sample data is output from an output layer of the neural network model. Exemplarily, if the neural network model is used for image recognition, the sample data includes image samples, and the output result includes the recognition result of the image recognition, which is characterized in a probabilistic form. Specifically, the sample image may be determined as a target. The probability of the image is 90%. If the neural network model is used for voice recognition, the sample data includes voice samples, and the output result includes the recognition result of the voice recognition characterized as a probabilistic form. Specifically, it may be determined that the sample voice is the target voice. The probability is 20%. . If the neural network model is used to obtain a super-resolution image, the sample data includes image samples, and the output result includes pixel values of the image after super-resolution processing.

It should be understood that the neural network model can also be used in other applications related to the field of artificial intelligence. Correspondingly, the sample data as input data and the output result as output data can also be other types of physical quantities, without limitation.

S502. Calculate an original error value of the output result and a preset expected result, where the original error value is a numerical representation of a difference between the output result and the expected result.

Corresponding to step S5014, the difference between the expected output result, that is, the expected result, and the actual output result is calculated in this step, and the difference is characterized in a numerical form. Exemplarily, the difference may be the difference between the recognition results, for example, the recognition result is 90%, the expected result is 100%, the original error value is 10%, or the original image corresponding to the sample image before the downsampling and the The pixel differences between the sample images after super-resolution processing can be expressed, for example, by the signal peak signal-to-noise ratio (PSNR) between the two, such as -0.2 decibels (dB), or the square between the two image pixels. Differences and the like are determined according to the specific application of the neural network model and are not limited.

S503. Correct the original error value based on the correction value to obtain a correction error value.

In the step, a correction value is first obtained. In a feasible implementation manner, the correction value is obtained according to the following formula:

R = (w _k -Q (w _k )) × Q (w _k )

R represents the correction value, w _k represents the k-th pending weight parameter of the neural network model, Q (w _k ) represents a quantized value of the k-th pending weight parameter, and k is a non-negative integer.

Correspondingly, the correction error value is obtained according to the following formula:

Among them, E1 represents the correction error value, E0 represents the original error value, α is a constant, m is the total number of pending weight parameters used to process the sample data, and F ((w _k -Q (w _k ) ) × Q (w _k )) represents a function in which the correction value is an independent variable, and m is a positive integer.

In a feasible implementation manner, the function of using the correction value as an independent variable is to calculate an absolute value of the correction value;

Correspondingly, the correction error value is obtained according to the following formula:

Among them, | (w _k -Q (w _k )) × Q (w _k ) | means calculating the absolute value of (w _k -Q (w _k )) × Q (w _k ).

In another feasible implementation manner, the neural network model includes p network layers, and each of the network layers includes q of the pending weight parameters, and the k-th pending weight parameter is the neural network model. The j-th pending weight parameter of the i-th network layer in the network;

Correspondingly, the correction error value is obtained according to the following formula:

Among them, p and q are positive integers, and i and j are non-negative integers.

It should be understood that in some feasible implementations, a certain network layer of the neural network model may not contain the undetermined weight parameters, that is, the corresponding q of the network layer is 0, and obviously the network layer will not be used to correct the error value. Calculation.

In the embodiment of the present application, the difference between the weight parameter and the quantized weight parameter is used as a penalty term by using the correction value (regularization function) to guide the non-quantized weight parameter close to its quantized weight value during the training process to reduce Quantization error. At the same time, the above-mentioned difference value as a penalty term is multiplied with the quantized weight parameter to avoid overfitting problems caused by the weighted values of some large values leading the reasoning results of the neural network.

Next, a model weight parameter of the neural network model is determined based on the modified error value and the pending weight parameter.

S504. Determine whether the iterative training satisfies an end condition.

It may be set to perform step S504 in the Nth training cycle in the iterative training, where N is an integer greater than 1, M is a positive integer less than N, and the ending condition includes one or more combinations of the following conditions :

The original error value in the Nth training cycle is less than a preset first threshold;

The correction error value in the Nth training cycle is less than a preset second threshold;

A difference between the original error value in the N-th training cycle and the original error value in the N-M training cycle is less than a preset third threshold;

A difference between the correction error value in the N-th training cycle and the correction error value in the N-M training cycle is less than a preset fourth threshold;

A difference between the pending weight parameter in the N-th training cycle and the pending weight parameter in the N-M training cycle is less than a preset fifth threshold; and

N is greater than a preset sixth threshold.

It should be understood that when M is 1, it means that in the relevant end condition, the difference values of the corresponding physical quantities in the two nearest neighboring training periods need to be compared.

It should also be understood that step S504 may be performed in each training cycle, and may also be performed in every M training cycles. The embodiment of this application does not limit the execution frequency of step S504.

For a training cycle, it may be understood as a process of calculating a correction error value, adjusting a weight parameter according to the correction error value, and then using the adjusted weight parameter to obtain a new training result.

Corresponding to step S504, when the Nth training cycle does not satisfy the end condition, one or more combinations of the following physical quantities are stored:

The original error value in the Nth training cycle;

A correction error value in the Nth training cycle;

The pending weight parameters in the Nth training cycle; and

The number N of the Nth training cycle.

The stored physical quantity will be called in the subsequent execution of step S504.

S505. When the iterative training meets an end condition, use the quantized weight parameter as a model weight parameter of the neural network model.

Generally, the end of iterative training means that through training, the quantized weight parameters have been optimized to the desired degree, that is, they can be determined as the model weight parameters of the neural network model.

In some feasible implementation modes, the training sample set A is used to train the model, and the test sample set B is used to test the model. After using A to train the model for N training cycles, test the model on test data set B to get the first test result X; continue to use A to train the model for M training cycles on test data set B Test the model to get the second test result Y. When the difference between X and Y is less than the threshold, end the training, otherwise continue to use A to train the model.

Correspondingly, in this embodiment, the ending condition includes that the difference between the first test result X and the second test result Y is smaller than a threshold.

S506. When the iterative training does not satisfy the end condition, according to the modified error value, a back propagation algorithm is used to adjust the pending weight parameters layer by layer for the network layer of the neural network model until the An input layer of a neural network model to obtain adjusted weight parameters of the neural network model.

The back-propagation algorithm has been introduced in the previous article, so I won't repeat it here. Exemplarily, the pending weight parameters of the neural network model are adjusted according to the following formula:

Wherein, w0 _k represents the k-th pending weight parameter, w1 _k represents the k-th adjusted weight parameter, and β is a normal number.

After the weights from the input layer are adjusted, all the weights of the neural network model have been obtained, that is, the adjusted weight parameters of the neural network model are obtained.

After that, the adjusted weight parameter is used as the parameter to be quantified, and step S5011 is executed to continue the iterative training.

The embodiments of the present application provide a method and a device for determining a weight parameter of a neural network model. In various data processing scenarios applied to the neural network model, such as image recognition, speech recognition, image super-resolution processing, etc., the model can be trained by The error between the output result and the expected result introduces a proper correction value, which reduces the quantization error and avoids the problem of over-fitting caused by some weight parameters with larger numerical values leading the reasoning results of the neural network.

It can be understood that, in order to execute the above method, the electronic device includes a hardware structure and / or a software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a certain function is performed by hardware or computer software-driven hardware depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

The embodiments of the present application may divide the functional modules of the electronic device according to the foregoing method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of the modules in the embodiments of the present application is schematic, and is only a logical function division. In actual implementation, there may be another division manner.

In a case where each functional module is divided corresponding to each function, FIG. 6 shows a possible composition diagram of the electronic device involved in the foregoing embodiment.

A device 600 for determining a weight parameter of a neural network model includes:

The forward propagation module 601 is configured to process the sample data based on the neural network model's pending weight parameters to obtain an output result;

A comparison module 602, configured to calculate an original error value of the output result and a preset expected result, where the original error value is a numerical representation of a difference between the output result and the expected result;

A correction module 603, configured to correct the original error value based on the correction value to obtain a correction error value;

A determining module 604, configured to determine a model weight parameter of the neural network model based on the modified error value and the pending weight parameter;

The correction value is obtained according to the following formula:

R = (w _k -Q (w _k )) × Q (w _k )

R represents the correction value, w _k represents the k-th pending weight parameter of the neural network model, Q (w _k ) represents a quantized value of the k-th pending weight parameter, and k is a non-negative integer.

In a feasible implementation manner, the correction error value is obtained according to the following formula:

Among them, E1 represents the correction error value, E0 represents the original error value, α is a constant, m is the total number of pending weight parameters used to process the sample data, and F ((w _k -Q (w _k ) ) × Q (w _k )) represents a function in which the correction value is an independent variable, and m is a positive integer.

In a feasible implementation manner, the function that uses the correction value as an independent variable is an absolute value for calculating the correction value; correspondingly, the correction error value is obtained according to the following formula:

Among them, | (w _k -Q (w _k )) × Q (w _k ) | means calculating the absolute value of (w _k -Q (w _k )) × Q (w _k ).

In a feasible implementation manner, the neural network model includes p network layers, and each network layer includes q of the pending weight parameters, and the k-th pending weight parameter is in the neural network model. The j-th pending weight parameter of the i-th network layer; correspondingly, the correction error value is obtained according to the following formula:

Among them, p and q are positive integers, and i and j are non-negative integers.

In a feasible implementation manner, the forward propagation module 601 is specifically configured to: obtain the pending weight parameter; quantize the obtained pending weight parameter to obtain a quantized weight parameter, where the quantized weight parameter is the A quantized value of a weight parameter to be determined; using the quantized weight parameter as a model weight parameter of the neural network model, using a forward propagation algorithm to process the sample data; obtaining the output layer from the neural network model Output results.

In a feasible implementation manner, the model weight parameters of the neural network model are obtained by iterative training. When the iterative training meets an end condition, the determination module 604 is specifically configured to: use the quantized weight parameters As a model weight parameter of the neural network model.

In a feasible implementation manner, it further includes a back propagation module 605. When the iterative training does not satisfy the end condition, the back propagation module 605 is specifically configured to: The forward propagation algorithm adjusts the pending weight parameters layer by layer for the network layer of the neural network model until the input layer of the neural network model to obtain the adjusted weight parameters of the neural network model.

In a feasible implementation manner, the pending weight parameters of the neural network model are adjusted according to the following formula:

Wherein, w0 _k represents the k-th pending weight parameter, w1 _k represents the k-th adjusted weight parameter, and β is a normal number.

In a feasible implementation manner, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the ending condition includes one or more of the following conditions Combinations: the original error value in the Nth training cycle is less than a preset first threshold; the modified error value in the Nth training cycle is less than a preset second threshold; the Nth training The difference between the original error value in the period and the original error value in the NMth training period is less than a preset third threshold; the corrected error value in the Nth training period and the NMth training period The difference between the corrected error value of is smaller than a preset fourth threshold; the difference between the pending weight parameter in the Nth training cycle and the pending weight parameter in the NMth training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold.

In a feasible implementation manner, when the Nth training cycle does not satisfy the end condition, one or more combinations of the following physical quantities are stored: the original error value in the Nth training cycle The correction error value in the Nth training cycle; the pending weight parameter in the Nth training cycle; and the number of cycles N of the Nth training cycle.

In a feasible implementation manner, the forward propagation module 601 is specifically configured to: during a first training cycle of the iterative training, use a preset initial weight parameter as the pending weight parameter; in the In the non-first training cycle of iterative training, the adjusted weight parameter of the neural network model is used as the pending weight parameter.

In a feasible implementation manner, the neural network model is used for image recognition; correspondingly, the sample data includes image samples; correspondingly, the output result includes a recognition result of the image recognition characterized as a probability form. .

In a feasible implementation manner, the neural network model is used for voice recognition; correspondingly, the sample data includes voice samples; correspondingly, the output result includes the recognition result of the voice recognition characterized as a probability form. .

In a feasible implementation manner, the neural network model is used for obtaining a super-resolution image; correspondingly, the sample data includes image samples; correspondingly, the output result includes a super-resolution processed image. Pixel values.

It should be noted that all relevant content of each step involved in the foregoing method embodiments can be referred to the functional description of the corresponding functional module, and will not be repeated here.

Of course, the electronic device includes but is not limited to the above-listed unit modules. For example, the electronic device may further include a communication unit. The communication unit may include a sending unit for sending data or signals to other devices, and receiving data or signals sent by other devices. Receiving unit, etc. In addition, the functions that can be implemented by the above functional units also include, but are not limited to, the functions corresponding to the method steps of the above examples. For detailed descriptions of other units of the electronic device, refer to the detailed description of the corresponding method steps. No longer.

The processing unit 701 in FIG. 7 may be a processor or a controller. For example, the processing unit 701 may be a central processing unit CPU, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit ASIC, or a field programmable gate. Array FPGA, graphics processor GPU, or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of this application. A processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on. The storage unit 702 may be a memory. The communication unit may be a transceiver, a radio frequency circuit, or a communication interface. The processing unit 701 executes a method for determining a weight parameter of a neural network model as shown in FIG. 5.

An embodiment of the present application further includes a computer storage medium including computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to execute a method for determining a weight parameter of a neural network model shown in FIG. 5.

The embodiment of the present application further includes a computer program product, when the computer program product is run on a computer, the computer is caused to execute a method for determining a weight parameter of a neural network model shown in FIG. 5.

Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of the description, only the division of the above functional modules is used as an example. In practical applications, the above functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or It can be integrated into another device, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or may be distributed to multiple different places. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present application essentially or partly contribute to the existing technology or all or part of the technical solutions may be embodied in the form of a software product, which is stored in a storage medium The instructions include a number of instructions for causing a device (which can be a single-chip microcomputer, a chip, or the like) or a processor to execute all or part of the steps of the methods in the embodiments of the present application. The foregoing storage media include: U disks, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, which can store program codes.

The above is only a specific implementation of this application, but the scope of protection of this application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in this application. It should be covered by the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims (31)

A method for determining weight parameters of a neural network model, which includes: Process the sample data based on the undetermined weight parameters of the neural network model to obtain output results; Calculating an original error value of the output result and a preset expected result, where the original error value is a numerical representation of a difference between the output result and the expected result; Modifying the original error value based on the correction value to obtain a correction error value; Determining a model weight parameter of the neural network model based on the modified error value and the pending weight parameter; The correction value is obtained according to the following formula: R = (w _k -Q (w _k )) × Q (w _k ) R represents the correction value, w _k represents the k-th pending weight parameter of the neural network model, Q (w _k ) represents a quantized value of the k-th pending weight parameter, and k is a non-negative integer. The method according to claim 1, wherein the correction error value is obtained according to the following formula:

Among them, E1 represents the correction error value, E0 represents the original error value, α is a constant, m is the total number of pending weight parameters used to process the sample data, and F ((w _k -Q (w _k ) ) × Q (w _k )) represents a function in which the correction value is an independent variable, and m is a positive integer. The method according to claim 2, wherein the function of the correction value as an independent variable is used to calculate an absolute value of the correction value; Correspondingly, the correction error value is obtained according to the following formula:

Among them, | (w _k -Q (w _k )) × Q (w _k ) | means calculating the absolute value of (w _k -Q (w _k )) × Q (w _k ). The method according to any one of claims 1 to 3, wherein the neural network model includes p network layers, and each of the network layers includes q of the pending weight parameters, and the kth pending The weight parameter is the j-th pending weight parameter of the i-th network layer in the neural network model; Correspondingly, the correction error value is obtained according to the following formula:

Among them, p and q are positive integers, and i and j are non-negative integers. The method according to any one of claims 1 to 4, wherein the processing of sample data based on the neural network model's pending weight parameters comprises: Obtaining the pending weight parameters; Quantizing the obtained pending weight parameters to obtain a quantized weight parameter, where the quantized weight parameter is a quantized value of the pending weight parameter; Using the quantized weight parameter as a model weight parameter of the neural network model, and using a forward propagation algorithm to process the sample data; The output result is obtained from an output layer of the neural network model. The method according to claim 5, wherein the model weight parameters of the neural network model are obtained in an iterative training manner, and when the iterative training meets an end condition, the based on the modified error value and the The weight parameter to be determined, and determining a model weight parameter of the neural network model includes: The quantized weight parameter is used as a model weight parameter of the neural network model. The method according to claim 6, characterized in that, when the iterative training does not satisfy the end condition, the determining a model weight of the neural network model based on the modified error value and the pending weight parameter Parameters, including: According to the modified error value, a back-propagation algorithm is used to adjust the pending weight parameter layer by layer for the network layer of the neural network model until the input layer of the neural network model to obtain the Adjusted weight parameters. The method according to claim 7, wherein the pending weight parameters of the neural network model are adjusted according to the following formula:

Wherein, w0 _k represents the k-th pending weight parameter, w1 _k represents the k-th adjusted weight parameter, and β is a normal number. The method according to any one of claims 6 to 8, wherein, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition Include a combination of one or more of the following conditions: The original error value in the Nth training cycle is less than a preset first threshold; The correction error value in the Nth training cycle is less than a preset second threshold; A difference between the original error value in the N-th training cycle and the original error value in the N-M training cycle is less than a preset third threshold; A difference between the correction error value in the N-th training cycle and the correction error value in the N-M training cycle is less than a preset fourth threshold; A difference between the pending weight parameter in the N-th training cycle and the pending weight parameter in the N-M training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold. The method according to claim 9, characterized in that when the Nth training cycle does not satisfy the end condition, one or more combinations of the following physical quantities are stored: The original error value in the Nth training cycle; A correction error value in the Nth training cycle; The pending weight parameters in the Nth training cycle; and The number N of the Nth training cycle. The method according to any one of claims 6 to 10, wherein the obtaining the pending weight parameter comprises: During the first training cycle of the iterative training, using a preset initial weight parameter as the pending weight parameter; In a non-first training cycle of the iterative training, the adjusted weight parameter of the neural network model is used as the pending weight parameter. The method according to any one of claims 1 to 11, wherein the neural network model is used for image recognition; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes a recognition result of the image recognition characterized as a probability form. The method according to any one of claims 1 to 11, wherein the neural network model is used for voice recognition; Correspondingly, the sample data includes sound samples; Correspondingly, the output result includes a recognition result of the voice recognition characterized as a probability form. The method according to any one of claims 1 to 11, wherein the neural network model is used for acquiring a super-resolution image; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes pixel values of the image after super-resolution processing. A device for determining a weight parameter of a neural network model, including: The forward propagation module is used to process the sample data based on the neural network model's pending weight parameters to obtain the output result; A comparison module, configured to calculate an original error value of the output result and a preset expected result, where the original error value is a numerical representation of a difference between the output result and the expected result; A correction module, configured to correct the original error value based on the correction value to obtain a correction error value; A determining module, configured to determine a model weight parameter of the neural network model based on the correction error value and the pending weight parameter; The correction value is obtained according to the following formula: R = (w _k -Q (w _k )) × Q (w _k ) R represents the correction value, w _k represents the k-th pending weight parameter of the neural network model, Q (w _k ) represents a quantized value of the k-th pending weight parameter, and k is a non-negative integer. The device according to claim 15, wherein the correction error value is obtained according to the following formula:

Among them, E1 represents the correction error value, E0 represents the original error value, α is a constant, m is the total number of pending weight parameters used to process the sample data, and F ((w _k -Q (w _k ) ) × Q (w _k )) represents a function in which the correction value is an independent variable, and m is a positive integer. The device according to claim 16, wherein the function of using the correction value as an independent variable is an absolute value for calculating the correction value; Correspondingly, the correction error value is obtained according to the following formula:

Among them, | (w _k -Q (w _k )) × Q (w _k ) | means calculating the absolute value of (w _k -Q (w _k )) × Q (w _k ). The device according to any one of claims 15 to 17, wherein the neural network model includes p network layers, each of the network layers includes q the pending weight parameters, and the kth pending The weight parameter is the j-th pending weight parameter of the i-th network layer in the neural network model; Correspondingly, the correction error value is obtained according to the following formula:

Among them, p and q are positive integers, and i and j are non-negative integers. The device according to any one of claims 15 to 18, wherein the forward propagation module is specifically configured to: Obtaining the pending weight parameters; Quantizing the obtained pending weight parameters to obtain a quantized weight parameter, where the quantized weight parameter is a quantized value of the pending weight parameter; Using the quantized weight parameter as a model weight parameter of the neural network model, and using a forward propagation algorithm to process the sample data; The output result is obtained from an output layer of the neural network model. The device according to claim 19, wherein the model weight parameters of the neural network model are obtained in an iterative training manner, and when the iterative training meets an end condition, the determination module is specifically configured to: The quantized weight parameter is used as a model weight parameter of the neural network model. The device according to claim 20, further comprising a back propagation module, and when the iterative training does not satisfy the end condition, the back propagation module is specifically configured to: According to the modified error value, a back-propagation algorithm is used to adjust the pending weight parameter layer by layer for the network layer of the neural network model until the input layer of the neural network model to obtain the Adjusted weight parameters. The device according to claim 21, wherein the pending weight parameters of the neural network model are adjusted according to the following formula:

Wherein, w0 _k represents the k-th pending weight parameter, w1 _k represents the k-th adjusted weight parameter, and β is a normal number. The device according to any one of claims 20 to 22, wherein for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition Include a combination of one or more of the following conditions: The original error value in the Nth training cycle is less than a preset first threshold; The correction error value in the Nth training cycle is less than a preset second threshold; A difference between the original error value in the N-th training cycle and the original error value in the N-M training cycle is less than a preset third threshold; A difference between the correction error value in the N-th training cycle and the correction error value in the N-M training cycle is less than a preset fourth threshold; A difference between the pending weight parameter in the N-th training cycle and the pending weight parameter in the N-M training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold. The device according to claim 23, wherein when the Nth training cycle does not satisfy the end condition, one or more combinations of the following physical quantities are stored: The original error value in the Nth training cycle; A correction error value in the Nth training cycle; The pending weight parameters in the Nth training cycle; and The number N of the Nth training cycle. The device according to any one of claims 20 to 24, wherein the forward propagation module is specifically configured to: During the first training cycle of the iterative training, using a preset initial weight parameter as the pending weight parameter; In a non-first training cycle of the iterative training, the adjusted weight parameter of the neural network model is used as the pending weight parameter. The device according to any one of claims 15 to 25, wherein the neural network model is used for image recognition; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes a recognition result of the image recognition characterized as a probability form. The device according to any one of claims 15 to 25, wherein the neural network model is used for voice recognition; Correspondingly, the sample data includes sound samples; Correspondingly, the output result includes a recognition result of the voice recognition characterized as a probability form. The device according to any one of claims 15 to 25, wherein the neural network model is used for acquiring a super-resolution image; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes pixel values of the image after super-resolution processing. An electronic device, comprising: one or more processors and one or more memories; The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, where the computer program code includes computer instructions, and when the one or more processors When the computer instruction is executed, the electronic device executes a method for determining a weight parameter of a neural network model according to any one of claims 1 to 14. A computer storage medium, comprising computer instructions that, when the computer instructions run on an electronic device, cause the electronic device to execute the weight parameter of the neural network model according to any one of claims 1 to 14. Determine the method. A computer program product, wherein when the computer program product is run on a computer, the computer is caused to execute the method for determining a weight parameter of a neural network model according to any one of claims 1 to 14.

Images