WO2021120036A1 - Data processing apparatus and data processing method - Google Patents

Abstract

Provided by the present application are a data processing apparatus and a data processing method in the field of artificial intelligence. Provided by the present application is a data processing apparatus, used to perform convolution processing, according to a two-dimensional convolution kernel, on a matrix to be convolved. The two-dimensional convolution kernel comprises a first parameter and M second parameters; the matrix to be convolved comprises a first eigenvalue and M second eigenvalues, the M second parameters corresponding one-to-one to the M second eigenvalues, and the first parameter corresponding to the first eigenvalue. The data processing apparatus comprises: M multipliers and M-1 first adders, which are used to perform a multiply-accumulate operation on the M second parameters and the M second eigenvalues to obtain a multiply-accumulate result; and a second adder, which is used to perform an addition operation on the multiply-accumulate result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved. The data processing apparatus and data processing method provided in the present application help prevent the wastage of resources and improve resource utilization.

Description

Data processing device and data processing method Technical field

This application relates to data calculation in the field of artificial intelligence, and more specifically, to a data processing device and a data processing method.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theories.

Neural network (NN), as an important branch of artificial intelligence, is a network structure that imitates the behavioral characteristics of animal neural networks for information processing. The structure of the neural network is composed of a large number of nodes (or neurons) connected to each other, and the purpose of processing information is achieved by learning and training the input information based on a specific operation model. A neural network includes an input layer, a hidden layer, and an output layer. The input layer is responsible for receiving input signals, the output layer is responsible for outputting the calculation results of the neural network, and the hidden layer is responsible for calculation processes such as learning and training. It is the memory unit of the network and the memory of the hidden layer. The function is represented by a weight matrix, usually each neuron corresponds to a weight coefficient.

Among them, convolutional neural network (convolutional neural network, CNN) is a multi-layer neural network, each layer is composed of multiple two-dimensional planes, and each plane is composed of multiple independent neurons, and each feature plane can be composed of It is composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.

In the convolutional neural network, the convolution operation performed by the processor usually converts the convolution of the feature information and the weight in the feature plane into a matrix multiplication operation between the signal matrix and the weight matrix. In the specific matrix multiplication operation, the signal matrix and the weight matrix are divided into blocks to obtain multiple fractal (fractional) signal matrices and fractal weight matrices, and then the multiple fractal signal matrices and fractal weight matrices are subjected to matrix multiplication and accumulation operations.

At present, when the processor convolves the input layer, each processing engine (PE) used for convolution operation in the processor usually includes a ²ⁿ multiply and accumulate (MAC) unit. Specifically, each PE used by the processor to perform convolution calculation includes 2 ⁿ multiplication units and 2 ⁿ -1 addition units. However, the size of the convolution kernel used in the convolution kernel operation is generally less than 2 ⁿ , which makes some multiplication units and addition units idle when the PE performs the convolution operation, which causes a waste of processor resources.

For example, as shown in Figure 15, when the PE performing the convolution operation includes 16 MACs and the size of the convolution kernel is 3*3, in the PE, at least the multiplication unit 9 to the multiplication unit 15, and the addition unit 8, the addition unit 9. The addition unit 10 and the addition unit 12 will run idly, resulting in a waste of processing resources.

Summary of the invention

The present application provides a data processing device and a data processing method, which help to avoid the waste of resources and improve the utilization rate of resources.

In a first aspect, the present application provides a data processing device configured to perform convolution processing on a convolution matrix to be convolved according to a two-dimensional convolution kernel. The two-dimensional convolution kernel includes a first parameter and M The second parameter, the matrix to be convolved includes a first eigenvalue and M second eigenvalues, the M second parameters are in one-to-one correspondence with the M second eigenvalues, and the first parameter corresponds to The first characteristic value. The data processing device includes: M multipliers and M-1 first adders, configured to perform multiplication and accumulation operations on the M second parameters and the M second characteristic values to obtain a multiplication and accumulation result ; A second adder for performing an addition operation on the multiplication and accumulation result and the first eigenvalue to obtain the convolution result of the two-dimensional convolution kernel and the matrix to be convolved. Among them, M is a positive integer greater than 1.

When the data processing device performs convolution processing on a matrix to be convolved with a size of M+1 according to a convolution kernel with a size of M+1, it uses M multipliers and M-1 adders to calculate the convolution kernel The convolution result of the M parameters and the M eigenvalues in the matrix to be convolved, and another adder calculates the sum of the convolution result and the M+1th eigenvalue in the matrix to be convolved, and finally the sum Take it as the convolution result of the convolution kernel on the matrix to be convolved. This makes it unnecessary to use multiple traditional data processing including M+1 multipliers and M adders when the size of the convolution kernel required for convolution calculation is one more than the number of multiplication and accumulation units in the data processing device. The device calculates, thereby avoiding the waste of resources and improving the utilization rate of resources.

The matrix to be convolved may include the features of the image input to the neural network, or may include the voice features included in the voice data input to the neural network, or may include the text features included in the text data input to the neural network, and so on.

In some possible implementation manners, the first parameter is equal to 1. That is, the first parameter corresponding to the first characteristic value input to the second adder is 1. In this way, the accuracy of the convolution result calculated by the data processing device can be guaranteed.

In some possible implementation manners, the device further includes a processor configured to determine the two-dimensional convolution kernel according to an initial convolution kernel, and the initial convolution kernel includes a first initial parameter and M Second initial parameters, the M second parameters correspond to the M second initial parameters one-to-one, the first parameters correspond to the first initial parameters, and the second parameters are equal to all The quotient of the second initial parameter corresponding to the second parameter and the first initial parameter, and the first initial parameter is not zero.

When the data processing device in this implementation is used to calculate the convolution result of the initial convolution kernel and the matrix to be convolved, the processor can first divide all the parameters in the initial convolution kernel by one of the non-zero parameters (ie The first initial parameter), so that one parameter (that is, the first parameter) in the obtained two-dimensional convolution kernel is 1. In this way, when the above-mentioned M multipliers and M adders are used to calculate the convolution result of the two-dimensional convolution kernel and the matrix to be convolved, more accurate values can be obtained.

Optionally, the processor may be further configured to: determine the convolution result of the initial convolution kernel and the matrix to be convolved according to the convolution result of the two-bit convolution kernel and the matrix to be convolved, Wherein, the convolution result of the initial convolution kernel and the matrix to be convolved is equal to the product of the convolution result of the two-dimensional convolution kernel and the matrix to be convolved and the first initial parameter.

In some possible implementation manners, when the quotient of at least one of the second initial parameters and the first initial parameter is greater than a first threshold, before the determining the two-dimensional convolution kernel according to the initial convolution kernel, The processor is further configured to: reduce the M second parameters or the M second initial parameters by m times, wherein the quotient of any second initial parameter reduced by m times and the first initial parameter Not greater than the first threshold. Among them, m can be a positive integer.

In the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is greater than the first threshold, all the second initial parameters may be reduced by m times to So that the quotient of any second initial parameter after reduction and the first initial parameter is not greater than the first threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not greater than The first threshold. Wherein, the first threshold may be less than or equal to the maximum expressible value of the processor, or in other words, the first threshold may be less than or equal to the maximum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the maximum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

Or, in the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is greater than the first threshold, all the second parameters may be reduced by m times to So that the reduced second parameter is not greater than the first threshold. Wherein, the first threshold may be less than or equal to the maximum expressible value of the processor, or in other words, the first threshold may be less than or equal to the maximum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the maximum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

In the case that the processor is further configured to reduce the M second parameters or the M second initial parameters by m times, before performing the addition operation on the multiplication and accumulation result and the first characteristic value, the processing The device is also used to: reduce the first characteristic value by m times; correspondingly, the second adder is specifically used to add the result of the multiplication and accumulation and the reduced first characteristic value. This is because when the second initial parameter or the second parameter is reduced by m times, the first parameter should be reduced by m times accordingly, so the processing result of the first feature value by the first parameter should also be reduced by m times. The data processing device of the present application directly reduces the first feature value by m times to ensure the accuracy of the convolution result.

Optionally, the processor may perform a shift operation to reduce the first characteristic value. Specifically, the first feature value can be shifted to the left.

In some possible implementation manners, when the quotient of at least one of the second initial parameters and the first initial parameter is less than a second threshold, before the determining the two-dimensional convolution kernel according to the initial convolution kernel, The processor is further configured to: expand the M second parameters or the M second initial parameters by n times, wherein the quotient of any second initial parameter expanded by n times and the first initial parameter Not less than the second threshold. Among them, n can be a positive integer.

In the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is less than the second threshold, all the second initial parameters may be expanded by m times to The quotient of any second initial parameter after expansion and the first initial parameter is not less than the second threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not less than The second one threshold. Wherein, the second threshold may be greater than or equal to the minimum expressible value of the processor, or in other words, the second threshold may be greater than or equal to the minimum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the minimum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

Or, the data processing device in this implementation manner, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is less than the second threshold, all the second parameters may be reduced by m times to So that the reduced second parameter is not less than the second threshold. Wherein, the second threshold may be greater than or equal to the minimum expressible value of the processor, or in other words, the second threshold may be greater than or equal to the minimum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the minimum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

In some possible implementation manners, before the addition operation is performed on the multiplication and accumulation result and the first eigenvalue, the processor is further configured to: expand the first eigenvalue by n times; correspondingly The second adder is specifically configured to perform an addition operation on the multiplication and accumulation result and the expanded first characteristic value.

This is because when the second initial parameter or the second parameter is enlarged by m times, the first parameter should be enlarged by m times accordingly, so the processing result of the first feature value by the first parameter should also be enlarged by m times. The data processing device of the present application directly enlarges the first feature value by m times to ensure the accuracy of the convolution result.

Optionally, the processor may perform an enlargement process on the first feature value through a shift operation. Specifically, the first feature value can be shifted to the right.

In some possible implementation manners, the processor includes one or a combination of more of the following: a central processing unit, a graphics processor, or a neural network processor.

In some possible implementation manners, M is equal to 8, and the two-dimensional convolution kernel is a 3*3 matrix. Optionally, the M multipliers and the M-1 first adders may constitute a multiplication accumulation adder.

In some possible implementation manners, M is equal to 24, and the two-dimensional convolution kernel is a 5*5 matrix. Optionally, the M multipliers and the M-1 first adders constitute three multiplication and accumulation adders, wherein each of the multiplication accumulation adders includes M/3 multipliers and M/3-1 A first adder.

In some possible implementation manners, the two-dimensional convolution kernel is a two-dimensional matrix component of an N-dimensional convolution kernel, and N is an integer greater than 2.

In a second aspect, the present application provides a data processing method, including: obtaining an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the equivalent convolution kernel of the Lth layer The parameter of is obtained based on the quotient of the corresponding parameter of the initial convolution kernel of the L-th layer and the first initial parameter in the initial convolution kernel of the L-th layer, and the value of the first initial parameter is K, and K is Non-zero number, the equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer; to obtain the initial value of the L+1th layer of the neural network and the Lth layer The initial convolution kernel of the L+1th layer whose convolution kernel has a mapping relationship; expand each parameter in the initial convolution kernel of the L+1th layer by K times; according to the expanded first convolution kernel The initial convolution kernel of the L+1 layer determines the equivalent convolution kernel of the L+1th layer. Among them, K can be a positive integer.

In this method, the parameters in the initial convolution kernel of the Lth layer of the neural network are processed, so that when convolution processing is performed according to the equivalent convolution kernel obtained by the processing, one multiplication processing can be reduced, so that the initial convolution When the number of parameters in the product kernel is one more than the number of multipliers in the device for performing convolution calculation, at least one device is not used, which causes other multipliers and adders in the device to run idle, thereby saving resources and improving resource utilization.

In addition, although the equivalent convolution kernel is reduced by K times during the initial convolution kernel processing, the resulting convolution result obtained by using the equivalent convolution kernel for convolution processing is higher than that obtained by using the initial convolution kernel for convolution processing. The problem that the result of the convolution is K times smaller can be solved by expanding the corresponding parameters in the L+1 layer of the neural network.

Among them, the feature map input to the Lth layer may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, etc. .

In some possible implementation manners, the equivalent convolution kernel of the L-th layer includes M second parameters and one first parameter, and the M second parameters respectively correspond to the M second features of the feature map. Value, the first parameter corresponds to the first feature value of the feature map, the first parameter is 1, and the convolution processing on the feature map of the Lth layer includes: Multiplying and accumulating the two parameters and the M second characteristic values to obtain a multiplying and accumulating result; and performing an adding operation on the multiplying and accumulating result and the first characteristic value.

In other words, the parameters in the equivalent convolution kernel can be obtained by dividing all the parameters in the initial convolution kernel by one of the non-zero first initial parameters. At this time, the equivalent convolution kernel and the first initial parameter The corresponding first parameter is 1. In this way, when using the equivalent convolution kernel to process the feature map, you can only calculate the M second parameters other than the first parameter in the equivalent convolution kernel to multiply and accumulate the corresponding M feature values. When it is out of date, the multiplication and accumulation unit is used, and then after the multiplication and accumulation result of the M second parameters and the M eigenvalues is calculated, the multiplication and accumulation result is added to the first eigenvalue corresponding to the first parameter, thereby Obtain the convolution result of the equivalent convolution kernel and the feature map.

In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial convolution kernel of the initial convolution kernel of the Lth layer. Obtaining the quotient of the parameter includes: when the quotient of at least one parameter of the initial convolution kernel of the L-th layer and the first initial parameter is greater than a first threshold, the corresponding to the initial convolution kernel of the L-th layer The parameter is reduced by m times, wherein the quotient of any parameter of the initial convolution kernel of the L-th layer reduced by m times and the first initial parameter is not greater than the first threshold. Among them, m can be a positive integer.

In the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is greater than the first threshold, all the second initial parameters may be reduced by m times to So that the quotient of any second initial parameter after reduction and the first initial parameter is not greater than the first threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not greater than The first threshold. Wherein, the first threshold may be less than or equal to the maximum expressible value of the processor, or in other words, the first threshold may be less than or equal to the maximum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the maximum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

Optionally, the performing an addition operation on the multiplication and accumulation result and the first feature value includes: reducing the first feature value by m times; and performing the multiplication and accumulation result and the reduced first feature value by a factor of m; The eigenvalues are added.

In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial convolution kernel of the initial convolution kernel of the Lth layer. Obtaining the quotient of the parameter includes: when the quotient of at least one parameter of the initial convolution kernel of the L-th layer and the first initial parameter is less than a second threshold, corresponding to the initial convolution kernel of the L-th layer The parameter is expanded by n times, wherein the quotient of the parameter of the initial convolution kernel of the L-th layer that is expanded by n times and the first initial parameter is not less than the second threshold. Among them, n can be a positive integer.

In the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is less than the second threshold, all the second initial parameters may be expanded by m times to The quotient of any second initial parameter after expansion and the first initial parameter is not less than the second threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not less than The second one threshold. Wherein, the second threshold may be greater than or equal to the minimum expressible value of the processor, or in other words, the second threshold may be greater than or equal to the minimum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the minimum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

Optionally, the performing an addition operation on the multiplication and accumulation result and the first eigenvalue includes: expanding the first eigenvalue by n times; and performing the multiplication and accumulation result and the expanded first The eigenvalues are added.

In a third aspect, the present application provides a data processing device, including: a processing module for obtaining the equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the Lth layer The parameter of the equivalent convolution kernel is obtained based on the quotient of the corresponding parameter of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, and the value of the first initial parameter Is K, and K is a non-zero number. The equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer; the acquisition module is used to obtain the L+1th layer of the neural network. The initial convolution kernel of the L+1th layer that has a mapping relationship with the initial convolution kernel of the Lth layer in the layer; the expansion module is used to change the initial convolution kernel of the L+1th layer Each parameter is expanded by K times; the determining module is configured to determine the equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion processing. Among them, K can be a positive integer.

Among them, the feature map input to the Lth layer may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, etc. .

In some possible implementation manners, the equivalent convolution kernel of the L-th layer includes M second parameters and one first parameter, and the M second parameters respectively correspond to the M second features of the feature map. Value, the first parameter corresponds to the first feature value of the feature map, and the first parameter is 1; wherein, performing convolution processing on the feature map of the Lth layer includes: Performing a multiplication and accumulation operation on the second parameters and the M second characteristic values to obtain a multiplication and accumulation result; and performing an addition operation on the multiplication and accumulation result and the first characteristic value.

In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial convolution kernel of the initial convolution kernel of the Lth layer. Obtaining the quotient of the parameter includes: when the quotient of at least one parameter of the initial convolution kernel of the L-th layer and the first initial parameter is greater than a first threshold, the corresponding to the initial convolution kernel of the L-th layer The parameter is reduced by m times, wherein the quotient of any parameter of the initial convolution kernel of the L-th layer reduced by m times and the first initial parameter is not greater than the first threshold. Among them, m can be a positive integer.

Optionally, the performing an addition operation on the multiplication and accumulation result and the first feature value includes: reducing the first feature value by m times; and performing the multiplication and accumulation result and the reduced first feature value by a factor of m; The eigenvalues are added.

In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial convolution kernel of the initial convolution kernel of the Lth layer. Obtaining the quotient of the parameter includes: when the quotient of at least one parameter of the initial convolution kernel of the L-th layer and the first initial parameter is less than a second threshold, the corresponding to the initial convolution kernel of the L-th layer The parameter is expanded by n times, wherein the quotient of the parameter of the initial convolution kernel of the L-th layer that is expanded by n times and the first initial parameter is not less than the second threshold. Among them, n can be a positive integer.

Optionally, the performing an addition operation on the multiplication and accumulation result and the first eigenvalue includes: expanding the first eigenvalue by n times; and performing the multiplication and accumulation result and the expanded first The eigenvalues are added.

In a fourth aspect, the present application provides a data device, which includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the The processor is used to execute the method in the second aspect.

In a fifth aspect, the present application provides a computer-readable medium that stores instructions for device execution, and the instructions are used to implement the method in the second aspect.

In the sixth aspect, this application provides a computer program product containing instructions, which when the computer program product runs on a computer, causes the computer to execute the method in the second aspect.

In a seventh aspect, the present application provides a chip including a processor and a data interface, and the processor reads instructions stored in a memory through the data interface, and executes the method in the second aspect.

Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory. When the instructions are executed, the The processor is used to execute the method in the second aspect.

In an eighth aspect, the present application provides a computing device. The computing device includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions to implement the method in the second aspect.

In a ninth aspect, this application provides a data processing device, including a programmable device and a memory, where the memory is used to store configuration files required for the operation of the programmable device, and the programmable device is used to read the configuration file from the memory and execute , In order to achieve the second aspect of the method.

Optionally, as an implementation manner, the programmable device includes a field programmable logic gate array (FPGA) or a complex programmable logic device (CPLD).

Description of the drawings

FIG. 1 is a schematic diagram of the structure of a convolutional neural network provided by this application.

Fig. 2 is a schematic structural diagram of another convolutional neural network provided by this application.

FIG. 3 is a schematic diagram of the hardware structure of a chip provided by this application.

FIG. 4 is a schematic structural diagram of a system architecture provided by this application.

FIG. 5 is a schematic flowchart of a data processing method provided by this application.

FIG. 6 is a schematic diagram of a method for obtaining an equivalent convolution kernel provided by this application.

FIG. 7 is a schematic diagram of a method for obtaining equivalent parameters provided by this application.

FIG. 8 is a schematic diagram of another method for obtaining equivalent parameters provided by this application.

FIG. 9 is a schematic diagram of another method for obtaining equivalent parameters provided by this application.

FIG. 10 is a schematic diagram of another method for obtaining an equivalent convolution kernel provided by this application.

FIG. 11 is a schematic diagram of another method for obtaining an equivalent convolution kernel provided by this application.

FIG. 12 is a schematic structural diagram of a data processing device provided by this application.

FIG. 13 is a schematic structural diagram of another data processing device provided by this application.

FIG. 14 is a schematic structural diagram of another data processing device provided by this application.

FIG. 15 is a schematic structural diagram of another data processing device provided by this application.

FIG. 16 is a schematic diagram of a method for reading data provided by this application.

FIG. 17 is a schematic diagram of a method for reading data provided by this application.

FIG. 18 is a schematic structural diagram of another data processing device provided by this application.

FIG. 19 is a schematic structural diagram of another data processing device provided by this application.

FIG. 20 is a schematic structural diagram of a data processing device provided by this application.

Detailed ways

The following describes the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The embodiments of the present application relate to related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the following first introduces related terms and other related concepts of neural networks that may be involved in the embodiments of the present application.

Convolutional neural network is a deep neural network with convolutional structure. The convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer. The feature extractor can be seen as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or convolution feature map. The convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can be connected to only part of the neighboring neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Sharing weight can be understood as the way of extracting image information has nothing to do with location. The underlying principle is that the statistical information of a certain part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learning image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally, the more the number of convolution kernels, the richer the image information reflected by the convolution operation.

The convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.

The following describes the structure of CNN in detail in conjunction with Figure 1. As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. The deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of. As a deep learning architecture, CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.

The structure of the convolutional neural network in the embodiment of the present application may be as shown in FIG. 1. In FIG. 1, a convolutional neural network (CNN) 300 may include an input layer 310, a convolutional layer/pooling layer 320 (the pooling layer is optional), and a neural network layer 330.

Take image processing as an example (the operation is similar when the input data is text or voice), where the input layer 310 can obtain the image to be processed, and pass the obtained image to be processed to the convolutional layer/pooling layer 320 and the following The neural network layer 330 performs processing to obtain the processing result of the image.

The following describes the internal layer structure in CNN 300 in Figure 1 in detail.

Convolutional layer/pooling layer 320:

Convolutional layer:

The convolutional layer/pooling layer 320 as shown in Figure 1 may include layers 321-326 as shown in the example. For example, in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, and layer 323 is a convolutional layer. Layers, 324 is a pooling layer, 325 is a convolutional layer, and 326 is a pooling layer; in another implementation, 321 and 322 are convolutional layers, 323 is a pooling layer, and 324 and 325 are convolutional layers. Layer, 326 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

In the following, taking the convolutional layer 321 as an example, and taking the input data as an image as an example, the internal working principle of a convolutional layer will be introduced. When the input data is voice or text or other types of data, the internal working principle of the convolutional layer is similar.

The convolution layer 321 can include many convolution operators. The convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same. During the convolution operation, the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row×column) are applied. That is, multiple homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image. Perform obfuscation and so on. The multiple weight matrices have the same size (row×column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are merged to form The output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions. .

When the convolutional neural network 300 has multiple convolutional layers, the initial convolutional layer (such as 321) often extracts more general features, which can also be called low-level features; with the convolutional neural network With the deepening of the network 300, the features extracted by the subsequent convolutional layers (for example, 326) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the 321-326 layers as illustrated by 320 in Figure 1, it can be a convolutional layer followed by a layer. The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. For example, in the image processing process, the sole purpose of the pooling layer is to reduce the size of the image space. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain an image with a smaller size. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of the average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural network layer 330:

After processing by the convolutional layer/pooling layer 320, the convolutional neural network 300 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network 300 needs to use the neural network layer 330 to generate one or a group of required classes of output. Therefore, the neural network layer 330 can include multiple hidden layers (331, 332 to 33n as shown in FIG. 3) and an output layer 340. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.

After the multiple hidden layers in the neural network layer 330, that is, the final layer of the entire convolutional neural network 300 is the output layer 340. The output layer 340 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 300 (as shown in Figure 1, the propagation from 310 to 340 directions is forward propagation) is completed, the back propagation (as shown in Figure 1 is the propagation from 340 to 310 directions is reverse propagation). Start to update the aforementioned weight values and deviations of each layer to reduce the loss of the convolutional neural network 300 and the error between the output result of the convolutional neural network 300 through the output layer and the ideal result.

The structure of the neural network in the embodiment of the present application may be as shown in FIG. 2. In FIG. 2, a convolutional neural network (CNN) 400 may include an input layer 410, a convolutional layer/pooling layer 420 (the pooling layer is optional), and a neural network layer 430. Compared with Figure 1, the multiple convolutional/pooling layers (421 to 426) in the convolutional/pooling layer 420 in Figure 2 are in parallel, and the extracted features are all input to the full neural network layer 430 for processing. deal with. The neural network layer 430 may include multiple hidden layers, that is, hidden layer 1 to hidden layer n, which may be denoted as 431 to 43n.

It should be noted that the convolutional neural network shown in FIG. 1 and FIG. 2 is only used as an example of two possible convolutional neural networks in the embodiment of the present application. In specific applications, the convolutional neural network in the embodiment of the present application Convolutional neural networks can also exist in the form of other network models.

The algorithms or operators of each layer in the convolutional neural network as shown in FIG. 1 and FIG. 2 can all be implemented in the chip as shown in FIG. 3.

FIG. 3 is a hardware structure of a chip provided by an embodiment of the application, and the chip includes a neural network processor 50. The chip can be set in the client device 240 as shown in FIG. 4 to implement corresponding services. The chip can also be set in the training device 220 shown in FIG. 4 to complete the training work of the training device 220 and output the target model 201.

The neural network processor NPU 50 is mounted as a coprocessor to a main central processing unit (central processing unit, CPU) (host CPU), and the main CPU distributes tasks. The core part of the NPU is the arithmetic circuit 503. The controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.

In some implementations, the arithmetic circuit 503 includes multiple processing units (process engines, PE).

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the matrix A data and matrix B from the input memory 501 to perform matrix operations, and the partial result or final result of the matrix C obtained is stored in an accumulator 508.

The vector calculation unit 507 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on. For example, the vector calculation unit 507 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .

In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a combined value, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 503, for example for use in a subsequent layer in a neural network.

The unified memory 506 is used to store input data and output data.

The weight data directly transfers the input data in the external memory to the input memory 501 and/or the unified memory 506 through the storage unit access controller 505 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 502, And the data in the unified memory 506 is stored in the external memory.

The bus interface unit (BIU) 510 is used to implement interaction between the main CPU, the DMAC, and the instruction fetch memory 509 through the bus.

An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504;

The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

Generally, the unified memory 506, the input memory 501, the weight memory 502, and the instruction fetch memory 509 are all on-chip (On-Chip) memories. The external memory is a memory external to the NPU. The external memory can be a double data rate synchronous dynamic random access memory. Memory (double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.

Among them, the operations of each layer in the convolutional neural network shown in FIG. 1 and FIG. 2 can be executed by the arithmetic circuit 503 or the vector calculation unit 507.

As shown in FIG. 4, an embodiment of the present application provides a system architecture 200. In FIG. 4, a data collection device 260 is used to collect training data. Taking the target model 201 used for image processing as an example, the training data may include training images and classification results corresponding to the training images, where the results of the training images may be manually pre-labeled results. The target model 201 may also be referred to as a target rule 201.

After the training data is collected, the data collection device 260 stores the training data in the database 230, and the training device 220 trains to obtain the target model 201 based on the training data maintained in the database 230.

The following describes the target model 201 obtained by the training device 220 based on the training data. The training device 220 processes the input original image and compares the output image with the original image until the difference between the image output by the training device 120 and the original image is If it is less than a certain threshold, the training of the target model 201 is completed.

The target model 201 in the embodiment of the present application may specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the collection of the data collection device 260, and may also be received from other devices. In addition, it should be noted that the training device 220 does not necessarily perform training of the target model 201 completely based on the training data maintained by the database 230. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to the embodiments of this application. The limit.

The target model 201 obtained by training according to the training device 220 can be applied to different systems or devices, such as the client device 240 shown in FIG. 2. The client device 240 may be a terminal, such as a mobile phone terminal, a tablet computer, or a notebook computer. , Augmented reality (AR)/virtual reality (VR), in-vehicle terminal, etc., it can also be a server or cloud.

The training device 220 can generate a corresponding target model 201 based on different training data for different goals or different tasks. The corresponding target model 201 can be used to achieve the above goals or complete the above tasks, so as to provide users with what they need. the result of.

The target model 201 is obtained by training according to the training device 220, which may be a CNN, a deep convolutional neural network (deep convolutional neural networks, DCNN), and so on.

It is worth noting that FIG. 4 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship among the devices, devices, modules, etc. shown in the figure, the type of training data, and the type or function of the neural network are different. Constitute any restriction. For example, in FIG. 4, the client device 240 and the training device 220 may be the same device.

After the neural network trained by the training device 220, if the target task is performed directly based on the trained neural network, such as image segmentation, image recognition, etc., and each PE in the arithmetic circuit in the neural network processor can only calculate the size The convolution result of the convolution kernel of M and the feature map of size M, and the number of parameters in at least one convolution kernel in the neural network is M+1, at least two PEs are required to calculate the size of M+ The convolution result of the convolution kernel of 1 and the feature map of size M+1. However, this will cause more multipliers and adders in the PE to spin idle, thereby wasting resources.

For example, the neural network includes a depth separable convolutional layer, the size of the convolution kernel of the depth separable convolution layer is 3*3*1, and when a PE can only calculate the multiplication and accumulation of 8, it needs two PE to calculate the convolution result of the convolution kernel. This will cause 6 multipliers and four adders in a PE to be idle, thus wasting resources.

Aiming at the problem of resource waste, this application proposes a new data processing method and data processing device.

Fig. 5 is a schematic flowchart of a data processing method according to an embodiment of the present application. The method includes at least S510 to S540. This method can be executed by the main CPU in FIG. 3.

S510: Obtain an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, where the parameters of the equivalent convolution kernel of the Lth layer are based on the initial convolution kernel of the Lth layer The quotient of the corresponding parameter of and the first initial parameter in the initial convolution kernel of the L-th layer is obtained, the value of the first initial parameter is K, and K is a non-zero number, and the equivalent volume of the L-th layer The product kernel is used to perform convolution processing on the feature map of the L-th layer.

The neural network in this embodiment may be any neural network including convolution processing. For example, the neural network in the embodiment of the present application may be the convolutional neural network shown in FIG. 1 or FIG. 2.

The Lth layer in this embodiment may be any layer that includes convolution processing, for example, it may be a convolution layer, and further, it may be a deeply separable convolution layer.

The initial convolution kernel in this embodiment may be a convolution kernel obtained by training a neural network, or may be a convolution kernel obtained by training and optimizing the neural network.

In this embodiment, the parameters in the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer. It is understandable As: call a non-zero parameter in the initial convolution kernel of the L-th layer as the first initial parameter, divide all the parameters in the initial convolution kernel by the first initial parameter, and obtain each parameter and the first initial parameter According to the quotient of each parameter and the first initial parameter, an equivalent convolution kernel used to perform convolution processing on the feature map of the Lth layer is determined. That is to say, after the equivalent convolution kernel is obtained, when the L-th layer in the neural network is subsequently used to process the feature map, the initial convolution kernel is no longer used, but the equivalent convolution kernel is used.

Among them, the feature map input to the Lth layer may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, etc. .

S520: Obtain the initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with the initial convolution kernel of the Lth layer.

That is, in the first layer after the Lth layer, the initial parameters that have a mapping relationship with the parameters in the aforementioned initial convolution kernel in the Lth layer are obtained.

The initial parameters of the L+1th layer may be the parameters of the L+1th layer obtained by training the neural network, or may be the parameters of the L+1th layer obtained by training and optimizing the neural network.

The initial parameters in the L+1 layer that have a mapping relationship with the parameters in the aforementioned initial convolution kernel in the Lth layer can be understood as: if the aforementioned initial convolution kernel is used in the Lth layer of the neural network to process the feature map and output the volume After the eigenvalues obtained by the convolution, after the eigenvalues obtained by the convolution are input to the L+1th layer, the parameters originally used to process these eigenvalues in the L+1th layer are those in the L+1th layer and the Lth layer. The parameters in the initial convolution kernel of the layer have the initial parameters of the mapping relationship.

The L+1th layer can be a deeply separable convolutional layer, a usual convolutional layer, or a fully connected layer. Among them, usually the convolutional layer can also be referred to as a conventional convolutional layer or a standard convolutional layer. Different types of the L+1th layer have different mapping relationships between their initial parameters and the initial parameters of the convolution kernel of the Lth layer.

Examples of the mapping relationship between the initial parameters in the L+1th layer of the neural network and the initial parameters in the initial convolution kernel of the Lth layer will be introduced in the following content with reference to Figures 7, 8 and 9 .

S530: Expand each parameter in the initial convolution kernel of the L+1th layer by K times.

Because in S510, the equivalent of the Lth layer, the parameters in the convolution kernel are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameters in the initial convolution kernel of the Lth layer. Therefore, the parameters in the equivalent convolution kernel are usually K times smaller than the initial convolution kernel of the Lth layer. In order to ensure the accuracy of the results of the input data processing by the neural network, the initial parameters of the L+1th layer can be enlarged by K times, so that the eigenvalues output by the Lth layer can be compensated.

S540: Determine the equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion processing.

In this embodiment, the equivalent convolution kernel of the L+1th layer is used to process the eigenvalues output by the Lth layer, or in other words, is used to process the eigenvalues input to the L+1th layer. That is to say, when using the neural network for business processing in application scenarios, such as image classification, image segmentation, image recognition and other business processing, the initial convolution kernel of the L+1th layer is no longer used, but the Lth convolution kernel is used instead. +1 layer equivalent convolution kernel.

The data processing method in this embodiment performs correlation processing on the initial convolution kernel of the Lth layer, so that when the processed equivalent convolution kernel is used for convolution processing on the feature map of the input Lth layer, the equivalent The convolution operation between the convolution kernel and the feature map no longer requires all multiplication and accumulation operations, but the convolution operation between the equivalent convolution kernel and the feature map can be decomposed into partial convolution parameters and corresponding feature values The multiply-accumulate operation of, and the non-multiply-accumulate operation between the multiply-accumulate operation result and the feature value without parameter multiply-accumulate operation in the feature map. In this way, when performing convolution processing on the feature map in the L-th layer of the neural network, if the number of parameters in the convolution kernel of the L-th layer is 1 more than the number of times of multiplication and accumulation that the arithmetic circuit for convolution processing can calculate at a time, then There is no longer a problem of waste of resources due to the need to use one more arithmetic circuit that causes most of the multipliers and adders in the arithmetic circuits to be used more idling.

In addition, when the equivalent convolution kernel is obtained from the initial convolution kernel of the L layer, the difference between the equivalent convolution kernel and the initial convolution kernel due to the processing used can be passed through the L+1th layer, etc. The difference between the effective convolution kernel and the initial convolution kernel is compensated back to ensure the accuracy of the neural network's processing of the input data.

In this embodiment, as a possible implementation, obtaining the equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer may include the following steps: adding one of the initial convolution kernels of the Lth layer The parameter is determined as the first initial parameter. When the first initial parameter is not zero, all the parameters in the initial convolution kernel are divided by the first initial parameter, and the convolution kernel formed by the quotient is the equivalent convolution. Product core. Among them, if the first initial parameter is 0, the initial convolution kernel can be directly used as the equivalent convolution kernel.

In this implementation, the value obtained by dividing the first initial parameter in the initial convolution kernel of the L-th layer by the first initial parameter is 1, that is, the value of one parameter in the equivalent convolution kernel is 1, and this value is The parameter of 1 is called the first parameter.

In this embodiment, assuming that the initial convolution kernel of the Lth layer includes a first initial parameter and M second initial parameters, the equivalent convolution kernel of the Lth layer obtained accordingly includes M Two parameters and one first parameter, and the M second parameters may respectively correspond to the M second feature values of the feature map to be processed, and the first parameter may correspond to the first feature of the feature map to be processed value.

In this embodiment, in some implementations, based on the quotient of the corresponding parameter of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the value of the equivalent convolution kernel of the Lth layer is obtained. The following steps may be included: when the quotient of at least one parameter in the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel is greater than the first threshold, the initial convolution of the Lth layer All the parameters in the product kernel are reduced by m times, and then the parameters after the reduction by m times are divided by the first initial parameter. Among them, any parameter in the initial convolution kernel of the Lth layer is reduced by m times and the parameter obtained after the reduction of m times is the same as the first parameter. The quotient of the initial parameter should not be greater than the first threshold.

The first threshold may be the maximum expressible value of the device used when performing convolution processing on the feature map based on the equivalent convolution kernel. For example, when the device used to perform convolution processing on the feature map based on the equivalent convolution kernel is shown in the neural network processor 50 in FIG. 3, the first threshold may be less than or equal to the maximum expressible value of the memory and the arithmetic unit therein. . In this way, data overflow can be avoided and the accuracy of business processing by the neural network can be improved.

In some possible implementations, when the parameters of the L-th equivalent convolution kernel are obtained based on the quotient of the parameters of the initial convolution kernel of the L-th layer and the first initial parameters in the initial convolution kernel of the L-th layer, you can The method includes the following steps: when the quotient of at least one parameter in the initial convolution kernel of the Lth layer and the first initial parameter is less than a second threshold, all parameters of the initial convolution kernel of the Lth layer may be expanded by n times , And then divide the expanded parameter by the first initial parameter to obtain the equivalent convolution kernel, where any parameter of the initial convolution kernel of the Lth layer is expanded by n times to obtain the quotient of the parameter and the first initial parameter Should not be less than the second threshold.

The second threshold may be the smallest expressible value of the device used when performing convolution processing on the feature map based on the equivalent convolution kernel. For example, when the device used to perform convolution processing on the feature map based on the equivalent convolution kernel is shown in the neural network processor 50 in FIG. 3, the second threshold may be greater than or equal to the minimum expressible value of the memory and the arithmetic unit therein. . In this way, numerical overflow can be avoided and the accuracy of business processing by the neural network can be improved.

Taking the L-th layer as the depth separable convolutional layer and the initial convolution kernel of the L-th layer as an example of a 3*3 convolution kernel, we will introduce how to obtain the L-th layer based on the initial convolution kernel of the L-th layer of the neural network. An implementation of the equivalent convolution kernel.

In this implementation, any parameter in the 3*3 convolution kernel in the Lth layer is taken as the first initial parameter (or can be called the common factor); when the first initial parameter is non-zero, the convolution kernel’s All 9 parameters (ie 8 second initial parameters and 1 first initial parameter) are divided by the absolute value of the first initial parameter. When the first initial parameter is 0, all parameters remain unchanged to ensure the equivalent There are 8 normal second parameters and a second parameter with a value of 1, 0 or -1 in the convolution kernel. In this way, the convolution operation of the L-th layer can be changed from 9 MAC operations to the calculation of the addition of 8 MAC results (that is, 8 second parameters and 8 second eigenvalues) and a constant (that is, the first eigenvalue) . Based on this implementation method, an adder can be added to the existing PE capable of processing 8 MACs, and processing can be performed based on a 3*3 convolution kernel at each clock, so that the new PE can process convolution processing. It can work with 100% efficiency and greatly improve the performance of the hardware.

As shown in Figure 6, when the initial convolution kernel includes nine initial parameters of w0, w1, w2, w3, w4, w5, w6, w7, and w8, select w4 as the first initial parameter; when w4 is greater than 0 , Divide these nine initial parameters by w4 to obtain the equivalent convolution kernel. In this case, w4 is called the common factor; when w4 is equal to 0, no additional processing is performed on these nine initial parameters, that is, wait The effective convolution kernel is the same as the initial convolution kernel. In this case, the common factor is 0; when w4 is less than 0, divide these nine initial parameters by -w4 to obtain the equivalent convolution kernel. In this case, -w4 is called the common factor.

Wherein, optionally, when w4 is less than 0, the nine initial parameters can be divided by w4 instead of -w4.

The equivalent convolution kernel obtained according to the initial convolution kernel is used to perform convolution processing on the eigenvalue input of the Lth layer, and the convolution result is used as the input of the L+1th layer.

The following describes that when the initial convolution kernel of the Lth layer is the convolution kernel shown in Figure 6, and the L+1th layer is also a deeply separable convolutional layer, it is obtained according to the initial parameters of the L+1th layer of the neural network The implementation of the equivalent parameters of the L+1 layer.

When the L+1th layer is a depthwise separable convolutional layer, if the common factor of the initial convolution kernel of the Lth layer is 0, the initial convolution kernel of the L+1th layer may not be processed additionally, that is, the L+th layer The initial convolution kernel of layer 1 is the equivalent convolution kernel; otherwise, the initial convolution kernel of layer L+1 can be multiplied by the common factor of the initial convolution kernel of layer L, and the similar operation in Figure 6 can be performed , The initial convolution kernel of the L+1th layer is a convolution kernel corresponding to the initial convolution kernel of the Lth layer in the L+1th layer. An example of the initial convolution kernel and equivalent convolution kernel of the L+1 layer is shown in FIG. 7.

It can be seen from Figure 7 that the second parameter in the equivalent convolution kernel of the L+1th layer is equal to the corresponding second initial parameter in the initial convolution kernel of the L+1th layer multiplied by the first parameter in the Lth layer. The quotient of the product of the initial parameters and the first initial parameter in the initial convolution kernel of the L+1th layer.

The following describes that the initial convolution kernel of the Lth layer is a depth separable convolution kernel, and when the L+1th layer is a normal convolutional layer, obtain the L+1th layer according to the initial parameters of the L+1th layer of the neural network The implementation of equivalent parameters.

As shown in Figure 8, the L-th layer has 16 input channels. The 16 input channels are numbered from 0 to 15. These 16 input channels correspond to the 16 initial convolution kernels one-to-one, and these 16 The common factors of the initial convolution kernel are w0 to w15. When obtaining the equivalent parameters of the L+1 layer according to the initial parameters of the L+1 layer of the neural network, each layer of the input channel of the L+1 layer The convolution kernel multiplies the corresponding common factors from w0 to w15 to obtain the equivalent parameters of the L+1th layer.

The following describes that when the initial convolution kernel of the Lth layer is a depth separable convolution kernel, and the L+1th layer is a fully connected layer, the L+1th layer is obtained according to the initial parameters of the L+1th layer of the neural network. Implementation of effective parameters.

As shown in Figure 9, when L+1 is a fully connected layer, it is first to clarify the position of each feature value in the feature map output by the L+1 layer after the feature map is changed to a 1-dimensional vector, based on the 1-dimensional feature map The vector and the parameters of the fully connected layer have the same length and the characteristics of the corresponding position. Identify each parameter of the L+1th layer and which initial convolution kernel in the Lth layer corresponds to, and multiply each parameter by the corresponding initial convolution kernel The common factor of, where the common factor is not zero.

In the data processing method of the present application, all or part of the steps in Figure 5 can be performed on all the deep separable convolutional layers in the neural network, or only part of the deep separable convolutional layers in the neural network. All or part of the steps.

The following uses the first threshold as the maximum expressible value of the built-in hardware data format of the device that realizes convolution processing, that is, the quotient of any second initial parameter in the initial convolution kernel of the L-th layer and the first initial parameter exceeds the maximum expressible value The value range and the maximum expressible value of 2 to the 16th power are taken as an example to introduce how to process the initial convolution kernel of the L-th layer to obtain an equivalent convolution kernel that does not exceed the expressible value range.

As shown in the left figure in Figure 10, the first initial parameter in the initial convolution kernel of the L-th layer is 0.001, and when one of the second initial parameters is 64000, if 64000 is directly divided by 0.001, the quotient will be greater than The problem of maximum value. At this time, as shown in the middle diagram in Figure 10, all the initial parameters can be divided by 2 to the power of 10 (or multiplied by 2 to the negative power of 10), and divided by 0.001 to get Figure 10 The equivalent convolution kernel shown in the right figure. Among them, the -10th power of 2 is the number that is closest to and smaller than the smallest expressible value among the powers of 2.

The following uses the second threshold as the minimum expressible value of the built-in hardware data format of the device that implements convolution processing, that is, the quotient of any second initial parameter in the initial convolution kernel of the L-th layer and the first initial parameter exceeds the minimum expressible value The value range, and the minimum expressible value of 2 to the power of 16 is taken as an example to introduce how to process the initial convolution kernel of the L-th layer to obtain an equivalent convolution kernel that does not exceed the expressible value range.

As shown in the left figure in Figure 11, the first initial parameter in the initial convolution kernel of the L-th layer is 64000, and when one of the second initial parameters is 0.001, if you directly divide 0.001 by 64000, the quotient will be less than The problem of maximum value. At this time, as shown in the middle diagram in FIG. 11, all initial parameters can be multiplied by 2 to the power of 16 and divided by 64000 to obtain the equivalent convolution kernel as shown in the right diagram in FIG. 11. Among them, the 16th power of 2 is the number closest to and greater than the maximum expressible value among the powers of 2.

The following describes the data device provided by this application. The data processing device provided in the present application can be used to perform convolution processing on a matrix to be convolved according to a two-dimensional convolution kernel. The two-dimensional convolution kernel includes a first parameter and M second parameters. The matrix to be convolved It includes a first characteristic value and M second characteristic values, the M second parameters are in one-to-one correspondence with the M second characteristic values, and the first parameter corresponds to the first characteristic value.

FIG. 12 is a schematic structural diagram of the data processing device of this application. The data processing device includes: M multipliers, M-1 first adders and a second adder, the M multipliers are Mul 0 to Mul M, and the M 1 first adders are ADD 0 To ADD M-2, this second adder is ADD M-1. M is an even number.

The M multipliers and the M-1 first adders are used for multiplying and accumulating the M second parameters and the M second characteristic values to obtain a multiplying and accumulating result. The second adder is used to perform an addition operation on the multiplication and accumulation result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved.

When the data processing device performs convolution processing on a matrix to be convolved with a size of M+1 according to a convolution kernel with a size of M+1, it uses M multipliers and M-1 adders to calculate the convolution kernel The convolution result of the M parameters and the M eigenvalues in the matrix to be convolved, and another adder calculates the sum of the convolution result and the M+1th eigenvalue in the matrix to be convolved, and finally the sum Take it as the convolution result of the convolution kernel on the matrix to be convolved. This makes it unnecessary to use multiple traditional data processing including M+1 multipliers and M adders when the size of the convolution kernel required for convolution calculation is one more than the number of multiplication and accumulation units in the data processing device. The device calculates, thereby avoiding the waste of resources and improving the utilization rate of resources.

For example, the data processing device shown in FIG. 12 may be used to perform convolution processing on the eigenvalue matrix to be convolved input to the Lth layer based on the equivalent convolution kernel of the Lth layer in S510.

The data processing device shown in FIG. 12 may be a component of the arithmetic circuit 503 in the neural network processor 50 in FIG. 3.

In some possible implementation manners, the first parameter is equal to 1. That is, the first parameter corresponding to the first characteristic value input to the second adder is 1. In this way, the accuracy of the convolution result calculated by the data processing device can be guaranteed.

In some possible implementation manners, the device further includes a processor configured to determine the two-dimensional convolution kernel according to an initial convolution kernel, and the initial convolution kernel includes a first initial parameter and M Second initial parameters, the M second parameters correspond to the M second initial parameters one-to-one, the first parameters correspond to the first initial parameters, and the second parameters are equal to all The quotient of the second initial parameter corresponding to the second parameter and the first initial parameter, and the first initial parameter is not zero. In other words, the processor can be used to execute S510.

When the data processing device in this implementation is used to calculate the convolution result of the initial convolution kernel and the matrix to be convolved, the processor can first divide all the parameters in the initial convolution kernel by one of the non-zero parameters (ie The first initial parameter), so that one parameter (that is, the first parameter) in the obtained two-dimensional convolution kernel is 1. In this way, when the above-mentioned M multipliers and M adders are used to calculate the convolution result of the two-dimensional convolution kernel and the matrix to be convolved, more accurate values can be obtained.

Optionally, the processor may be further configured to: determine the convolution result of the initial convolution kernel and the matrix to be convolved according to the convolution result of the two-bit convolution kernel and the matrix to be convolved, Wherein, the convolution result of the initial convolution kernel and the matrix to be convolved is equal to the product of the convolution result of the two-dimensional convolution kernel and the matrix to be convolved and the first initial parameter.

In some possible implementation manners, when the quotient of at least one of the second initial parameters and the first initial parameter is greater than a first threshold, before the determining the two-dimensional convolution kernel according to the initial convolution kernel, The processor is further configured to: reduce the M second parameters or the M second initial parameters by m times, wherein the quotient of any second initial parameter reduced by m times and the first initial parameter Not greater than the first threshold.

In the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is greater than the first threshold, all the second initial parameters may be reduced by m times to So that the quotient of any second initial parameter after reduction and the first initial parameter is not greater than the first threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not greater than The first threshold. Wherein, the first threshold may be less than or equal to the maximum expressible value of the processor, or in other words, the first threshold may be less than or equal to the maximum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the maximum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

Wherein, the operation of reducing the first feature value by m times may be executed by the processor, or may be executed by a shifter. For example, when m is the s power of 2, the first eigenvalue can be reduced by m times by a shifter capable of shifting at least s bits to the left, and s is a non-negative integer. In this scenario, an exemplary structure diagram of the device for performing convolution processing on the feature map based on the equivalent convolution kernel is shown in FIG. 13.

When the convolution processing is performed by the device described in FIG. 13, the M second parameters in the equivalent convolution kernel and the M second eigenvalues in the feature map can be input to M multipliers, the first in the feature map The characteristic value can be input to the shifter for shifting. The M second parameters and the M second eigenvalues are multiplied and accumulated by the M multipliers and the M adders in the multiplication and accumulation process, and the first eigenvalues are shifted to the left by the shifter to reduce m The eigenvalue obtained by the multiplication is input to ADD M-1, and the sum of the output of ADD M-1 is the convolution result of the equivalent convolution kernel and the feature map.

This is because when the second initial parameter or the second parameter is reduced by m times, the first parameter should be reduced by m times accordingly, so the processing result of the first feature value by the first parameter should also be reduced by m times. The data processing device of the present application directly reduces the first feature value by m times to ensure the accuracy of the convolution result.

For example, when the feature map is convolved based on the equivalent convolution kernel shown in the right image of Figure 10, the first feature value in the feature map can be input to the shifter and shifted to the left by 10 bits, and the resulting feature value can be multiplied by The accumulation result is added to obtain the convolution result.

In some possible implementation manners, when the quotient of at least one of the second initial parameters and the first initial parameter is less than a second threshold, before the determining the two-dimensional convolution kernel according to the initial convolution kernel, The processor is further configured to: expand the M second parameters or the M second initial parameters by n times, wherein the quotient of any second initial parameter expanded by n times and the first initial parameter Not less than the second threshold.

In the data processing device in this implementation, when the quotient of any one of the second initial parameters in the initial convolution kernel and the first initial parameter is less than the second threshold, all the second initial parameters may be expanded by m times to The quotient of any second initial parameter after expansion and the first initial parameter is not less than the second threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not less than The second one threshold. Wherein, the second threshold may be greater than or equal to the minimum expressible value of the processor, or in other words, the second threshold may be greater than or equal to the minimum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not overflow the minimum expressible value range of the data processing device and cause the problem of inaccurate calculation results.

Wherein, the operation of expanding the first feature value by n times may be executed by the main CPU attached to the neural network processor that executes the convolution processing, or may be executed by a shifter. For example, when n is 2 to the power of t, the first eigenvalue can be expanded by n times by a shifter capable of shifting at least t bits to the right, and n is a non-negative integer. In this scenario, an exemplary structure diagram of the device for performing convolution processing on the feature map based on the equivalent convolution kernel is shown in FIG. 13.

When the convolution processing is performed by the device described in FIG. 13, the M second parameters in the equivalent convolution kernel and the M second eigenvalues in the feature map can be input to M multipliers, the first in the feature map The characteristic value can be input to the shifter for shifting. The M second parameters and the M second eigenvalues are multiplied and accumulated by the M multipliers and the M adders in the multiplication and accumulation process, and the first eigenvalues are shifted to the right and expanded by n by the shifter. The eigenvalue obtained by the multiplication is input to ADD M-1, and the sum of the output of ADD M-1 is the convolution result of the equivalent convolution kernel and the feature map.

This is because when the second initial parameter or the second parameter is enlarged by m times, the first parameter should be enlarged by m times accordingly, so the processing result of the first feature value by the first parameter should also be enlarged by m times. The data processing device of the present application directly enlarges the first feature value by m times to ensure the accuracy of the convolution result.

For example, when the feature map is convolved based on the equivalent convolution kernel shown in the right image of Figure 11, the first feature value in the feature map can be input to the shifter and shifted to the right by 16 bits, and the resulting feature value can be multiplied by The accumulation result is added to obtain the convolution result.

In some possible implementation manners, the processor includes one or a combination of the following: a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (network process units, NPU).

In some possible implementation manners, M is equal to 8, and the two-dimensional convolution kernel is a 3*3 matrix. Optionally, the M multipliers and the M-1 first adders may constitute a multiplication accumulation adder.

For example, as shown in Figure 14, the data processing device may include 8 multipliers and 8 adders, of which 8 multipliers and 7 adders are used to multiply 8 convolution kernel parameters and 8 eigenvalues. Accumulation, another adder is used to add the result of the multiplication and accumulation to another eigenvalue, thereby obtaining the convolution result of the two-bit convolution kernel and the eigenvalue matrix to be convolved.

It is understandable that the data processing device may be based on processing 8 multiply and accumulate PEs and add an adder to obtain the data processing device in this application. That is to say, the data processing device of the present application can reuse the existing hardware logic, and then add an adder to obtain the data processing device of the present application.

Alternatively, the 8 multipliers and 7 adders in FIG. 14 can be composed of two PEs that can handle 4 multiplication and accumulation.

Or, as shown in FIG. 15, based on processing 16 multiply-accumulated PEs, an adder may be added to obtain two data processing devices as shown in FIG. 14. Among them, the output of ADD 6 is no longer output to ADD 14, but to the newly added ADD 15.

In some possible implementation manners, M is equal to 24, and the two-dimensional convolution kernel is a 5*5 matrix. Optionally, the M multipliers and the M-1 first adders constitute three multiplication and accumulation adders, wherein each of the multiplication accumulation adders includes M/3 multipliers and M/3-1 A first adder.

For example, based on 3 PEs that can handle 8 multiplication and accumulation, three adders can be added to obtain a 5*5 two-bit convolution kernel based on the 5*5 feature matrix to be convolved in this application. Data processing device. Among them, the three newly added adders are used to add the multiplication and accumulation results of the three PEs to the first eigenvalue in the feature matrix to be convolved.

In some implementation manners, the device shown in FIG. 14 may further include a shifter, and the output port of the shifter is connected to an input port of ADD 7; or, the device shown in FIG. 15 may also include Two shifters, the output port of one shifter is connected to the input port of ADD 15, and the output port of the other shifter is connected to the input port of ADD 14. In this case, the first eigenvalue in the feature matrix to be convolved is first input to the shifter for shifting, and the result of the shift and the multiplication and accumulation result are input to the adder and added to obtain the convolution result. This device including a shifter is used when performing convolution processing based on the equivalent convolution kernel obtained by the expansion or reduction of the initial convolution kernel. Of course, when convolution processing is performed based on the equivalent convolution kernel obtained by the initial convolution kernel without expansion or reduction processing, a device containing a shifter can also be used, but in this case, the shifter does not need to input The first eigenvalue of is just shifted.

In some possible implementation manners of this embodiment, the two-dimensional convolution kernel is a two-dimensional matrix component of an N-dimensional convolution kernel, and N is an integer greater than 2.

Taking a PE that can convolve a feature map containing 9 feature values based on a convolution kernel containing 9 parameters as an example, the data reading method proposed by this application will be exemplarily introduced in conjunction with FIG. 16 and FIG. 17. For example, the PE has the structure shown in FIG. 14 or FIG. 15. Further, for example, the PE is obtained by adding an adder or even a shifter based on a PE capable of processing 8 multiplication and accumulation.

As shown in Figure 16, the solid line box on the left represents the feature map, and a square dashed border in the solid line box on the left represents a convolution window. The convolution window moves to the right, and the moving step is 1, the size of the convolution window Determined by the size of the convolution kernel represented by the solid line box on the right, the feature values of the first row of the feature map are represented by A0 to An, and the feature values of the second row are represented by B0 to Bn, and so on; a solid box in the middle Represents a PE; the solid box on the right represents the convolution kernel K. An example of the convolution kernel is the aforementioned equivalent convolution kernel, and the size of the convolution kernel is 3*3. 16 PEs are exemplarily given in FIG. 16, which are respectively PE0 to PE15. For example, the neural network processor includes such 16 PEs. In Figure 16, the arrows indicate the flow of data.

Among them, each PE can convolve the eigenvalues in a convolution kernel window based on a convolution kernel, and then 16 PEs in a clock cycle can perform convolution on the eigenvalues in the 16 convolution kernel windows based on the same convolution kernel. convolution.

As shown in Figure 17, at the time of the initial clock, the directly connected buffer directly connected to the unit for convolution processing can read 3*(16+2) data from the lower-level buffer, these 3*(16+2) The data are respectively A0 to A17, B0 to B17 and C0 to C17, and then the direct connection cache can transmit A0 to A2, B0 to B2, C0 to C2 to PE0, and A1 to A3, B1 to B3 and C1 to C3 to PE1, and so on, until A15 to A17, B15 to B17, and C15 to C17 are output to PE15.

At the next clock, the direct-connected cache can only read (16+2) data from the lower-level cache. The 18 data are D0 to D17 respectively, and then B0 to B2, C0 to C2, D0 to D2 are transmitted to PE0 , Transmit B1 to B3, C1 to C3, D1 to D3 to PE1, and so on, until B15 to B17, C15 to C17, D15 to D17 are transmitted to PE15.

This is because the data that the direct-connected buffer needs to transmit to PE0 to PE15 partially overlap in two clock cycles, so the direct-connected buffer can reuse the repeated part read from the lower-level cache in the previous clock cycle in the next clock cycle Data, and only need to read part of the data that needs to be updated. This can save transmission resources and improve transmission efficiency.

It can be understood that the transmission of the parameters in the convolution kernel is not shown in FIG. 17.

If a PE includes two devices with the structure shown in Fig. 14 or Fig. 15, for example, the PE is obtained by adding an adder or even a shifter based on a PE capable of processing 16 multiplication and accumulation, then the direct connection buffer reads The method of data is similar to that shown in Figure 17. The difference is that, in the initial clock cycle, the direct-connected buffer in the horizontal latitude of the feature map needs to read 16 more data in the convolution window, or the vertical latitude of the feature map is from the next line of the convolution window in Figure 16. 16 more data are read in. That is to say, in the initial clock cycle, the direct-connected buffer needs to read 4*(16+2) data, and only 2*(16+2) data can be updated in each subsequent clock cycle.

Fig. 18 is an exemplary structural diagram of the data processing device of the present application. The device 1800 includes a processing module 1810, an acquisition module 1820, an expansion module 1830, and a determination module 1840. The device 1800 can implement the method shown in FIG. 5 described above.

For example, the processing module 1810 can be used to perform S510, the acquisition module 1820 can be used to perform S520, the expansion module 1830 can be used to perform S530, and the determination module 1840 can be used to perform S540.

In some possible implementation manners, the device 1800 may be the main CPU in FIG. 3; in other possible implementation manners, the device 1800 may be the training device 220 shown in FIG. 4; in other possible implementation manners , The apparatus 1800 may be the client device 240 described in FIG. 4.

The present application also provides an apparatus 1900 as shown in FIG. 19. The apparatus 1900 includes a processor 1902, a communication interface 1903, and a memory 1904. An example of the device 1900 is a chip. Another example of the apparatus 1900 is a computing device.

The processor 1902, the memory 1904, and the communication interface 1903 may communicate through a bus. Executable code is stored in the memory 1904, and the processor 1902 reads the executable code in the memory 1904 to execute the corresponding method. The memory 1904 may also include other software modules required for running processes, such as an operating system. The operating system can be LINUX ^TM , UNIX ^TM , WINDOWS ^TM etc.

For example, the executable code in the memory 1904 is used to implement the method shown in FIG. 5, and the processor 1902 reads the executable code in the memory 1804 to execute the method shown in FIG. 5.

Among them, the processor 1902 may be a central processing unit (CPU) or a graphics processing unit (GPU) or the like. The memory 1904 may include a volatile memory (volatile memory), such as random access memory (RAM). The memory 1904 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state starter (read-only memory, ROM). solid state disk, SSD).

This application also provides a data processing device 2000 as shown in FIG. 20, including a programmable device 2001 and a memory 2002, where the memory 2002 is used to store configuration files required for the operation of the programmable device 2001, and the programmable device 2001 uses The configuration file is read from the memory 2002 and executed to implement the corresponding method.

Among them, the programmable device may include a field programmable logic gate array (Field Programmable Gate Array, FPGA) or a complex programmable logic device (Complex Programmable Logic Device, CPLD).

Taking FPGA as an example, those skilled in the art can understand that its basic working principle is to change the content of the configuration RAM inside the FPGA by loading a configuration data (for example, in the form of a configuration file), thereby changing the configuration of various logic resources inside the FPGA In order to realize different circuit functions, and the configuration data can be loaded multiple times, so that the FPGA can complete different functions by loading different configuration data, which has good flexibility. In this practical application, it is often necessary to update the function of the FPGA. At this time, you can load the new configuration data into the FPGA configuration memory in advance, and then let the FPGA load the new configuration data to realize the functions defined by the new configuration data. This process is called FPGA The upgrade process. At the same time, when the FPGA leaves the factory, it will have a configuration loading circuit for loading configuration data. This configuration loading circuit can be used to ensure that the user-defined circuit function (that is, the function defined by the configuration data) fails. The most basic loading operation.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which is not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (28)

A data processing device, characterized in that the data processing device is used to perform convolution processing on a convolution matrix according to a two-dimensional convolution kernel, and the two-dimensional convolution kernel includes a first parameter and M second parameters , The matrix to be convolved includes a first eigenvalue and M second eigenvalues, the M second parameters are in one-to-one correspondence with the M second eigenvalues, and the first parameter corresponds to the first A characteristic value, the data processing device includes: M multipliers and M-1 first adders are used for multiplying and accumulating the M second parameters and the M second characteristic values to obtain a multiplying and accumulating result; The second adder is configured to perform an addition operation on the multiplication and accumulation result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved. The device of claim 1, wherein the first parameter is equal to one. The device according to claim 1 or 2, wherein the device further comprises a processor, the processor is configured to: determine the two-dimensional convolution kernel according to an initial convolution kernel, the initial convolution kernel comprising One first initial parameter and M second initial parameters, the M second parameters correspond to the M second initial parameters one-to-one, and the first parameter corresponds to the first initial parameter, The second parameter is equal to the quotient of the second initial parameter corresponding to the second parameter and the first initial parameter, and the first initial parameter is not zero. The device according to claim 3, wherein when the quotient of at least one of the second initial parameters and the first initial parameter is greater than a first threshold, the two-dimensional Before the convolution kernel, the processor is also used to: The M second parameters or the M second initial parameters are reduced by m times, wherein the quotient of any second initial parameter reduced by m times and the first initial parameter is not greater than the first threshold. The device according to claim 4, wherein, before the addition operation is performed on the multiplication and accumulation result and the first eigenvalue, the processor is further configured to: Reduce the first characteristic value by m times; Correspondingly, the second adder is specifically configured to perform an addition operation on the multiplication and accumulation result and the reduced first characteristic value. The device according to claim 3, wherein when the quotient of at least one of the second initial parameters and the first initial parameter is less than a second threshold, the two-dimensional Before the convolution kernel, the processor is also used to: The M second parameters or the M second initial parameters are expanded by n times, wherein the quotient of any second initial parameter expanded by n times and the first initial parameter is not less than the second threshold. 7. The device according to claim 6, wherein before the addition operation is performed on the multiplication and accumulation result and the first eigenvalue, the processor is further configured to: Expanding the first characteristic value by n times; Correspondingly, the second adder is specifically configured to perform an addition operation on the multiplication and accumulation result and the expanded first characteristic value. The device according to any one of claims 3 to 7, wherein the processor comprises one or a combination of the following: central processing unit (CPU), graphics processing unit (GPU), or neural network processing Device (NPU). The device according to any one of claims 1 to 8, wherein M is equal to 8, and the two-dimensional convolution kernel is a 3*3 matrix. 8. The device of claim 8, wherein the M multipliers and the M-1 first adders form a multiplication accumulation adder. The device according to any one of claims 1 to 8, wherein M is equal to 24, and the two-dimensional convolution kernel is a 5*5 matrix. The apparatus according to claim 11, wherein the M multipliers and the M-1 first adders form three multiplication accumulation adders, wherein each of the multiplication accumulation adders includes M/3 Multipliers and M/3-1 first adders. The device according to any one of claims 1 to 12, wherein the two-dimensional convolution kernel is a two-dimensional matrix component of an N-dimensional convolution kernel, and N is an integer greater than 2. A data processing method, characterized in that it comprises: According to the initial convolution kernel of the Lth layer of the neural network, the equivalent convolution kernel of the Lth layer is obtained, wherein the parameters of the equivalent convolution kernel of the Lth layer are based on the correspondence of the initial convolution kernel of the Lth layer The quotient of the parameter and the first initial parameter in the initial convolution kernel of the Lth layer is obtained, the value of the first initial parameter is K, and K is a non-zero number, and the equivalent convolution kernel of the Lth layer For performing convolution processing on the feature map of the L-th layer; Acquiring the initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with the initial convolution kernel of the Lth layer; Expand each parameter in the initial convolution kernel of the L+1th layer by K times; The equivalent convolution kernel of the L+1th layer is determined according to the initial convolution kernel of the L+1th layer after the expansion processing. The method according to claim 14, wherein the equivalent convolution kernel of the L-th layer includes M second parameters and a first parameter, and the M second parameters respectively correspond to those of the feature map. M second feature values, the first parameter corresponds to the first feature value of the feature map, and the first parameter is 1; Wherein, the performing convolution processing on the feature map of the L-th layer includes: Multiply and accumulate the M second parameters and the M second eigenvalues to obtain a multiply and accumulate result; An addition operation is performed on the multiplication and accumulation result and the first characteristic value. The method according to claim 14, wherein the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the initial convolution kernel of the Lth layer. The quotient of the first initial parameter in is obtained, including: When the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is greater than a first threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is reduced by m times, wherein, The quotient of any parameter of the initial convolution kernel of the L-th layer reduced by m times and the first initial parameter is not greater than the first threshold. The method according to claim 16, wherein said performing an addition operation on said multiplication and accumulation result and said first eigenvalue comprises: Reduce the first characteristic value by m times; An addition operation is performed on the multiplication and accumulation result and the reduced first characteristic value. The method according to claim 14, wherein the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the initial convolution kernel of the Lth layer. The quotient of the first initial parameter in is obtained, including: When the quotient of at least one parameter of the initial convolution kernel of the L-th layer and the first initial parameter is less than a second threshold, the corresponding parameter of the initial convolution kernel of the L-th layer is expanded by n times, wherein, The quotient of any parameter of the initial convolution kernel of the L-th layer expanded by n times and the first initial parameter is not less than the second threshold. The method according to claim 18, wherein said performing an addition operation on said multiplication and accumulation result and said first eigenvalue comprises: Expanding the first characteristic value by n times; An addition operation is performed on the multiplication and accumulation result and the expanded first eigenvalue. A data processing device, characterized in that it comprises: The processing module is used to obtain the equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the parameters of the equivalent convolution kernel of the Lth layer are based on the initial convolution kernel of the Lth layer The quotient of the corresponding parameter of the convolution kernel and the first initial parameter in the initial convolution kernel of the Lth layer is obtained, the value of the first initial parameter is K, K is a non-zero number, and the value of the Lth layer The equivalent convolution kernel is used to perform convolution processing on the feature map of the Lth layer; An obtaining module, configured to obtain the initial convolution kernel of the L+1th layer that has a mapping relationship with the initial convolution kernel of the Lth layer in the L+1th layer of the neural network; An expansion module, configured to expand each parameter in the initial convolution kernel of the L+1th layer by K times; The determining module is configured to determine the equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion processing. The device according to claim 20, wherein the equivalent convolution kernel of the Lth layer comprises M second parameters and a first parameter, and the M second parameters respectively correspond to those of the feature map. M second feature values, the first parameter corresponds to the first feature value of the feature map, and the first parameter is 1; Wherein, the performing convolution processing on the feature map of the L-th layer includes: Multiply and accumulate the M second parameters and the M second eigenvalues to obtain a multiply and accumulate result; An addition operation is performed on the multiplication and accumulation result and the first characteristic value. The device according to claim 20, wherein the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the initial convolution kernel of the Lth layer. The quotient of the first initial parameter in is obtained, including: When the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is greater than a first threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is reduced by m times, wherein, The quotient of any parameter of the initial convolution kernel of the L-th layer reduced by m times and the first initial parameter is not greater than the first threshold. The device according to claim 22, wherein said performing an addition operation on said multiplication and accumulation result and said first eigenvalue comprises: Reduce the first characteristic value by m times; An addition operation is performed on the multiplication and accumulation result and the reduced first characteristic value. The device according to claim 20, wherein the parameters of the equivalent convolution kernel of the Lth layer are based on the corresponding parameters of the initial convolution kernel of the Lth layer and the initial convolution kernel of the Lth layer. The quotient of the first initial parameter in is obtained, including: When the quotient of at least one parameter of the initial convolution kernel of the L-th layer and the first initial parameter is less than a second threshold, the corresponding parameter of the initial convolution kernel of the L-th layer is expanded by n times, wherein, The quotient of any parameter of the initial convolution kernel of the L-th layer expanded by n times and the first initial parameter is not less than the second threshold. The device according to claim 24, wherein said performing an addition operation on said multiplication and accumulation result and said first eigenvalue comprises: Expanding the first characteristic value by n times; An addition operation is performed on the multiplication and accumulation result and the expanded first eigenvalue. A computer-readable storage medium, comprising instructions, when the instructions run on a processor, the processor executes the method according to any one of claims 14 to 19. A data processing device, characterized by comprising a programmable device and a memory, the memory is used to store configuration files required for the operation of the programmable device, and the programmable device is used to read the The configuration file executes the method according to any one of claims 14 to 19. The device of claim 27, wherein the programmable device comprises a field programmable logic gate array (FPGA) or a complex programmable logic device (CPLD).

Images