CN108875925A - A kind of control method and device for convolutional neural networks processor - Google Patents

Abstract

The present invention provides a control method, including: 1) determining the size n*n of the convolution operation that needs to be performed; 2) selecting m ² 7*7 convolutions according to the size n*n of the convolution operation that needs to be performed Load the value of the convolution kernel corresponding to the size in the product calculation unit, and fill the remaining values with 0, 7m≥n; 3) According to the size of the convolution operation that needs to be performed, the input that needs to perform convolution The size of the feature map determines the number of cycles required for the convolution calculation process; 4) in each cycle of the convolution calculation process, load the value of the corresponding input feature map into the m ² 7*7 convolutions In the calculation unit, the distribution of the value of the input feature map in the m ² 7*7 convolution calculation units and the value of the convolution kernel in the m ² 7*7 convolution calculation units The distribution in is kept consistent; the m ² 7*7 convolution calculation units loaded with the convolution kernel and the value of the input feature map are controlled to perform convolution calculations corresponding to the number of cycles.

Description

A control method and device for a convolutional neural network processor

technical field

The invention relates to a convolutional neural network processor, in particular to improvements in hardware acceleration for the convolutional neural network processor.

Background technique

Artificial intelligence technology has developed rapidly in recent years and has attracted widespread attention all over the world. Both industry and academia have carried out research work on artificial intelligence technology, infiltrating artificial intelligence technology into visual perception, Speech recognition, assisted driving, smart home, traffic dispatching and other fields. Deep learning technology is a booster for the development of artificial intelligence technology. Deep learning uses the topological structure of the deep neural network for training, optimization, and reasoning. The deep neural network white block convolutional neural network, deep belief network, and cyclic neural network, etc., undergo repeated iterations and training. Taking the application of image recognition as an example, the deep learning algorithm can automatically obtain the characteristic data of the hidden image through the deep neural network, and produce better results than the traditional analysis methods based on pattern recognition.

However, the implementation of existing deep learning techniques relies on a huge amount of computation. In the training phase, it is necessary to obtain the weight data in the neural network through repeated iterative calculations in massive data; in the reasoning phase, it is also necessary to use the neural network to complete the calculation and processing of the input data within a very short response time (usually milliseconds). This requires the deployed neural network computing circuits (including CPUs, GPUs, FPGAs, and ASICs, etc.) to achieve hundreds of billions or even trillions of calculations per second. Therefore, hardware acceleration for deep learning techniques, such as convolutional neural network processors, is very necessary.

It is generally believed that there are two ways to achieve hardware acceleration, one is to use larger-scale hardware to perform calculations in parallel, and the other is to increase processing speed or efficiency by designing dedicated hardware circuits.

For the above-mentioned second method, some existing technologies directly map the neural network into a hardware circuit, and use different computing units for each network layer, so that the calculation for each network layer is performed in a pipelined manner. For example, each computing unit except the first computing unit takes the output of the previous computing unit as its input, and each computing unit is only used to perform calculations for its corresponding network layer, in different unit times of the pipeline , the calculation unit performs calculations on different inputs of the network layer. Such existing technologies are generally aimed at scenarios that require continuous processing of different inputs, such as processing a video file containing multiple frames of images. Also, such prior art usually targets neural networks with fewer network layers. This is because the number of network layers and the scale of the deep neural network are large, and the neural network is directly mapped to a hardware circuit. The cost of the circuit area is very large, and the power consumption will increase with the increase of the circuit area. In addition, considering that there is a large difference in the calculation time of each network layer, in order to realize the function of the pipeline, the running time provided to each pipeline level needs to be forced to be equal to each other, that is, equal to the operation of the slowest pipeline level time. For a deep neural network with a large number of network layers, the design pipeline needs to consider a lot of factors to reduce the waiting time required for the relatively fast processing pipeline level in the pipeline calculation process.

There are also some existing technologies that, in the case of referring to the calculation rules of the neural network, propose that "time division multiplexing" can be performed on the calculation units in the neural network processor to improve the multiplexing rate of the calculation units, which is different from the above-mentioned pipeline In this way, the same computing unit is used to sequentially calculate each network layer in the neural network. For example, calculations are performed on the input layer, the first hidden layer, the second hidden layer, ... the output layer one by one, and the above process is repeated in the next iterative calculation. Such prior art can be aimed at neural networks with fewer network layers, and can also be aimed at deep neural networks, and it is especially suitable for application scenarios with limited hardware resources. For such an application scenario, after the neural network processor calculates the network layer A for an input, it may not need to perform calculations for the network layer A for a long time. If each network layer uses different hardware As its calculation unit, it will lead to restrictions on hardware, so that the reuse rate of hardware is not high. Most of the existing technologies are based on such considerations, adopting different "time division multiplexing" methods for computing units to improve the hardware of the neural network processor accordingly.

However, no matter which of the above-mentioned existing techniques is used to design a convolutional neural network processor, there are still areas where hardware utilization needs to be improved.

Contents of the invention

Therefore, the object of the present invention is to overcome the defects of the above-mentioned prior art, and provide a control method for a convolutional neural network processor, the convolutional neural network processor has 7*7 convolution calculation units, the Control methods include:

1) Determine the convolution kernel size n*n of the convolution operation to be performed;

2) According to the convolution kernel size n*n of the convolution operation that needs to be performed, choose to load the value of the convolution kernel corresponding to the size in m ² 7*7 convolution calculation units, and use the rest Each value is filled with 0, 7m≥n;

3) Determine the number of cycles required for the convolution calculation process according to the size of the convolution operation that needs to be performed and the size of the input feature map that needs to be performed; and

4) According to the number of cycles, in each cycle of the convolution calculation process, the value of the corresponding input feature map is loaded into the m2 ⁷ *7 convolution calculation units, and the input feature map The distribution of the value in the m ² 7*7 convolution computing units is consistent with the distribution of the value of the convolution kernel in the m ² 7*7 convolution computing units;

Controlling the m ² 7*7 convolution calculation units loaded with the convolution kernel and the value of the input feature map respectively performs convolution calculations corresponding to the number of cycles;

5) Accumulate the corresponding elements in the convolution calculation results of the m ² 7*7 convolution calculation units to obtain the final output feature map of the convolution operation.

Preferably, according to the method, wherein step 2) comprises:

If the size of the convolution operation to be performed is smaller than 7*7, load the value of the convolution kernel corresponding to the size in the same 7*7 convolution calculation unit and fill the remaining values with 0;

If the size of the convolution operation to be performed is greater than 7*7, load the value of the convolution kernel corresponding to the size in the corresponding number of 7*7 convolution calculation units and fill the remaining values with 0 .

Preferably, according to the method, wherein step 4) comprises:

In each cycle of the convolution calculation process, if the value of the input feature map to be loaded contains the elements in the first column on the left of the input feature map, then the input feature map and the required execution Multiple elements that match the size of the convolution operation are loaded into the corresponding positions of the convolution calculation unit and the values of the remaining positions are filled with 0, otherwise, the same elements as in the previous cycle are used as a Move one unit to the left as a whole, and load multiple elements in the input feature map that are different from those in the previous cycle and need to be updated to the positions vacated by the shift.

Preferably, according to the method, wherein step 4) comprises:

In each cycle of the convolution calculation process, control the m ² 7*7 convolution calculation units to perform multiplication for the elements loaded on the input feature map and the corresponding positions of the convolution kernel, and The results of the multiplications are accumulated to obtain the elements at the corresponding positions in the output feature map.

Preferably, according to the method, wherein step 2) comprises:

If the size of the convolution operation to be performed is 5*5, load the value of the 5*5 convolution kernel in the same 7*7 convolution calculation unit and fill the remaining values with 0;

And, step 4) includes:

In each cycle of performing all cycles of convolution calculation, the value of the corresponding input feature map is loaded into the 7*7 convolution calculation unit, and the value of the input feature map is in the 7*7 The distribution in the convolution computing unit is consistent with the distribution of the value of the 5*5 convolution kernel in the 7*7 convolution computing unit;

Among them, in each cycle of the convolution calculation process, if the value of the input feature map to be loaded contains the elements in the first column on the left side of the input feature map, the size of the input feature map 25 elements of 5*5 are loaded into the corresponding positions of the convolution calculation unit and the values of the remaining positions are filled with 0, otherwise, the same elements as in the previous cycle are moved to the left as a whole One unit, and the 5 elements in the input feature map that are different from the previous cycle and need to be updated are loaded into the positions vacated by the move.

Preferably, according to the method, wherein step 2) comprises:

If the size of the convolution operation that needs to be performed is 3*3, load the value of the 3*3 convolution kernel for at most 4 channels in the same 7*7 convolution calculation unit and use the remaining The value is filled with 0;

And, step 4) includes:

In each cycle of all cycles of performing convolution calculation, the value of the corresponding input feature map is loaded into the 7*7 convolution calculation unit, and the value of the input feature map is the same as the 3*3 The form of one or more copies of the same number of convolution kernels is loaded into the 7*7 convolution computing unit, and the input feature map is in the 7*7 convolution computing unit The distribution corresponds to the distribution of the value of the 3*3 convolution kernel in the 7*7 convolution calculation unit;

Among them, in each cycle of the convolution calculation process, if the value of the input feature map to be loaded contains the elements in the first column on the left side of the input feature map, the size of the input feature map The 9 elements of 3*3 are loaded into the corresponding positions of the convolution calculation unit and the values of the remaining positions are filled with 0, otherwise, the same elements as in the previous cycle are moved to the left as a whole One unit, and load the three elements in the input feature map that are different from those in the previous cycle and need to be updated to the corresponding positions vacated by the move.

Preferably, according to the method, wherein step 4) also includes:

If the value of the 3*3 convolution kernel for 2 or 4 channels is loaded in the same 7*7 convolution calculation unit, and the value of the input feature map to be loaded does not include the input For the elements in the first column on the left in the feature map, the two copies of the input feature map in the 7*7 convolution calculation unit that are in the same number of columns and in different rows are compared with the previous cycle The same elements in are moved one unit to the left as a whole, and the three elements in the input feature map that are different from those in the previous cycle and need to be updated are loaded into the corresponding positions vacated by the movement.

Preferably, according to the method, wherein step 2) comprises:

If the size of the convolution operation to be performed is 11*11, control the four 7*7 convolution calculation units to jointly load the value of the 11*11 convolution kernel and fill the remaining values with 0;

And, step 4) includes:

In each cycle of all cycles of performing convolution calculation, the values of the corresponding input feature maps are loaded into the four 7*7 convolution calculation units, and the values of the input feature maps are in the four The distribution in the 7*7 convolution computing unit is consistent with the distribution of the value of the 11*11 convolution kernel in the four 7*7 convolution computing units;

Among them, in each cycle of the convolution calculation process, if the value of the input feature map to be loaded contains the elements in the first column on the left side of the input feature map, the size of the input feature map The 121 elements of 11*11 are loaded into the corresponding positions of the convolution calculation unit and the values of the remaining positions are filled with 0, otherwise, the same elements as in the previous cycle are moved to the left as a whole One unit, and the 11 elements in the input feature map that are different from those in the previous cycle and need to be updated are loaded into the corresponding positions vacated by the movement.

Preferably, according to the method, wherein step 4) comprises:

In each cycle of the convolution calculation process, the four 7*7 convolution calculation units are controlled to perform multiplication for the elements loaded on the input feature map and the corresponding positions of the convolution kernel, and to The results of the multiplication are accumulated in each cycle of the convolution calculation process, and the four 7*7 convolution calculation units are controlled respectively to the elements loaded for the input feature map and the corresponding positions of the convolution kernel Perform multiplication and accumulate the results of the multiplication;

And step 5) includes: accumulating the calculation results of all the four 7*7 convolution calculation units to obtain elements at corresponding positions in the output feature map.

And, a control unit, which is used to implement the control method described in any one of the above.

And, a convolutional neural network processor, comprising: a 7*7 convolutional calculation unit, and a control unit, the control unit is used to implement any one of the methods described above.

Compared with the prior art, the present invention has the advantages of:

The multiplexing rate of computing units used to perform convolution is improved, achieving the effect of reducing the hardware computing units that must be set in the convolutional neural network processor. The convolutional neural network processor does not need to set a large number of hardware computing units with different sizes for different convolutional layers that require convolution kernels of different sizes. When performing calculations for a convolutional layer, other computing units that do not match the size of the convolution kernel of the convolutional layer can be used for calculations, thereby improving the utilization of hardware computing units in convolutional neural network processors .

Description of drawings

Embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

Fig. 1 is a schematic diagram of using M kinds of convolution kernels to perform convolution calculation on an input layer to obtain an output layer in the prior art, wherein each convolution kernel has N channels;

FIG. 2 is a schematic diagram of a 7*7 convolution operation realized by a 7*7 computing unit in the prior art;

FIG. 3 is a schematic diagram of implementing a 5*5 convolution operation using a 7*7 computing unit according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of realizing a 3*3 convolution operation on 4 channels at one time by using a 7*7 computing unit according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of implementing 11*11 convolution operations by using four 7*7 computing units according to an embodiment of the present invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

The inventor found in the process of researching the existing technology that various existing classical neural networks, such as Alexnet, GoogleNet, VGG, Resnet, etc., all contain different numbers of convolutional layers, and different convolutional layers The size of the convolution kernel used is also different. For example Alexnet, the first layer of the network is a convolutional layer with a convolution kernel size of 11*11, the second layer of the network is a convolutional layer with a convolution kernel size of 5*5, and the third layer of the network is a convolutional layer A convolutional layer with a kernel size of 3*3 and so on.

However, in various existing neural network processors, different computing units are set for convolution kernels of different sizes. As a result, when the calculation of a certain convolution layer is performed, other calculation units that do not match the size of the convolution kernel of the convolution layer are idle.

For example, as shown in Figure 1, the neural network processor can provide M different convolution kernels, which are denoted as convolution kernel 0 to convolution kernel M-1, and each convolution kernel has N channels, respectively used Convolution calculation is performed on N channels of the input layer, and an output layer can be obtained after each convolution kernel is convolved with an input layer. For an input layer, M-1 output layers can be calculated by using all M kinds of convolution kernels. If a certain input layer needs to perform a convolution operation using convolution kernel 1, at this time, other computing units except the computing unit corresponding to convolution kernel 1 are in an idle state.

In this regard, the present invention proposes a multiplexing scheme for the computing unit, which adjusts the data actually loaded by the computing unit through control (for the same computing unit, it needs to load the value of the convolution kernel, or Need to load the value in the input feature map), so as to realize the convolution operation for multiple sizes with a computing unit of 7*7 scale, so as to reduce the scale of the hardware computing unit that must be used for convolution operation.

The system architecture of the neural network processor used in the present invention may include the following five parts, an input data storage unit, a control unit, an output data storage unit, a weight storage unit, and a calculation unit.

The input data storage unit is used to store the data involved in the calculation; the output data storage unit stores the calculated neuron response value; the weight storage unit is used to store the trained neural network weights;

The control unit is respectively connected with the output data storage unit, the weight storage unit and the calculation unit, and the control unit can control the calculation unit to perform neural network calculation according to the control signal obtained through analysis.

The calculation unit is used for performing corresponding neural network calculation according to the control signal generated by the control unit. The calculation unit completes most of the operations in the neural network algorithm, that is, vector multiplication and addition operations, etc.

The multiplexing of the calculation unit according to the present invention can be realized by controlling the calculation unit by the above-mentioned control unit, which will be specifically introduced through several embodiments below.

In the following, firstly, it will be introduced how the traditional prior art utilizes 7*7 computing units to implement 7*7 convolution operations. Referring to an example given in FIG. 2, for the prior art, a computing unit with a scale of 7*7 implements convolution operation in the following manner:

In the first cycle, each element in rows 1-7 and columns 1-7 in the input feature map (here referred to as a sliding window for the input feature map) is combined with each element in the corresponding position in the convolution kernel The cumulative sum of the results obtained by multiplying the elements separately is used as the element in the first row and the first column in the output feature map, that is, 2×(-4)+(3×2)+(-2×(-4))+ (2×(-8))+(-7×3)=-31.

In the second cycle, each element in rows 1-7 and columns 2-8 in the input feature map (that is, the value in the sliding window for the current cycle) is multiplied by each element in the corresponding position in the convolution kernel The cumulative sum of the obtained results is used as the element in row 1 and column 2 in the output feature map (not shown in FIG. 2 ).

By analogy, by moving the sliding window of size 7*7 to the right or down for a total of 15 times, an output feature map of size 4*4 is obtained.

The present invention does not exclude the use of the above method to realize the 7*7 convolution operation by using the 7*7 computing units. And, further, in the present invention, it is also possible to control the calculation unit of 7*7 scale to realize the operation of convolution kernels of other sizes than 7*7, such as 5*5, 3*3, 11 *11 convolution operation.

As mentioned above, in the traditional prior art, the size of the output feature map depends on the number of shifts of the sliding window and the size of the convolution kernel, for example, a 7*7 convolution for a 10*10 input feature map The product operation, the horizontal movement range and the vertical movement range of the sliding window are 4 units, and the output feature map of 4*4 can be obtained through the calculation of multiple cycles, which makes it possible to use 7*7 calculation units to realize volumes of other sizes. Multiplication is very difficult. It can be understood that, in the case of continuing to use the existing technology, using a 7*7 computing unit to perform convolution calculations on an input feature map of 10*10 can only obtain an output feature map of 4*4 (for example, as shown in Figure 2 Ground), the calculation unit and the processor do not know how to move the sliding window to be able to use the 7*7 calculation unit to obtain such as 5*5 convolution operation.

In this regard, the present invention proposes a corresponding control method, by scheduling the input feature maps and convolution kernels loaded into the computing unit, and controlling the execution of multiplication and addition operations, the 7*7 computing unit can perform 5* 5 convolution operations.

According to an embodiment of the present invention, referring to Fig. 3, the specific control method is as follows:

When the calculation unit performs convolution calculation, the control loads the value of the corresponding convolution kernel and the value of the corresponding input feature map into the 7*7 calculation unit each time the window is slid.

As shown in Figure 3, the size of the input feature map is 10*10, and the size of the convolution operation to be performed is 5*5, so it can be determined that the convolution calculation needs to be executed for a total of 6×6=36 cycles.

In the first cycle, the elements of rows 1-5 and columns 1-5 of the input feature map are loaded into a 7*7 computing unit as elements of rows 1-5 and columns 1-5, and Fill the remaining elements in rows 6, 7, and columns 6 and 7 with "0"; load the 5*5 convolution kernel into the 7*7 computing unit as rows 1-5, row 1 -5 column elements, and fill the remaining 6th, 7th row, 6th, 7th column elements with "0", thus loading the input feature map and convolution kernel in the 7*7 computing unit value. Control the 7*7 calculation unit to perform multiplication and accumulation on the input feature map and the corresponding position elements in the convolution kernel to obtain the elements of the first row and the first column in the output feature map, that is, (2×(-4)) +(3×2)+(-2×(-4))+(2×(-8))=-10. Since all the elements in the calculation unit except the value of the original 5*5 convolution kernel are 0, the calculation result is exactly the same as the result of the actual convolution calculation using the 5*5 calculation unit.

In the second cycle, all elements in rows 1-5 and columns 2-6 of the feature map will be input (ie "0,0,2,0,-3;0,3,-2,5,0;0 ,0,0,2,0; 0,0,0,3,0; 0,0,0,0,0") are loaded into the computational unit as rows 1-5, columns 1-5 of new elements. The control calculation unit performs multiplication and accumulation operations on the loaded elements to obtain the elements in row 1 and column 2 of the output feature map.

According to a preferred embodiment of the present invention, the above-mentioned manner of loading the data of the input feature map in the 7*7 computing units can also be improved in the second cycle, so as to improve the loading efficiency. That is, referring to FIG. 3 , all the elements in rows 1-5 and columns 2-5 in the 7*7 calculation unit (that is, "0,0,2,0; 0,3,-2,5; 0, 0,0,2; 0,0,0,3; 0,0,0,0") overall move to the left by 1 unit as a new element in rows 1-5, columns 1-4, and will The elements of rows 1-5 and column 6 in the input feature map (ie "-3; 0; 0; 0; 0") are loaded into the calculation unit as new elements of rows 1-5 and column 5 elements, thus updating the value of the input feature map loaded in the 7*7 computing unit, achieving an effect similar to the sliding window used in the traditional solution. And similarly control the calculation unit to perform multiplication and accumulation operations on the loaded elements to obtain the elements in the first row and the second column in the output feature map.

By analogy, the third to sixth cycles are completed.

In the seventh cycle, the elements of rows 2-6 and columns 1-5 of the input feature map are loaded into the computing unit of 7*7 as elements of rows 1-5 and columns 1-5, and The control calculation unit performs multiplication and accumulation operations on the loaded elements to obtain the elements in the second row and the first column in the output feature map. And in the subsequent eighth to twelfth cycles, the corresponding elements of the input feature map are loaded into the calculation unit in a manner similar to the aforementioned second to sixth cycles. By analogy, until all thirty-six cycles are completed, a 6*6 output feature map is obtained.

It can be seen that through the above control method, 25 values of 5*5 in the input feature map are loaded into the calculation unit at one time in the first cycle. Similarly, the seventh, thirteenth, nineteenth, twenty-fifth, and thirty-first cycles also loaded 25 values in the input feature map at one time. Correspondingly, in the second to sixth cycles, only 5 values of the input feature map need to be loaded each time, and the 20 values also used in the previous cycle are moved to the left. The value for the convolution kernel is not modified. Similarly, the eighth-twelfth, fourteenth-eighteenth, twenty-sixth-thirtieth, thirty-second-thirty-sixth cycles also use a similar method to the second to sixth cycle to load the input feature map elements in .

Thus, it can be ensured that in each cycle, the position of each element of the input feature map in the calculation unit is in one-to-one correspondence with the position of the corresponding element in the convolution kernel that is multiplied therewith. Moreover, for other units except the unit used to implement the control method of the present invention, such as the computing unit itself or the processor, they will not realize that the 7*7 convolution unit actually implements 5 *5 convolution operation. In addition, through the above control method, the value of the input feature map loaded by the calculation unit in each cycle does not directly depend on the sliding window. On the one hand, it is reflected in the fact that the numerical arrangement of the input feature map loaded into the computing unit does not depend on the actual arrangement of each numerical value in the sliding window with a size of 5*5; Depending on the number of shifts of the sliding window with a size of 7*7 (i.e. 4*4), the number and size of the output results can be controlled by the control method of the present invention, thus the 7*7 calculation unit can be used for 10*10 The input feature map is subjected to a 5*5 convolution operation and thus an output result of 6*6 is obtained.

Similarly, in a manner similar to the above-mentioned example in FIG. 3 , a convolution operation with a size smaller than 7*7, for example, a 3*3 convolution operation, can be performed in the same 7*7 computing unit. That is, load the 3*3 convolution kernel into the 7*7 computing unit, and fill the remaining values with "0". According to the size of the input feature map and the size 3*3 of the convolution operation to be performed, determine the number of cycles required for the convolution operation. In each cycle, the corresponding value in the input feature map is loaded into the 7*7 computing unit to perform the convolution operation.

It can be understood that, for a calculation unit using 7*7, the convolution operation such as 3*3 is only performed once at a time, and the utilization rate of its hardware is not high. In this regard, the present invention also proposes a solution to control the 3*3 convolution operations of four channels to be performed on the same input feature map at one time in the same 7*7 computing unit.

According to an embodiment of the present invention, a control method is also provided to implement a 3*3 convolution operation with a 7*7 calculation unit. Referring to FIG. 4, the specific control method is as follows:

The size of the input feature map is 10*10, and the size of the convolution operation to be performed is 3*3, so it can be determined that the convolution calculation needs to be performed for a total of 8×8=64 cycles.

In the first cycle, copy the elements in rows 1-3 and columns 1-3 of the input feature map into 4 copies, and load them into rows 1-3 and columns 1-3 and columns 1-3 of the 7*7 computing unit respectively. Line 1-3, column 4-6, line 4-6, column 1-3, line 4-6, column 4-6, and fill the remaining elements in line 7 and column 7 with "0"; The four 3*3 convolution kernels of the four channels are respectively loaded into the 1-3 rows 1-3 columns, 1-3 rows 4-6 columns, 4-6 rows 1 of the 7*7 computing unit -3 columns, 4-6 rows and 4-6 columns, and fill the remaining elements of the 7th row and 7th column with "0". In the embodiment shown in Figure 3, each convolution kernel is used for one channel, and when calculating the result of the convolution operation for multiple channels, the convolution results of the same position of each channel can be accumulated together as the output of this location. In the present invention, after the above-mentioned input feature map and elements of the convolution kernel are loaded, the 7*7 calculation unit can be controlled to perform multiplication and accumulation on the input feature map and the corresponding elements in the convolution kernel, which The obtained result is consistent with the accumulation result of the convolution results at the same position of the four channels, so that the elements in the first row and the first column of the output feature map can be obtained.

In the second cycle, all the elements in rows 1-3, columns 2-3 (ie "0,0; 0,3; 0,0") in the 7*7 calculation unit are moved to the left by 1 unit as a whole as New elements in rows 1-3 and columns 1-2, and the elements in rows 1-3 and columns 4 of the input feature map (ie "2;-2;0") are loaded into the calculation unit to As a new element for rows 1-3, column 3. Similarly, for the original 1-3 rows and 4-6 columns, 4-6 rows and 1-3 columns, and 4-6 rows and 4-6 columns in the calculation unit, the above-mentioned movement and loading of new elements are also performed operate. Thus, the value of the input feature map loaded in the 7*7 calculation unit is updated. Further, in the present invention, the original 1-6th rows, 2-3 columns, and 1-6th rows, 5-6 columns in the calculation unit can also be regarded as a whole to move, and/or will need The loaded new element is copied into two copies and loaded into the calculation unit as a whole, thereby reducing the steps required for control and operation. Similarly, the calculation unit is controlled to perform multiplication and accumulation operations on the loaded elements to obtain the elements in row 1 and column 2 of the output feature map.

By analogy, until all cycles are completed, an 8*8 output feature map is obtained.

It can be seen that through the above control method, the 3*3 convolution operation of four channels can be performed on the input feature map at one time, which is especially suitable for the case of a large number of channels. For the case where the number of channels is not equal to four, you can also choose to fill at least one area with a size of 3*3 in the 7*7 computing unit used to load the convolution kernel with "0", for example, fill the 4-6 Rows 4-6 are all filled with "0". For the case where the number of channels is greater than four, such as the case with seven channels, two calculations can be performed by controlling, 4 convolution kernels are loaded in the first calculation, and 3 volumes are loaded in the second calculation Accumulate kernel, and control the accumulation of the elements at the same position in the results of the two calculations to obtain the output result for this position.

The present invention introduces how to control the calculation unit of 7*7 to realize the convolution calculation of 5*5 and 3*3 through the above-mentioned embodiment. The following will introduce how to control the calculation unit of 7*7 to realize the convolution with a size exceeding 7*7 calculate.

According to an embodiment of the present invention, a control method is provided to implement a 11*11 convolution operation with a 7*7 computing unit. Referring to FIG. 5, the specific control method is as follows:

First of all, it is judged that 11>7, so more than one 7*7 computing unit is required to jointly complete the convolution operation with a size of 11*11. Here, k computing units that can just be used to load data with a size of 11*11 can be selected. Here, the choice of k is: k=m ² , and 7m can choose the smallest positive integer greater than or equal to n. Of course, more 7*7 computing units than the above number can also be selected to perform 11*11 convolution operations. For the example shown in Fig. 5, k=4 calculation units are selected here.

The control divides the value of the used convolution kernel into four parts and loads them into four 7*7 computing units respectively, and fills the rest with "0"; and, in each cycle, the control will input the corresponding The data is divided into four parts and loaded into the four 7*7 computing units respectively, and the remaining part is filled with "0". Here, in the four 7*7 computing units, the distribution of the values of the convolution kernel is consistent with the distribution of the values of the input feature map.

In addition, each calculation unit is controlled to perform multiplication and accumulation operations on the loaded elements, and the corresponding calculation results of all four calculation units are accumulated to obtain the value of the corresponding output feature map.

In this embodiment, the value of the input feature map in each calculation unit can be updated by moving and loading the value of the corresponding input feature map in each cycle in a manner similar to the previous embodiment . For example, in the second cycle, move the values in columns 2-7 of the first 7*7 calculation unit (which is in the upper left corner of Figure 5) to the left by 1 unit, and load it in column 7 New value; move the value of the 2nd-4 column of the second 7*7 calculation unit (it is in the upper right corner in Figure 5) to the left by 1 unit, and load the new value in the 4th column; Move the value of the third 7*7 calculation unit (it is in the lower left corner of Figure 5) in the 2-7 column 1-4 row to the left by 1 unit, and load it in the 7th column 1-4 row New value; move the value of the fourth 7*7 calculation unit (it is in the lower right corner of Figure 5) in the 2-4 column 1-4 row to the left by 1 unit, and place it in the 4th column 1-4 Load the new value into the row.

In the present invention, a corresponding control unit can be set for the above-mentioned control method. Such a control unit can be adapted to an existing convolutional neural network processor, and the calculation unit used for convolution can be implemented by implementing the above-mentioned control method. For multiplexing, a supporting convolutional neural network processor can also be designed based on the hardware resources required by such a control unit, for example, using the minimum number of hardware resources when the above multiplexing scheme is satisfied.

The solution provided by the present invention involves improving the multiplexing rate of the computing units used to perform convolution, so as to reduce the hardware computing units that must be set in the convolutional neural network processor, and the convolutional neural network processor does not have to adopt different A large number of hardware computing units with different sizes are set for different convolution layers of convolution kernels of different sizes. When performing calculations for one convolutional layer, computing units of the same size can be used to implement convolutional calculations for different convolutional layers, thereby improving the utilization of hardware computing units in the convolutional neural network processor.

It can be understood that the present invention does not exclude the use of larger-scale hardware to perform calculations in parallel and increase the multiplexing rate of calculation units through "time division multiplexing" as introduced in the background art.

Moreover, it should be noted that not all the steps described in the foregoing embodiments are necessary, and those skilled in the art may make appropriate trade-offs, replacements, modifications, etc. according to actual needs.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limit them. Although the present invention has been described in detail above with reference to the embodiments, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all of them should be covered in Within the scope of the claims of the present invention.

Claims (11)

1. a kind of control method for convolutional neural networks processor, the convolutional neural networks processor has the volume of 7*7 Product computing unit, the control method include：

1) the convolution kernel size n*n of the convolution algorithm needed to be implemented is determined；

2) the convolution kernel size n*n of the convolution algorithm executed as needed is selected in m²It is loaded into the convolutional calculation unit of a 7*7 The numerical value of convolution kernel corresponding with the size, and remaining each numerical value is filled with 0,7m >=n；

3) size of the convolution algorithm executed as needed and need to be implemented convolution input feature vector figure size, determine volume Periodicity needed for product calculating process；And

4) according to the periodicity, the numerical value of corresponding input feature vector figure is loaded by each period during convolutional calculation To the m²In the convolutional calculation unit of a 7*7, the numerical value of the input feature vector figure is in the m²The convolutional calculation unit of a 7*7 In distribution with the numerical value of the convolution kernel in the m²Distribution in the convolutional calculation unit of a 7*7 is consistent；

Control is loaded with the m of the numerical value of convolution kernel and input feature vector figure²The convolutional calculation unit of a 7*7 execute respectively with The corresponding convolutional calculation of the periodicity；

5) to the m²Corresponding element adds up in the convolutional calculation result of the convolutional calculation unit of a 7*7, final to obtain Convolution algorithm output characteristic pattern.

2. according to the method described in claim 1, wherein step 2) includes：

If the size of the convolution algorithm needed to be implemented be less than 7*7, then in the convolutional calculation unit of the same 7*7 be loaded into and institute It states the numerical value of the corresponding convolution kernel of size and remaining each numerical value is filled with 0；

If the size of the convolution algorithm needed to be implemented is greater than 7*7, then it is loaded into the convolutional calculation unit of the 7*7 of respective numbers Remaining each numerical value is simultaneously filled with 0 by the numerical value of convolution kernel corresponding with the size.

3. according to the method described in claim 1, wherein step 4) includes：

In each period during convolutional calculation, if comprising the input in the numerical value for the input feature vector figure for needing to be loaded into The element of left side first row in characteristic pattern, then disposably by the size in the input feature vector figure with the convolution algorithm needed to be implemented The multiple elements to match are loaded into the corresponding position of the convolutional calculation unit and fill out the numerical value of remaining each position Filling is 0, a unit otherwise will be then moved to the left as a whole with element identical in previous cycle, and will input spy Multiple elements that and needs different from previous cycle update in sign figure are loaded into the position vacated by the movement Place.

4. according to the method described in claim 1, wherein step 4) includes：

In each period during convolutional calculation, the m is controlled²The convolutional calculation unit of a 7*7 is respectively to loaded by it For input feature vector figure and multiplication is executed for the element of the corresponding position of convolution kernel and is added up to the result of multiplication, To obtain the element of corresponding position in output characteristic pattern.

5. method described in any one of -4 according to claim 1, wherein step 2) includes：

If the size of the convolution algorithm needed to be implemented is 5*5, then it is loaded into 5*5's in the convolutional calculation unit of the same 7*7 Remaining each numerical value is simultaneously filled with 0 by the numerical value of convolution kernel；

Also, step 4) includes：

In each period in the whole periods for executing convolutional calculation, the numerical value of corresponding input feature vector figure is loaded into the 7* In 7 convolutional calculation unit, distribution of the numerical value of the input feature vector figure in the convolutional calculation unit of the 7*7 and the 5* Distribution of the numerical value of 5 convolution kernel in the convolutional calculation unit of the 7*7 is consistent；

Wherein, in each period during convolutional calculation, if comprising institute in the numerical value for the input feature vector figure for needing to be loaded into The element for stating left side first row in input feature vector figure, then disposably by 25 elements in the input feature vector figure having a size of 5*5 It is loaded into the corresponding position of the convolutional calculation unit and the numerical value of remaining each position is filled with 0, it otherwise then will be with Identical element is moved to the left a unit as a whole in previous cycle, and by input feature vector figure with previous cycle 5 elements that middle different and needs update are loaded at the position vacated by the movement.

6. method described in any one of -4 according to claim 1, wherein step 2) includes：

If the size of the convolution algorithm needed to be implemented is 3*3, then spininess is loaded into the convolutional calculation unit of the same 7*7 Remaining each numerical value is simultaneously filled with 0 by numerical value to 4 channels, 3*3 convolution kernel；

Also, step 4) includes：

In each period in the whole periods for executing convolutional calculation, the numerical value of corresponding input feature vector figure is loaded into the 7* In 7 convolutional calculation unit, the numerical value of the input feature vector figure is with one or more equal with the quantity of the convolution kernel of the 3*3 The form of a copy is loaded into the convolutional calculation unit of the 7*7, and the input feature vector figure is in the convolution of the 7*7 Distribution in computing unit is corresponding with distribution of the numerical value of the convolution kernel of the 3*3 in the convolutional calculation unit of the 7*7；

Wherein, in each period during convolutional calculation, if comprising institute in the numerical value for the input feature vector figure for needing to be loaded into The element for stating left side first row in input feature vector figure, then disposably carry 9 elements in the input feature vector figure having a size of 3*3 Enter the corresponding position to the convolutional calculation unit and the numerical value of remaining each position be filled with 0, otherwise then will with it is preceding Identical element is moved to the left a unit as a whole in one period, and will be in input feature vector figure and in previous cycle 3 elements that different and needs update are loaded into the corresponding position vacated by the movement.

7. according to the method described in claim 6, wherein step 4) further includes：

If being loaded into the numerical value for being directed to 2 or 4 channels, 3*3 convolution kernel in the convolutional calculation unit of the same 7*7, And the element for not including left side first row in the input feature vector figure in the numerical value for the input feature vector figure for needing to be loaded into, then by institute State 2 copies in the convolutional calculation unit of 7*7 in same number of columns and in different line numbers, about the input feature vector figure In with element identical in previous cycle be moved to the left a unit as a whole, and by input feature vector figure with it is previous 3 elements that different and needs update in period are loaded into the corresponding position vacated by the movement.

8. method described in any one of claim 1-4, wherein step 2) includes：

If the size of the convolution algorithm needed to be implemented is 11*11, then control is loaded into jointly by the convolutional calculation unit of four 7*7 Remaining each numerical value is simultaneously filled with 0 by the numerical value of the convolution kernel of 11*11；

Also, step 4) includes：

In each period in the whole periods for executing convolutional calculation, the numerical value of corresponding input feature vector figure is loaded into described four In the convolutional calculation unit of a 7*7, distribution of the numerical value of the input feature vector figure in the convolutional calculation unit of four 7*7 It is consistent with the distribution of the numerical value of the convolution kernel of the 11*11 in the convolutional calculation unit of four 7*7；

Wherein, in each period during convolutional calculation, if comprising institute in the numerical value for the input feature vector figure for needing to be loaded into The element for stating left side first row in input feature vector figure, then disposably by 121 members in the input feature vector figure having a size of 11*11 Element is loaded into the corresponding position of the convolutional calculation unit and the numerical value of remaining each position is filled with 0, otherwise then will Be moved to the left a unit as a whole with element identical in previous cycle, and by input feature vector figure with the last week 11 elements that interim different and needs update are loaded into the corresponding position vacated by the movement.

9. according to the method described in claim 8, wherein step 4) includes：

In each period during convolutional calculation, the convolutional calculation unit of four 7*7 is controlled respectively to loaded by it For input feature vector figure and execute and multiplication and the result of multiplication carried out tired for the element of the corresponding position of convolution kernel Add；

And step 5) includes：It adds up to the calculated result by all convolutional calculation units of four 7*7, to obtain Export the element of corresponding position in characteristic pattern.

10. a kind of control unit, for realizing the control method as described in any one of claim 1-9.

11. a kind of convolutional neural networks processor, including：The convolutional calculation unit and control unit of 7*7, the control are single Member is for realizing any one of such as claim 1-9 the method.