WO2022044844A1 - Signal processing device and method, and program - Google Patents

Abstract

The present technology pertains to a signal processing device and method and a program that make it possible to reduce the amount of memory. Provided is a signal processing device for performing processing composed of one or a plurality of arithmetic processes on input data on a frame by frame basis. The signal processing device is provided with an arithmetic processing unit that preforms, on the basis of some or all of the data of a predetermined frame that has been input, processing for part of the arithmetic process for a future frame relative to the predetermined frame, and causes a processing result of the partial processing to be retained. The present technology is applicable to personal computers.

Description

Signal processing equipment and methods, as well as programs

The present technology relates to signal processing devices and methods, and programs, and in particular, to signal processing devices, methods, and programs capable of reducing the amount of memory.

Conventionally, automatic identification technology such as identification processing for audio data such as voice recognition, speaker identification, and environmental sound identification, and identification processing for image data such as image recognition is known.

As a method for performing such identification processing, a method using linear identification, a decision tree, a support vector machine, a neural network, etc. has been proposed.

For example, a technology that reduces the amount of memory by matching the processing boundaries of arithmetic processing in adjacent frames when the identification processing is realized by performing arithmetic processing such as convolution processing in frame units using a neural network. Has been proposed (see, for example, Patent Document 1).

International Publication No. 2019/146398

However, with the above-mentioned technology, it may not be possible to sufficiently reduce the amount of memory, such as when the number of input data channels is large.

This technology was made in view of such a situation, and makes it possible to reduce the amount of memory.

The signal processing device of one aspect of the present technology is a signal processing device that performs processing consisting of one or a plurality of arithmetic processes on an input data in a frame unit, and is used for a part or all of the input predetermined frame data. Based on this, it is provided with an arithmetic processing unit that performs a part of the arithmetic processing of a frame in the future from the predetermined frame and holds the processing result of the partial processing.

The signal processing method or program of one aspect of the present technology is a signal processing method or program of a signal processing device that performs processing consisting of one or a plurality of arithmetic processes on an input data in a frame unit, and is a predetermined frame input. The present invention includes a step of performing a part of the arithmetic processing of a frame after the predetermined frame based on a part or all of the data of the above, and holding the processing result of the part of the processing.

In one aspect of the present technology, in a signal processing device that performs processing consisting of one or a plurality of arithmetic processes on an input data in frame units, the predetermined data is based on a part or all of the input data of a predetermined frame. A part of the arithmetic processing of the frame in the future than the frame is performed, and the processing result of the part of the processing is retained.

It is a figure explaining the DNN processing in the frame unit. It is a figure explaining an input holding method. It is a figure explaining the convolution process and the pooling process. It is a figure explaining an output holding method. It is a figure explaining the specific example of the convolution processing by an output holding method. It is a figure explaining the specific example of the convolution processing by an output holding method. It is a figure explaining the specific example of the pooling process by the output holding method. It is a figure which shows the configuration example of a signal processing apparatus. It is a flowchart explaining the identification process. It is a figure which shows the configuration example of a computer.

Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

<First Embodiment>
<About this technology>
This technology performs a part of the arithmetic processing of the future frame based on the part used for the arithmetic processing of the future frame in the data of the current frame when the arithmetic processing of the predetermined layer constituting the neural network is performed. By holding the calculation result in the memory, the amount of memory can be reduced.

In this case, for example, in the next frame, the calculation in the predetermined layer in the next frame is performed from the result of the calculation processing based on the data of the next frame and the calculation result of a part of the data of the current frame held in the memory. The result of processing can be obtained.

For example, a device that performs processing by a neural network (hereinafter referred to as DNN (Deep Neural Network)) on input data that is time-series data such as voice in a certain time interval unit, here, in a frame unit of input data. think. The processing in frame units is to perform processing with the position of the beginning or the end of the frame as the start position of processing.

In this case, when the frame length of the input data is shorter than the length of the input data used in the entire processing in DNN performed for that frame, the amount of calculation for the calculation processing of the part that overlaps with the calculation processing of the past frame. There is a possibility that (calculation amount) can be reduced.

For example, as shown in FIG. 1, it is assumed that there is a device (discriminator) that performs identification processing on the input data DT11 by DNN and outputs a predetermined identification result (inference result) for the input data DT11 as the processing result. ..

Note that this technology can be applied not only to classifiers but also to any other processing, but an example of applying this technology to classifiers will be described below.

In this example, the input data DT11 is 3 channels of audio data with a sampling frequency of 24 kHz.

In the identification process by DNN (hereinafter, also referred to as DNN process), when the input data DT11 is subjected to the identification process that outputs one identification result for the input data at 1024 sample intervals, that is, at intervals of one frame. do.

At this time, it is assumed that the input data DT11 having a length of 23760 samples is input to the DNN in order to obtain one identification result. For example, the audio data for 23760 samples is the data for about 1 second.

Specifically, for example, as shown in FIG. 2, the DNN that performs the identification process for the input data DT11 is composed of nine layers of layers L1 to L9, and the convolution process or pooling process is layered (layer process) in each layer. It is assumed that it is performed as arithmetic processing).

That is, FIG. 2 shows the configuration of DNN, which is a discriminator of a neural network structure that performs identification processing on the input input data DT11 and outputs the identification result.

In FIG. 2, each quadrangle represents the shape of data, and in particular, in the figure of each data, the vertical direction shows the time direction, and the horizontal direction in the figure shows the dimension.

Also, in the figure, the horizontal arrow indicates data conversion, that is, layer processing. Here, the data shape of each data is determined by the number of dimensions of the data and the number of data (number of data samples) in each dimension, that is, the number of samples in the time direction.

Further, in FIG. 2, the numerical value shown on the left side in the figure of each data indicates the length (number of samples) of the data in the time direction, and is shown on the upper side in the figure of each data. The numbers shown indicate the number of dimensions of those data.

For example, in the figure, the square input data DT11 at the left end is audio data for 3 channels input to the DNN.

Here, in the figure of the input data DT11, the vertical direction indicates the time direction, and the horizontal direction in the figure indicates the dimension (channel). In particular, in the figure of the input data DT11, the downward direction is the newer time direction (future direction).

Input data DT11 is audio data for 3 channels in the time interval (length) for 23760 samples.

Here, the audio data channel corresponds to the dimension of the input data DT11, and the shape of the input data DT11 is 23760 samples x 3 dimensions. Further, in the figure of the input data DT11, the sample on the upper side is a past sample.

In the DNN shown in FIG. 2, layer processing (arithmetic processing) is performed in each of the nine layers, that is, each layer L1 to L9, with the input data DT11 as an input, and the identification result data DT20 at the right end in the figure is Obtained as output.

That is, one identification result data DT20 is obtained for the input of the input data DT11 having a length of 23760 samples per processing in one frame.

In this example, in DNN, convolution processing and pooling processing are repeated in order as arithmetic processing in each layer, and finally one identification result data DT20 is output.

This identification result data DT20 is 1 × 1 (1 sample × 1 dimension) data showing the identification result by DNN. For example, the identification result data DT20 is data indicating the probability that the voice based on the input data DT11 is a predetermined specific voice.

In the DNN shown in FIG. 2, first, in the first layer L1, the input data DT11 is filtered for each of the 16 types of convolution filters. In other words, for each of the 16 types of convolution filters, a convolution process is performed between the filter coefficients constituting the convolution filter and the input data DT11.

The convolution filter used in the convolution process of layer L1 is a filter with a shape of 96 x 3 taps. In other words, the number of taps on the convolution filter, that is, the number of filter coefficients, is 96 × 3. In addition, in the convolution process with the convolution filter, 16 sample advance filter processing is performed.

In FIG. 2, "size" indicates the size of the convolution filter in the time direction. For example, the layer L1 convolution filter has 96 taps in the time direction and 3 taps in the dimension (channel) direction, so the size of the layer L1 convolution filter is "96". Hereinafter, the number of taps in the time direction of the convolution filter will also be referred to as a size.

Further, in the following, a layer to which the convolution process is performed, such as layer L1, will be referred to as a convolution layer in particular. Similarly, hereinafter, the layer on which the pooling process is performed will also be referred to as a pooling layer.

By such convolution processing in layer L1, the input data DT11 is converted into the intermediate data DT12. The intermediate data DT12 is 16-dimensional data. That is, the intermediate data DT12 consists of 16 data of each dimension arranged in the horizontal direction in the figure.

In the second layer L2, the pooling process of extracting the maximum value from the data area of 8 samples x 1 dimension is performed for the intermediate data DT12 by advancing 8 samples.

In other words, the pooling process of 8 samples forward, which extracts the maximum value for the data area of 8 samples that are continuously arranged in the time direction, is performed on the intermediate data DT12.

In the following, the length of the data area to be convolved or pooled in the time direction is also referred to as "size", and the length of the data area in the dimensional direction is also referred to as "width".

For example, in FIG. 2, the layer L2 is marked with "size 8" and "advance 8", and the layer L2 performs a pooling process of 8 samples advancing for an 8 sample × 1-dimensional data area. You can see that.

By such pooling processing, the intermediate data DT12 is converted into the 16-dimensional intermediate data DT13.

Here, with reference to FIG. 3, the convolution process in the layer L1 and the pooling process in the layer L2 will be described.

Note that, in FIG. 3, the same reference numerals are given to the parts corresponding to the cases in FIG. 2, and the description thereof will be omitted as appropriate. Further, in FIG. 3, the horizontal direction indicates the dimensional direction, and the vertical direction indicates the time direction. In particular, in FIG. 3, the downward direction is the future direction.

For example, in layer L1, assuming that the convolution process is performed on the 96 sample × 3D data area W11 in the input data DT11 as shown on the left side in the figure, the convolution process is advanced by 16 samples and the 96 sample × 3D data area. The following convolution process is performed on W12.

Here, the data area W11 and the data area W12 are displaced by 16 samples. Therefore, for example, if the sample at the lower end of the data area W12 is counted upward in the figure as the first sample, the sample at the lower end of the data area W11 is the 17th sample.

As described above, in layer L1, the convolution process of shifting the data area to be processed by 16 samples is repeatedly performed for each of the 16 types of convolution filters, and the intermediate data DT12 is generated.

For example, the value (processing result) obtained by convolving the data area W11 with the first convolution filter is the sample value of the sample SP11, which is the second from the bottom in the figure of the intermediate data DT12. ..

Similarly, the value obtained by convolving the data area W12 with the first convolution filter is the sample value of the sample SP12, which is the first from the bottom in the figure of the intermediate data DT12.

Further, for example, the value obtained by convolving the data area W11 with the second convolution filter is the sample value of the sample adjacent to the right side in the figure of the sample SP11 in the intermediate data DT12.

In layer L2, assuming that the pooling process is performed on the 8 samples × 1 dimensional data area W21 in the intermediate data DT12 as shown in the center of the figure, the pooling process is advanced by 8 samples and the 8 samples × 1 dimensional data area W22. The following pooling process is performed on the data.

For example, in the pooling process for the data area W21, the maximum value of the sample value of the sample in the data area W21 is extracted, and the maximum value is obtained as the processing result of the pooling process.

Also, here, the data area W21 and the data area W22 are offset by 8 samples.

In layer L2, pooling processing is performed not only for the data of the leftmost dimension in the figure of the intermediate data DT12, but also for the data of each dimension. For example, pooling processing is also performed on the data area W23 of 8 samples × 1 dimension adjacent to the data area W21.

As described above, in layer L2, the pooling process of shifting the data area to be processed by 8 samples is repeated for each dimension of the intermediate data DT12, and the intermediate data DT13 is generated.

For example, the value (processing result) obtained by the pooling process for the data area W21 is the sample value of the sample SP21, which is the second from the bottom on the left end in the figure of the intermediate data DT13.

Similarly, the value obtained by the pooling process for the data area W22 is the sample value of the sample SP22 adjacent to the lower side in the figure of the sample SP21 in the intermediate data DT13. Further, the value obtained by the pooling process for the data area W23 is taken as the sample value of the sample SP23 adjacent to the right side in the figure of the sample SP21 in the intermediate data DT13.

Returning to the explanation of FIG. 2, when the intermediate data DT13 is obtained by the pooling process in the layer L2, the third layer, the layer L3, uses 32 types of convolution filters and the same convolution process as in the layer L1. Is done.

That is, in layer L3, the convolution processing (filter processing) of 1 sample advance by the 8 × 16 (8 samples × 16 dimensions) convolution filter having a size of 8 and a width in the dimensional direction of 16 is applied to the intermediate data DT13. It is done for each of the 32 types of convolution filters.

As a result, the intermediate data DT13 is converted into the 32-dimensional intermediate data DT14.

In layer L4, the pooling process targeting a 2 × 1 (2 samples × 1 dimension) data area with a size of 2 and a width of 1 in the dimension direction is advanced by 2 samples to each dimension of the intermediate data DT14. It is done against. As a result, the intermediate data DT14 is converted into the 32-dimensional intermediate data DT15.

In layer L5, one-sample forward convolution processing by a 4 x 32 (4 samples x 32 dimensions) convolution filter with a size of 4 and a width of 32 in the dimensional direction performs 64 types of convolutions for the intermediate data DT15. It is done for each filter. As a result, the intermediate data DT15 is converted into the 64-dimensional intermediate data DT16.

In layer L6, the pooling process targeting a 2 × 1 (2 samples × 1 dimension) data area with a size of 2 and a width of 1 in the dimension direction is advanced by 2 samples to each dimension of the intermediate data DT16. The intermediate data DT16 is converted into the 64-dimensional intermediate data DT17.

In layer L7, the convolution process of 1 sample advance by the 2 × 64 (2 samples × 64 dimensions) convolution filter with the size 2 and the width in the dimension direction is 64, 128 kinds of convolutions for the intermediate data DT17. It is done for each filter. As a result, the intermediate data DT17 is converted into the 128-dimensional intermediate data DT18.

In layer L8, pooling processing targeting a 2 × 1 (2 samples × 1 dimension) data area with a size of 2 and a width in the dimension direction of 1 is performed for each dimension of intermediate data DT18 by advancing 2 samples. The intermediate data DT18 is converted into 128-dimensional intermediate data DT19.

In layer L9, one sample forward convolution process by one type of convolution filter of 21 x 128 (21 samples x 128 dimensions) with a size of 21 and a width in the dimension direction of 128 is performed for the intermediate data DT19. Will be.

As a result, the intermediate data DT19 is converted into 1 sample × 1-dimensional identification result data DT20 which is the output of DNN, and the identification result data DT20 thus obtained is the result of the identification process (identification result) for the input data DT11. ) Is output.

Here, an example in which the identification result data DT20 is 1 sample × 1 dimensional data will be described, but what kind of sample number (length in the time direction) and dimension of the identification result data DT20 which is the output data of the DNN will be described. It may be numerical data.

The arithmetic processing in each of the plurality of layers from layer L1 to layer L9 as described above is performed by the discriminator (DNN) of the neural network structure.

In particular, in the convolution layer, the convolution process by the convolution filter in the shape of "size" x "number of dimensions of the input data" is performed on the input data of the convolution layer by a predetermined number of convolutions. It is done at "forward" sample intervals by the amount of the filter.

In the pooling layer, the pooling process that outputs the maximum value of the sample values of multiple samples included in the "size" x one-dimensional data area for the data input to the pooling layer is performed for each data dimension. It is done at "forward" sample intervals.

In FIG. 2, the entire rectangle representing each data of the input data DT11 and the intermediate data DT12 to the intermediate data DT19 shows the data portion to be substantially processed in the layer in which the data is input.

The simplest way to operate such a DNN is to hold the input data DT11 of the past frame of the length of 22736 (= 23760-1024) samples, and hold the input data DT11 and the current input data DT11. This is a method of inputting input data DT11 with a length of 23760 samples to DNN together with input data DT11 with a length of 1024 samples of a frame. Hereinafter, such a method will also be referred to as a general method.

In such a general method, it is necessary to perform identification processing (DNN processing) for 23760 samples × 3 dimensions in each frame, and the amount of calculation (calculation amount) of the identification processing becomes large. In addition, a memory amount of 22736 samples x 3 dimensions is required to hold the input data DT11 of the past frame.

Therefore, for example, by holding a part of the processing result of the layer processing in the frame immediately before the time of the current frame (hereinafter, also referred to as the previous frame) in the memory, the calculation amount of the layer processing performed in the current frame is minimized. It is conceivable to reduce the amount of calculation and the amount of memory of the entire identification process by limiting it to the limit. Hereinafter, such a method will also be referred to as an input holding method.

In the input holding method, a part of the data of the input previous frame is held in the memory in a predetermined layer, and the current frame is based on the part of the data of the previous frame and the data input in the current frame. Layer processing is performed for.

In the following description, it is assumed that the DNN (discriminator) for each method has a structure in which the processing boundaries of the previous frame and the current frame in the layer processing of each layer match.

In such a DNN, in at least one layer, the processing unit of the layer processing performed on the frame in that layer, that is, the data area to be processed or the number of sample advances is the entire input data DT11 or one frame. The size or the number of advances is determined by the data shape of the part used in the processing of. Further, for example, the frame length (number of frame samples) of the input data DT11 is also set to be a length determined by the data shape of the entire input data DT11.

In other words, DNN is the processing unit in those layer processing, that is, the size (length) of the data area to be processed and the sample advance so that the processing boundaries of adjacent frames match each other in the layer processing of each layer. It is assumed that the number and the frame length of the input data DT11 are defined.

The processing unit here is an area that is targeted when performing one layer processing. For example, in the case of convolution processing, it is an area of size determined by size x number of dimensions, that is, the number of taps of the convolution filter. be.

The fact that the frame processing boundaries match at each layer of DNN means that the frame length of the input data DT11 is an integral multiple of the processing interval that DNN originally has.

For example, in the DNN shown in FIG. 2, the processing interval of the DNN is the product of the number of advances (sample advance number) in the layer processing in all the layers of layers L1 to L9. In this example, the product of the number of forward movements is 1024, and 1 times the product is 1024 samples, which is the frame length of the input data DT11.

Here, the input holding method will be described with reference to FIG.

In FIG. 2, the entire data input to each layer shows a data portion to be substantially processed in the layer to which those data are input.

In the input holding method, a part of the processing result of the past frame, more specifically, the shaded part in each data is held in the memory, so that the amount of data of the part to be processed in the current frame can be suppressed to a small amount. Can be done.

Specifically, in layer L1, the partial data PDT11 of the previous frame in the input data DT11 is held in the memory, and the convolution of the current frame is performed based on the data PDT11 and the data IDT11 of the current frame of the input data DT11. Processing is done.

Here, the data PDT11 is three-dimensional data having a length of 80 samples in the time direction, that is, data having a length of 80 samples from the beginning in the previous frame of the input data DT11. The data ID T11 is a part of the current frame of the input data DT11, that is, data having a length of one frame.

In layer L2, pooling processing for the current frame is performed for the data IDT12 of the current frame of the intermediate data DT12.

In layer L3, a part of the data PDT12 of the previous frame in the intermediate data DT13 is held in the memory, and the convolution process for the current frame is performed based on the data PDT12 and the data IDT13 of the current frame of the intermediate data DT13.

In layer L4, pooling processing for the current frame is performed for the data IDT14 of the current frame of the intermediate data DT14.

In layer L5, a part of the data PDT13 of the previous frame in the intermediate data DT15 is held in the memory, and the convolution processing for the current frame is performed based on the data PDT13 and the data IDT15 of the current frame of the intermediate data DT15.

In layer L6, pooling processing for the current frame is performed for the data IDT16 of the current frame of the intermediate data DT16.

In layer L7, a part of the data PDT14 of the previous frame in the intermediate data DT17 is held in the memory, and the convolution process for the current frame is performed based on the data PDT14 and the data IDT17 of the current frame of the intermediate data DT17.

In layer L8, pooling processing for the current frame is performed for the data IDT18 of the current frame of the intermediate data DT18.

In layer L9, the data PDT15 of the past frame in the intermediate data DT19 is held in the memory, and the convolution processing for the current frame is performed based on the data PDT15 and the data IDT19 of the current frame of the intermediate data DT19.

Therefore, in the input holding method, it is sufficient to have a memory amount sufficient to hold the data PDT11 to the data PDT15 when performing each layer processing of the current frame, and the required memory amount can be significantly reduced.

In the input holding method, by holding the data PDT11 to the data PDT15 in the memory, the identification result data can be obtained by performing layer processing of the current frame only for the data IDT11 to the data IDT19 and the data PDT11 to the data PDT15. You can get DT20. Therefore, the amount of calculation can be significantly reduced as compared with the general method.

The reduction in the amount of memory and the amount of calculation will be explained in more detail.

First, the data PDT15 held in the memory at layer L9 is 20 (size) x 128 (dimension) data.

This data PDT15 performs convolution processing and pooling processing on the part of the length of 22736 samples excluding the part of the current frame (data IDT11) in the part of the length of 23760 samples which is the entire input data DT11. It is the data obtained by performing in order.

If the data IDT19 of 1 (size) × 128 (dimension) adjacent to the future side of the data PDT15 in the intermediate data DT19 can be calculated in the current frame, the data IDT19 and the data PDT15 are combined and convolved in the layer L9. The processing can be performed and the identification result data DT20 can be obtained.

The calculation of the data IDT19 part corresponds to the boundary part between the previous frame and the current frame in each convolution layer, that is, the data of the previous frame, and the data PDT11 to the data PDT14 which are also used for the convolution processing of the current frame should be retained. Can be done.

Therefore, by retaining the data PDT11 to data PDT15 parts of the past frame, it is possible to perform the identification processing (DNN processing) for the current frame, that is, the processing of the entire DNN.

As described above, in the input holding method, the data input from the layer immediately before the past frame, that is, the input data or the intermediate data, which is required for the convolution processing in the current frame in each convolution layer, is held in the memory. ..

At this time, the amount of memory required for the convolution layer can be obtained by the following equation (1).

In the equation (1), the "convolution size" is the number of taps in the time direction of the convolution filter used in the convolution process, that is, the size in the time direction, and the "advance number" is the number of advances in the convolution process (sample advance number). ). Further, in the equation (1), the “number of dimensions of the input data” is the number of dimensions (number of channels) of the input data or the intermediate data input to the convolution layer.

Therefore, in the input holding method shown in FIG. 2, the total amount of memory required for holding the data PDT 11 to the data PDT 15 is 3072 from the equation (1).

However, in the input holding method, the larger the number of dimensions of the input data (input data or intermediate data) in each layer, the larger the amount of memory required to hold the processing result of the previous frame. In particular, in the example of FIG. 2, the memory amount of 2560 (= 20 × 128) is required in the layer L9 which is the final layer.

Therefore, one of the layer processing in the future frame based on a part or all of the data input in the current frame, that is, the part of the data in the current frame in the layer used for the layer processing in the future frame. The result of performing the part may be retained in the memory.

Hereinafter, such a method will also be referred to as an output holding method.

By performing identification processing (DNN processing) with the output holding method, it is possible to reduce the amount of memory required even more than with the input holding method, with the same amount of calculation as the input holding method.

In the output retention method, in each layer, not the data from the previous layer of the input past frame, but a part of the intermediate data output to the next layer, or the data for obtaining the intermediate data is stored in the memory. Be retained.

Here, with reference to FIG. 4, an example in which the output holding method is applied to the DNN having the configuration described with reference to FIG. 2 will be described. In FIG. 4, the same reference numerals are given to the portions corresponding to those in FIG. 2, and the description thereof will be omitted as appropriate.

In the output holding method shown in FIG. 4, basically the same operation (layer processing) as the input holding method is performed in each layer of the DNN.

That is, the DNN consists of nine layers, layer L1 to layer L9, and the same layer processing as in FIG. 2 is performed on those layers.

However, the convolution layers Layer L1, Layer L3, Layer L5, Layer L7, and Layer L9 do not retain the data PDT11 to the data PDT15 as in the input retention method, but are the processing results in those layers. Data PDT21 to data PDT25 are held in memory.

These data PDT21 to data PDT25 perform a part of layer processing in the frame ahead (future) in time from the current frame based on the data of the current frame or the past frame input to the layer. It is the data obtained in.

For example, the data PDT21 uses a part of the convolution process for obtaining the data IDT12 of the current frame, that is, the data PDT11, based on the data PDT11 which is a part of the input data of the previous frame input to the layer L1. It is obtained by performing the calculation part. In other words, PDT21 is a part of the processing result of the convolution processing for the current frame.

In this example, the data PDT21 is 5 (size) x 16 (dimension) data, and the amount of memory required to hold the data PDT21 is 80 (= 5 x 16). Therefore, it can be seen that the amount of memory can be reduced as compared with the amount of memory 240 (80 × 3) required to hold the data PDT11.

Further, in the example of FIG. 4, the data PDT22 is 7 (size) × 32 (dimension) data, the data PDT23 is 3 (size) × 64 (dimension) data, and the data PDT24 is 1 (size) × 128. It is (dimensional) data, and the data PDT25 is 20 (size) × 1 (dimensional) data.

The amount of memory required to hold these data PDT21 to data PDT25 is 644, and it can be seen that the amount of memory required for the input holding method as a whole can be significantly reduced compared to the amount of memory 3072. In particular, in the example of FIG. 4, the effect of reducing the amount of memory is large in the first layer L1 and the final layer 9.

In the output holding method, the amount of memory required for the convolution layer can be obtained by the following equation (2).

In equation (2), "ceil" indicates rounding up, and "convolution size" is the number of taps in the time direction of the convolution filter used in the convolution process, that is, the size in the time direction. Further, the "advance number" is the advance number (sample advance number) at the time of the convolution process, and the "dimension number of output data" is the dimension number of the data obtained by the convolution process.

Comparing the input holding method shown in FIG. 2 with the output holding method shown in FIG. 4, the output holding method can reduce the amount of memory as a whole, but the output holding method may be used depending on the layer. The input holding method may be more effective in reducing the amount of memory than the method.

Therefore, the input holding method and the output holding method may be used in combination. That is, it may be possible to select whether to use the input holding method or the output holding method for each layer.

For example, in the example described with reference to FIGS. 2 and 4, the output holding method may be adopted for the layers L1 and L9, and the input holding method may be adopted for the layers L3, L5, and layer L7.

In such a case, the memory amount for holding the data PDT21, the data PDT12 to the data PDT14, and the data PDT25 is required as a whole, so the memory amount is 372. Therefore, the amount of memory can be reduced to about 1/8 from 3072 to 372 as compared with the case of the input holding method.

Here, a specific example of the output holding method will be described.

First, with reference to FIG. 5, the convolution process in the layer L1 of the output holding method in FIG. 4 will be described. In FIG. 5, the same reference numerals are given to the portions corresponding to those in FIGS. 2 or 4, and the description thereof will be omitted as appropriate.

In layer L1, in the current frame, the data PDT11, which is a part of the 1024 × 3 data input in the previous frame, and the data IDT11 input in the current frame are targeted.

Then, for the target data, a 16-sample forward convolution process is performed for each of the 16 types of convolution filters by a convolution filter of size "96" x dimension "3", and the resulting data IDT12 is obtained. It is output to the next layer L2.

Of such convolution processing of the current frame, the convolution of the part straddling from the previous frame to the current frame, that is, the convolution using the data PDT11 cannot be performed only by the data IDT11 input in the current frame.

Therefore, in the input retention method, the last 80 samples (= size 96-advance number 16) of the input data DT11 for one frame input in the previous frame x 3D data PDT11 itself is retained in the memory. As a result, the amount of calculation and the amount of memory were reduced. In this case, the required amount of memory is 240 (= 80 × 3).

On the other hand, in the output holding method, as shown in Q11, in the previous frame, a part of the convolution processing in the current frame performed using the input data PDT11 is performed, and the processing result is held in the memory M11. I will leave it. That is, at the stage of the previous frame, the convolution filter is applied up to the current frame in the future as seen from the previous frame.

In the part indicated by arrow Q11, one quadrangle represents the part of the input data DT11 used in the convolution process in the current frame, and the shaded part in the quadrangle is the data part (data) of the previous frame used for convolution. Represents part or all of PDT11).

In particular, the quadrangle at the nth position (n = 1,2, ..., 5) from the right in the part indicated by arrow Q11 represents the data used in the nth convolution process performed in the current frame.

In addition, the memory M11 consists of five memory areas arranged in the vertical direction in the figure, and here, the memory area at the nth position (n = 1,2, ..., 5) from the top in the figure is simply n. It will also be referred to as the second memory area.

Here, of the multiple convolution processes performed in the current frame, the data PDT11 is required for the 1st to 5th convolution processes.

For example, in the first convolution process, the entire data PDT11, that is, the part of the length of the last 80 samples of the previous frame is used, and in the second convolution process, the part of the length of the last 64 samples of the data PDT11 is used. Be done. In the fifth convolution process, the length of the last 16 samples of the data PDT11 is used.

Specifically, as part of the nth (n = 1,2, ..., 5) convolution process of the current frame, the length of the last (96-16n) sample of the data PDT11 input in the previous frame. Convolution is performed between the part and the filter coefficient of the part of the head (96-16n) sample of the convolution filter, and the processing result is stored and held in the nth memory area of the memory M11.

Since the convolution process for the length of the end (96-16n) sample of the data PDT11 is performed for each type of convolution filter, the nth memory area of memory M11 is obtained for each of the 16 types of convolution filters. The result of the convolution process is retained.

Therefore, in the output holding method, the amount of memory required to hold the data for the convolution processing for each frame in the layer L1 is 5 (size) × 16 (dimensions), which is the memory amount of the memory M11.

In the nth (n = 1,2, ..., 5) convolution process of the current frame, the data area of 96 (size) x 3 (dimension) is actually targeted, but in the previous frame, that There is no data part of the length of 16n samples input in the current frame in the data area.

Therefore, when performing a part of the nth convolution process of the current frame, which is performed in advance in the previous frame, the convolution calculation for the data part of the length of the 16n sample is not performed, that is, in the previous frame. Convolution processing is performed only for the data part of the length of the input (96-16n) sample.

In addition, when performing a part of the nth convolution process of the current frame, which is performed in advance in the previous frame, the data part of the length of 16n samples input in the current frame is regarded as zero data and the convolution process is performed. You may do so.

As a result, even if there is no data for the last 16n samples of the entire target data in the nth convolution process of the current frame, when a part of the nth convolution process is executed. The processing result can be obtained.

As described above, when the input data DT11 of the previous frame is input, a part of the convolution processing for the first n times of the current frame is performed before the input data DT11 of the current frame is input. , The processing result is held in the memory M11. At this time, the data held in the memory M11 is the data PDT21 shown in FIG.

Then, since the data IDT11 is input in the current frame, the data IDT11 is used as shown by the arrow Q12, and the convolution processing is performed by 16 samples forward for each of the 16 types of convolution filters.

In the part indicated by the arrow Q12, one quadrangle represents the part of the input data DT11 used in the convolution process in the current frame, and the shaded part in the quadrangle is used in the convolution process of the data ID T11 in the current frame. Represents a part.

In particular, the quadrangle at the nth position (n = 1,2, ..., 5) from the right in the part indicated by arrow Q12 represents the data used in the nth convolution process performed in the current frame.

In the current frame, for the first 5 convolution processes, some calculations (processes) have already been performed in the previous frame, and the process results are held in the memory M11.

Therefore, for the n (n = 1,2, ..., 5) convolution processing of the current frame, the part of the length of the first 16n samples of the input data IDT11 and the part of the last 16n samples of the convolution filter. Convolution processing with the filter coefficient of is performed.

Then, the result of the convolution process and the result of the convolution process by the same convolution filter held in the nth memory area of the memory M11 are added, and the addition result is the final nth convolution of the current frame. It is output as the processing result of the processing.

As a result, of the convolution processing in the current frame, the processing result of the convolution processing for the data area portion extending from the previous frame to the current frame can be obtained.

For example, in the first convolution process of the current frame, the convolution process is performed with the length part of the first 16 samples of the data IDT11 and the filter coefficient of the part of the last 16 samples of the convolution filter. Then, the processing result and the result of the convolution processing held in the first memory area of the memory M11 are added and output as data for the first sample of the data ID T12 × 1 dimension.

In the nth convolution process using the data IDT11 of the current frame, the part of the head (96-16n) sample of the target data area is the same as in the convolution process for the current frame of the previous frame. Since it does not exist, the convolution calculation is not performed for that part, or the convolution process is performed with zero data.

Also, in the current frame, for the 6th and subsequent convolution processes, the target data area is all included in the data IDT11, so the convolution process is performed by the convolution filter for the target data area of the data IDT11. Will be. As a result, the data IDT12 is obtained as the final processing result of the convolution processing of the current frame.

Furthermore, in the current frame, as shown in the part of arrow Q13, a part of the convolution process in the next frame of the current frame is performed in the same manner as in the case of the part shown in arrow Q11.

That is, of the multiple convolution processes performed in the next frame, the part that uses the data IDT11 in the first to fifth convolution processes that require the data IDT11 is processed (calculated), and the processing result is Stored and retained in memory M11.

In layer L1, the process described above is repeated frame by frame.

The amount of memory required for layer L1 is 80 (= ceil ((96-16) / 16) × 16) from the above equation (2), which is compared with the amount of memory 240 required for the input holding method. It can be seen that the amount of memory required can be reduced to 1/3.

Next, with reference to FIG. 6, the convolution process in the layer L9 of the output holding method in FIG. 4 will be described. In FIG. 6, the same reference numerals are given to the portions corresponding to those in FIGS. 2 or 4, and the description thereof will be omitted as appropriate.

In layer L9, data IDT19 of 1 (size) x 128 (dimension) is input from the immediately preceding layer L8 in the current frame.

Then, as the convolution processing of the current frame, the convolution processing of one sample forward by the convolution filter of size "21" x dimension "128" is performed by one type of convolution filter, and the identification result data DT20 obtained as a result is output. To.

In this case, the data area to be convolved is the part of the data IDT19 input in the current frame and the past (most recent) 20 frames of the intermediate data DT19 input from the immediately preceding layer L8. It is an area consisting of a part of data PDT15 of 20 × 128.

Of the convolution processing in the current frame, the convolution of the part straddling from the previous frame to the current frame, that is, the convolution using the data PDT15, cannot be performed only by the data IDT19 input in the current frame.

Therefore, in the input holding method, the amount of calculation and the amount of memory are reduced by holding the intermediate data DT19 for the past 20 frames input from layer L8, that is, the data PDT15 itself for 20 samples x 128 dimensions in the memory. Was going. In this case, the required amount of memory is 2560 (= 20 × 128).

On the other hand, in the output retention method, as shown in Q21, a part of the convolution processing in the future frame, which is performed using the data PDT15 consisting of the data for the past 20 frames, is performed, and the processing result is stored in the memory M21. Try to keep it. That is, at the stage of the previous frame, the convolution filter is applied to the frames in the future as seen from the previous frame.

In the part indicated by arrow Q21, one quadrangle represents the part of the intermediate data DT19 used in the convolution process in the frame in the future when viewed from the previous frame.

Also, the shaded area in those rectangles represents the data part (part or all of the data PDT15) of the frame past the current frame used for convolution.

In particular, the quadrangle (partly not shown) at the nth position (n = 1,2, ..., 20) from the right in the part indicated by arrow Q21 is the future when viewed from the previous frame (the frame immediately before the current frame). It represents the data used in the convolution process at the nth frame in the direction.

In addition, the shaded part in those rectangles represents the data part of the frame past the current frame in the intermediate data DT19, and the part without diagonal lines in the rectangle represents the current frame. It represents the subsequent data part, that is, the data part of the future frame when viewed from the previous frame.

For example, the quadrangle at the first (right end) from the right in the part indicated by arrow Q21 represents the data used for the convolution process in the current frame of layer L9, and the unshaded part in this quadrangle is. , Represents the data IDT19 input in the current frame.

Also, for example, the second quadrangle from the right in the part indicated by arrow Q21 represents the data used for the convolution process in the next frame of the current frame of layer L9, and the diagonal line in this quadrangle is not applied. The part represents the data input after the current frame.

The memory M21 consists of 20 memory areas arranged in the vertical direction in the figure. Here, the memory area at the nth position (n = 1,2, ..., 20) from the top in the figure is simply the nth memory area. It will also be referred to as a memory area.

Here, the operating principle of the output holding method will be explained inductively.

Temporarily, in the current frame, the intermediate data DT19 for the past (21-n) samples input to layer L9, that is, the length part for the end (21-n) samples of the data PDT15 and the beginning of the convolution filter (21-). n) It is assumed that the result of convolution with the filter coefficient of the sample portion is held in the nth memory area of the memory M21.

That is, the processing result when a part of the convolution processing in the nth frame in the future direction from the previous frame is performed using the part of the head (21-n) sample of the convolution filter is the memory M21. It is assumed that it is held in the nth memory area.

At this time, it is assumed that in the current frame, the data IDT19 of the length of one sample input in the current frame and the filter coefficient of the part of the last one sample of the convolution filter are convoluted as shown by the arrow Q22.

Then, if the processing result of the convolution for the data IDT19 and the processing result held in the first memory area of the memory M21 are added, the processing result of the convolution processing for the current frame, that is, the total of the intermediate data DT19 is 21. The identification result data DT20, which is the processing result of the convolution process for the data area of the length of the sample, can be obtained.

The quadrangle (partly not shown) at the nth position (n = 1,2, ..., 21) from the right in the part indicated by arrow Q22 is the future when viewed from the previous frame (the frame immediately before the current frame). It represents the part of the intermediate data DT19 used in the convolution process of layer L9 at the nth frame in the direction.

In particular, the shaded areas in those rectangles represent the data IDT19 input in the current frame. The part above the data IDT19 part in the rectangle represents the data part of the past frame, and the part below the data IDT19 part in the rectangle represents the data part of the future frame. ing.

For example, the quadrangle at the first (right end) from the right in the part indicated by arrow Q22 represents the data used for the convolution process in the current frame of layer L9.

In the output retention method, as shown by the square at the right end of the part indicated by arrow Q22, the data IDT19 input in the current frame is stored in the convolution processing result and the first memory area of the memory M21. The processing result is added and output as the identification result data DT20 of the current frame.

If a part of the convolution process is performed using only the filter coefficients of some samples of the convolution filter, the convolution calculation of other parts may not be performed as described above, or The calculation of the convolution is performed with the other parts as zero data.

In addition, in the current frame, in preparation for the convolution processing in the future frame, the convolution processing using the part up to the data IDT19 in each frame is performed in advance for the future 20 frames, and the processing result is stored in the memory M21. Be held.

Specifically, the data IDT19 input in the current frame and n from the end of the convolution filter, as represented by the nth (n = 2, ..., 21) rectangle from the right in the part indicated by arrow Q22. Convolution with the filter coefficient of the sample eye part is performed.

Then, the processing result and the processing result held in the nth memory area of the memory M21 are added, and the addition result is stored and held in the (n-1) th memory area of the memory M21. .. However, in the part indicated by arrow Q22, which is the 21st from the right, that is, the part represented by the rectangle at the left end, the data IDT19 input in the current frame and the first sample at the beginning (21st sample from the end) of the convolution filter. The processing result of convolution with the filter coefficient of the part is stored and held as it is in the 20th memory area of the memory M21. The data held in the memory M21 in this way is the data PDT25 shown in FIG.

As a result, the n (n = 1, ..., 20) th memory area of the memory M21 again contains the part of the length of the past (21-n) sample of the intermediate data DT19 including the sample of the current frame. , The result of convolution with the filter coefficient of the part of the head (21-n) sample of the convolution filter is retained.

Therefore, it can be seen that the layer processing (convolution processing) of the layer L9 can be performed by the output holding method inductively.

In the output retention method, the amount of memory required for the convolution process at layer L9, that is, the size of the memory M21, is 20 (= ceil ((21-1) / 1) × 1) from the above equations (2) to 20 (= ceil ((21-1) / 1) × 1). It can be seen that the amount of memory required for the input holding method can be reduced to 1/128 compared to 2560.

It is possible to generalize the convolution process in the convolution layer such as layer L1 and layer L9 by the output retention method explained above.

For example, the number of samples (size) in the time direction of one frame of data input to a predetermined layer is a, the size of the convolution process, that is, the number of taps in the time direction of the convolution filter is b, and the number of advances in the convolution process. Let be c.

In this case, the output length d of the data output as a result of the convolution process in the predetermined layer, that is, the number of samples in the time direction is d = a / c, and the length of the memory required for the convolution process in the time direction. The memory length e is e = ceil ((bc) / c).

Now, consider the nth convolution process performed in the current frame at a predetermined layer, that is, the n (n = 1,2, ..., a / c + ceil ((b-c) / c) th convolution process.

In this case, the following processing is first performed as processing TR1.

That is, when n≤e, the processing result for the data of the past frame held at the nth position of the memory is added to the result of the convolution processing for the data input in the current frame, and the processing TR1 It is the processing result of.

On the other hand, when n> e, the convolution process is performed on the data input in the current frame, and the process result is the process result of the process TR1.

Next, processing TR2 is performed based on the processing result of processing TR1.

That is, when n ≦ d, the processing result of the processing TR1 is output to the next layer as the data of the part corresponding to the processing result of the nth convolution processing in the processing result of the convolution processing in the current frame.

On the other hand, when n> d, the processing result of processing TR1 is stored and held in the (n-d) th memory area of the memory of the predetermined layer.

By adopting the output holding method as described above, the amount of calculation and the amount of memory can be significantly reduced as compared with the general method, and the amount of memory can be further reduced with the same amount of calculation as compared with the input holding method. be able to.

In the above, an example of applying the output holding method to the convolution layer has been described, but the output holding method is not limited to this and can be applied to other layers as well.

The following is an example of applying the output retention method to the pooling layer.

For example, in the DNN shown in FIG. 2, since the size of the pooling process and the number of advances were the same, it was not necessary to hold the processing results of the past frames in the memory in the pooling layer.

However, as shown in FIG. 7, for example, the size, that is, the number of samples in the time direction of the target data area is 96, and the pooling layer that performs the pooling process of advancing 16 samples retains the processing results of the past frames. You will need it.

In the example shown in FIG. 7, 1024 samples × 3D data IDT41 is input from the immediately preceding layer in the current frame.

Then, the data PDT41 with the length of the last 80 samples of the 1024 samples x 3D data input from the previous layer in the previous frame and the data IDT41 input in the current frame are targeted for pooling processing. Is done.

At this time, the pooling process targeting the 96-sample x 1-dimensional data area is performed by advancing 16 samples for each dimension of the target data, and the processing result of the pooling process in the current frame is 64 samples x 3 dimensions. Data IDT42 is output.

The pooling process referred to here is a process of extracting the maximum value of the sample value of the sample in the target data area and outputting it as a processing result as described above.

When the pooling process for the current frame is performed, the pooling process for the data area including the data PDT41 of the part extending from the previous frame to the current frame cannot be performed only by the data IDT41 input in the current frame. ..

In such a case, in the input retention method, the last 80 samples (= size 96-advance number 16) x 3D data PDT41 itself of the data for one frame input in the previous frame is retained in the memory. By setting it, the amount of calculation and the amount of memory can be reduced as compared with the general method. At this time, the required amount of memory is 240 (= 80 × 3).

On the other hand, in the output retention method, as shown in Q41, in the previous frame, a part of the pooling process in the current frame is performed using the input data PDT41, and the process result is retained in the memory M31. To do so. That is, at the stage of the previous frame, the pooling process is applied to the current frame in the future as seen from the previous frame.

In the part indicated by the arrow Q41, one quadrangle represents the part of the data (data area) to be pooled in the current frame, and the shaded part in the quadrangle is the part of the previous frame used for the pooling process. Represents a data part (part or all of data PDT41).

In particular, the nth (n = 1,2, ..., 5) rectangle from the right in the part indicated by arrow Q41 represents the data area part that is the target of the nth pooling process performed in the current frame.

In addition, the memory M31 consists of five memory areas arranged in the vertical direction in the figure, and here, the memory area at the nth position (n = 1,2, ..., 5) from the top in the figure is simply n. It will also be referred to as the second memory area.

Here, of the multiple pooling processes performed in the current frame, the data PDT41 is required for the 1st to 5th pooling processes.

For example, in the first pooling process, the entire data PDT41, that is, the part of the length of the last 80 samples of the previous frame is used, and in the second pooling process, the part of the data PDT41 of the last 64 samples is used. Be done.

Therefore, as part of the nth (n = 1,2, ..., 5) pooling process of the current frame, for the part of the length of the last (96-16n) sample of the data PDT41 input in the previous frame. Then, a pooling process for extracting the maximum value is performed for each dimension of the data PDT41, and the process result is stored and held in the nth memory area of the memory M31.

Then, since the data IDT41 is input in the current frame, the data IDT41 is used as shown by the arrow Q42, and the pooling process for advancing 16 samples is performed for each dimension.

In the part indicated by arrow Q42, one quadrangle represents the part of the data used in the pooling process in the current frame, and the shaded part in the quadrangle represents the part used in the pooling process of the data IDT41 in the current frame. Represents.

In particular, the nth (n = 1,2, ..., 5) rectangle from the right in the part indicated by arrow Q42 represents the data used in the nth pooling process performed in the current frame.

In the current frame, for the first 5 pooling processes, some calculations (processes) have already been performed in the previous frame, and the process results are held in the memory M31.

Therefore, for the n (n = 1,2, ..., 5)th pooling process of the current frame, only the part of the length of the first 16n samples of the data IDT41 input in the current frame is targeted, and the dimension is The maximum value is extracted for each.

Then, the extracted maximum value and the value (processing result) for each dimension held in the nth memory area of the memory M31 are compared for each dimension, and the larger value is the nth pooling process. It is output as the processing result for each dimension of.

This makes it possible to obtain the processing result of the pooling process for the data area portion extending from the previous frame to the current frame in the current frame.

Also, in the current frame, for the sixth and subsequent pooling processes, the target data area is all included in the data IDT 41, so the pooling process is performed on the target data area of the data ID T41. As a result, the data IDT42 is obtained as the final processing result of the pooling processing of the current frame.

Furthermore, in the current frame, as shown in the part of arrow Q43, a part of the pooling process is performed in the next frame of the current frame in the same manner as in the case of the part shown by arrow Q41.

That is, of the multiple pooling processes performed in the next frame, the process (calculation) of the part using the data IDT41 in the first to fifth pooling processes that require the data IDT41 is performed, and the processing result is the result. It is stored and held in memory M31.

In the pooling layer, the process described above is repeated frame by frame.

For the amount of memory required for such a pooling layer, the "convolution size" in the above equation (2) is replaced with a "pooling size" indicating the length (size) of the data area to be pooled in the time direction. It can be obtained by.

Therefore, in this example, the amount of memory required for the pooling layer is 15 (= (ceil ((96-16) / 16) × 3), which is compared with the amount of memory required for the input holding method of 240. It can be seen that the amount of memory required can be significantly reduced. Moreover, the amount of computation in the pooling layer in the output holding method is the same as in the case of the input holding method.

As described above, even if the output holding method is applied to the pooling layer, the amount of computation and the amount of memory can be reduced as in the case of the convolution layer.

<Configuration example of signal processing device>
FIG. 8 is a diagram showing a configuration example of an embodiment of a signal processing device to which the present technology is applied.

The signal processing device 11 shown in FIG. 8 is composed of, for example, a personal computer, a smart phone, headphones, a portable player, or the like, performs predetermined identification processing based on audio data supplied as input data, and is identified to show the identification result. Output the result data.

In this example, the signal processing device 11 performs an identification process consisting of a plurality of layer processes on a frame-by-frame basis for the input data, and outputs the identification result data.

The signal processing device 11 has a layer processing unit 21-1 to a layer processing unit 21-9, and a control unit 22.

In the signal processing device 11, a DNN, that is, a neural network is configured by layer processing units 21-1 to layer processing units 21-9 provided hierarchically. In other words, the layer processing unit 21-1 to the layer processing unit 21-9 realize a discriminator having a neural network configuration.

Here, the layer processing units 21-1 to the layer processing units 21-9 correspond to the layers L1 to L9 constituting the DNN shown in FIG. 2 or FIG. 4, and the arithmetic processing (layer processing) in those layers. It functions as an arithmetic processing unit that performs the above.

Further, each of the layer processing unit 21-1, the layer processing unit 21-3, the layer processing unit 21-5, the layer processing unit 21-7, and the layer processing unit 21-9 is a memory 31-1 to a memory 31-. It has each of five.

Hereinafter, when it is not necessary to particularly distinguish between the layer processing unit 21-1 and the layer processing unit 21-9, it is also simply referred to as the layer processing unit 21. Further, hereinafter, when it is not necessary to distinguish between the memory 31-1 and the memory 31-5, it is also simply referred to as the memory 31. Each memory 31 holds data required for convolution processing in the input holding method or the output holding method.

The layer processing unit 21-1 performs convolution processing by an input holding method or an output holding method based on the input data and the data held in the memory 31-1, and the intermediate data obtained as a result is used. It is supplied to the layer processing unit 21-2.

That is, in the layer processing unit 21-1, the convolution process in the current frame is performed by the input holding method or the output holding method based on the input data input in the current frame and the data held in the memory 31-1. Intermediate data that is the processing result of is obtained.

The layer processing unit 21-2 performs pooling processing on the intermediate data supplied from the layer processing unit 21-1, and supplies the intermediate data obtained as a result to the layer processing unit 21-3.

The layer processing unit 21-3 performs convolution processing by an input holding method or an output holding method based on the intermediate data supplied from the layer processing unit 21-2 and the data held in the memory 31-2, and the convolution processing thereof is performed. The resulting intermediate data is supplied to the layer processing unit 21-4.

The layer processing unit 21-4 performs pooling processing on the intermediate data supplied from the layer processing unit 21-3, and supplies the intermediate data obtained as a result to the layer processing unit 21-5.

The layer processing unit 21-5 performs convolution processing by an input holding method or an output holding method based on the intermediate data supplied from the layer processing unit 21-4 and the data held in the memory 31-3, and the convolution processing thereof is performed. The resulting intermediate data is supplied to the layer processing unit 21-6.

The layer processing unit 21-6 performs pooling processing on the intermediate data supplied from the layer processing unit 21-5, and supplies the intermediate data obtained as a result to the layer processing unit 21-7.

The layer processing unit 21-7 performs convolution processing by an input holding method or an output holding method based on the intermediate data supplied from the layer processing unit 21-6 and the data held in the memory 31-4, and the convolution processing thereof is performed. The resulting intermediate data is supplied to the layer processing unit 21-8.

The layer processing unit 21-8 performs pooling processing on the intermediate data supplied from the layer processing unit 21-7, and supplies the intermediate data obtained as a result to the layer processing unit 21-9.

The layer processing unit 21-9 performs convolution processing by an input holding method or an output holding method based on the intermediate data supplied from the layer processing unit 21-8 and the data held in the memory 31-5, and the convolution processing thereof is performed. The identification result data obtained as a result is output as the processing result of the identification process.

As described above, each layer processing unit 21 processes the input data in frame units. That is, each layer processing unit 21 performs data input, layer processing, and data output for each frame. Therefore, the layer processing unit 21-9 outputs the identification result data for each frame.

The control unit 22 controls the processing of the entire signal processing device 11.

For example, the control unit 22 selects, for each layer processing unit 21 of the convolution layer, whether to perform the convolution processing by the input holding method or the output holding method in the layer processing unit 21, and each layer processing unit 21 is used. Control to execute the convolution process by the selected method.

In other words, the control unit 22 switches between the convolution processing by the input holding method and the convolution processing by the output holding method in each layer processing unit 21.

Here, which method is used by each layer processing unit 21 to perform the convolution processing is, for example, the number of dimensions of each data input / output in the user input (specified by the user), the convolution size, and the advancement of the convolution processing. It is selected based on the number (number of sample advances) and so on.

In the following, of the five layer processing units 21 that perform convolution processing, at least one layer processing unit 21 performs convolution processing by the input holding method, and at least one layer processing unit 21 performs convolution processing by the output holding method. An example in which is performed will be described. However, the present invention is not limited to this, and for example, the convolution processing may be performed by the output holding method in the layer processing unit 21 of all the convolution layers.

Further, here, an example in which the memory is provided only in the layer processing unit 21 of the convolution layer will be described. However, for example, a memory may be provided in the layer processing unit 21 of the pooling layer, and the layer processing unit 21 may perform the pooling processing by the input holding method or the output holding output according to the selection by the control unit 22.

Further, whether the layer processing is performed by the input holding method or the output holding method in each layer processing unit 21 is dynamically selected by the control unit 22 in frame units, arbitrary timing, etc., and can be appropriately switched. It may be selected by the control unit 22 at the start of the identification process, or it may be predetermined.

In addition, the configuration of the DNN realized by the signal processing device 11 is not limited to the example shown in FIG. 8, and may be any other configuration.

For example, in the example of FIG. 8, in the layer constituting the DNN, that is, the layer processing unit 21, a convolution process or a pooling process is performed as a layer process, but the process is not limited to this, and any arithmetic process such as a register dual process is performed. You may be broken. That is, for example, in DNN, at least one of the convolution process, the pooling process, and the resilient process may be performed.

Residual processing is data conversion processing that adds the output of the layer before the current layer to the data input of the current layer to obtain the output of the current layer.

<Explanation of identification process>
Next, the operation of the signal processing device 11 will be described. That is, the identification process performed by the signal processing device 11 will be described below with reference to the flowchart of FIG. This identification process is performed for each frame of the input data.

In step S11, the control unit 22 relates to each layer processing unit 21 of the layer processing unit 21-1, the layer processing unit 21-3, the layer processing unit 21-5, the layer processing unit 21-7, and the layer processing unit 21-9. , Select which method to perform the convolution process.

For example, the control unit 22 has the number of dimensions of the data (input data or intermediate data) input to the layer processing unit 21 and the data (intermediate data or identification result) output from the layer processing unit 21 for each layer processing unit 21. The selection is made based on the number of dimensions of the data), the size of the convolution process, the number of sample advances in the convolution process, and the like.

Specifically, for example, the control unit 22 calculates equations (1) and (2) for each layer processing unit 21, and the amount of memory required for the input holding method and the amount of memory required for the output holding method. And select the method that requires less (smaller) memory.

When the control unit 22 selects either the input holding method or the output holding method for each layer processing unit 21, the control unit 22 instructs each layer processing unit 21 to execute the convolution processing by the selected method.

Further, when an input (instruction) is made for each layer processing unit 21 as to which method is to be selected by a user's input operation or the like, the control unit 22 follows the input and causes the input holding method in each layer processing unit 21. Select which method of the convolution processing is to be performed.

For example, when the identification process is actually executed by the hardware that actually functions as the signal processing device 11, the processing efficiency is basically higher if the method with a small amount of memory obtained by the equations (1) and (2) is adopted. That is, the amount of memory used and the amount of calculation are reduced. However, this may not always be the case, and in such cases, it is effective to select either the input holding method or the output holding method according to the user's input operation or the like.

In the following, the control unit 22 selects an output holding method for the layer processing unit 21-1 and the layer processing unit 21-9, and the layer processing unit 21-3, the layer processing unit 21-5, and the layer processing unit 21 are selected. In -7, the explanation will be continued assuming that the input holding method is selected.

In step S12, the layer processing unit 21-1 performs the convolution processing by the output holding method according to the control (instruction) of the control unit 22, and supplies the intermediate data obtained as a result to the layer processing unit 21-2.

For example, in step S12, as described with reference to FIG. 5, for each of the 16 types of convolution filters of 96 × 3 taps, the convolution process at the current frame is performed by advancing 16 samples.

That is, the layer processing unit 21-1 performs a convolution process on the input data of the input current frame using a part of the convolution filter for the first five convolution processes, and the process result and the memory. The result of the convolution process for the input data of the previous frame held in 31-1 is added and output as the final process result of the convolution process.

For the sixth and subsequent convolution processes, the layer processing unit 21-1 performs the convolution process using the convolution filter on the input data of the input current frame.

Further, after that, for example, the layer processing unit 21-1 performs a part of the convolution processing of the next frame (future frame) based on a part of the input data and a part of the convolution filter, and the processing result. Is supplied to the memory 31-1 and held.

In step S13, the layer processing unit 21-2 performs an 8-sample advance pooling process targeting an 8-sample × 1-dimensional data area on the intermediate data supplied from the layer processing unit 21-1, and as a result. The obtained intermediate data is supplied to the layer processing unit 21-3.

In step S14, the layer processing unit 21-3 performs the convolution processing by the input holding method according to the control of the control unit 22, and supplies the intermediate data obtained as a result to the layer processing unit 21-4.

For example, in step S14, as described with reference to FIG. 2, for each of the 32 types of convolution filters of 8 × 16 taps, the convolution process in the current frame is performed by advancing one sample.

At this time, the layer processing unit 21-3 is supplied from the intermediate data supplied from the layer processing unit 21-2 in the current frame and from the layer processing unit 21-2 in the previous frame, and is held in the memory 31-2. Convolution processing is performed using 7 × 16 intermediate data.

Further, when the convolution processing of the current frame is completed, the layer processing unit 21-3 stores the data of the length of the last 7 samples of the intermediate data supplied from the layer processing unit 21-2 in the current frame in the memory 31-2. Supply and hold. The data held in the memory 31-2 is used for the processing of step S14 of the next frame.

In step S15, the layer processing unit 21-4 performs a pooling process of advancing two samples targeting a two-sample × one-dimensional data area on the intermediate data supplied from the layer processing unit 21-3, and as a result. The obtained intermediate data is supplied to the layer processing unit 21-5.

In step S16, the layer processing unit 21-5 performs the convolution processing by the input holding method according to the control of the control unit 22, and supplies the intermediate data obtained as a result to the layer processing unit 21-6.

For example, in step S16, the same processing as in the case of step S14 is performed. However, in step S16, the convolution process in the current frame is performed by advancing one sample for each of the 64 types of convolution filters of 4 × 32 taps.

Further, when the convolution processing of the current frame is completed, the layer processing unit 21-5 stores the data of the length of the last three samples of the intermediate data supplied from the layer processing unit 21-4 in the current frame in the memory 31-3. Supply and hold. The data held in the memory 31-3 is used for the processing of step S16 of the next frame.

In step S17, the layer processing unit 21-6 performs a pooling process of advancing two samples targeting a two-sample × one-dimensional data area on the intermediate data supplied from the layer processing unit 21-5, and as a result. The obtained intermediate data is supplied to the layer processing unit 21-7.

In step S18, the layer processing unit 21-7 performs a convolution process by an input holding method according to the control of the control unit 22, and supplies the intermediate data obtained as a result to the layer processing unit 21-8.

For example, in step S18, the same processing as in the case of step S14 is performed. However, in step S18, the convolution process in the current frame is performed by advancing one sample for each of the 128 types of convolution filters of 2 × 64 taps.

Further, when the convolution processing of the current frame is completed, the layer processing unit 21-7 stores the data of the length of the last one sample of the intermediate data supplied from the layer processing unit 21-6 in the current frame in the memory 31-4. Supply and hold. The data held in the memory 31-4 is used for the processing of step S18 of the next frame.

In step S19, the layer processing unit 21-8 performs a pooling process of advancing two samples targeting a two-sample × one-dimensional data area on the intermediate data supplied from the layer processing unit 21-7, and as a result. The obtained intermediate data is supplied to the layer processing unit 21-9.

In step S20, the layer processing unit 21-9 performs the convolution processing by the output holding method according to the control of the control unit 22, outputs the identification result data obtained as a result, and the identification processing ends.

For example, in step S20, as described with reference to FIG. 6, for one type of convolution filter of 21 × 128 taps, the convolution process at the current frame is performed by advancing one sample.

That is, the layer processing unit 21-9 performs a convolution process on the input data of the input current frame using a part of the convolution filter, and the processing result and the first memory area of the memory 31-5 are displayed. The result of the convolution process for the intermediate data of the past frames held is added to obtain the identification result data.

Further, the layer processing unit 21-9 performs convolution processing based on the filter coefficient of the last n (n = 2, ..., 21) sample of the convolution filter on the intermediate data input in the current frame.

Further, the layer processing unit 21-9 adds the result of the convolution process and the process result held in the n (n = 2, ..., 20) th memory area of the memory 31-5, and the addition result. Is newly stored and held in the (n-1) th memory area of the memory 31-5. Further, the layer processing unit 21-9 stores and holds the result of the convolution processing based on the filter coefficient of the last 21 samples of the convolution filter for the above-mentioned intermediate data in the 20th memory area of the memory 31-5. Let me. The data held in the memory 31-5 in this way is used for the processing of step S20 performed for the future frame.

As described above, the signal processing device 11 realizes the identification processing by selectively using the input holding method and the output holding method in combination and performing the layer processing in each layer (layer) constituting the DNN. do. By doing so, it is possible to reduce the amount of calculation and the amount of memory required.

Note that the processing by DNN to which this technology is applied (DNN processing) may be for detecting a specific sound such as a human voice or a running sound of a car, or is for performing voice recognition. You may. That is, the DNN process is not limited to the above-mentioned identification process, and may be any process such as various detection processes.

In addition, the time-series data (input data) that is the input of the DNN to which this technology is applied is not limited to audio data, but is not limited to audio data, for example, output data (sensor data) of sensors such as acceleration sensors and brain wave sensors, video data, and time-series. Any kind of data such as feature amount data may be used.

For example, the time-series feature amount data extracted from the audio data is used as the data of the first dimension (channel), and the time-series feature amount data extracted from the video data is used as the data of the second dimension. The time series data of may be input to the DNN as input data.

In addition, in a DNN to which this technology is applied, for example, input data consisting of audio data and video data is input, and layer processing for audio data and layer processing for video data are performed in parallel up to a predetermined layer of the DNN. You may be asked. In such a case, for example, in a certain layer, the intermediate data obtained based on the audio data and the intermediate data obtained from the video data are integrated and used as an input for one layer processing.

The DNN to which this technology is applied is not limited to those having a plurality of layers, and may be those having only one layer. Further, this technique is not limited to DNN, and can be applied to a device that performs various arithmetic processes such as convolution processing in one or a plurality of layers (layers) and outputs a final arithmetic processing result.

For example, even when the input data of 16 channels is input, layer processing (arithmetic processing) such as convolution processing is performed in one layer, and the data of 1 channel obtained as a result of the processing is used as the output of DNN. The technology is applicable.

The above technology can be applied to devices that perform various real-time time-series signal processing such as voice recognition.

<Computer configuration example>
By the way, the series of processes described above can be executed by hardware or software. When a series of processes is executed by software, the programs constituting the software are installed on the computer. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 10 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.

In a computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are connected to each other by a bus 504.

An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

The input unit 506 includes a keyboard, a mouse, a microphone, an image pickup device, and the like. The output unit 507 includes a display, a speaker, and the like. The recording unit 508 includes a hard disk, a non-volatile memory, and the like. The communication unit 509 includes a network interface and the like. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 501 loads the program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the above-mentioned series. Is processed.

The program executed by the computer (CPU501) can be recorded and provided on a removable recording medium 511 as a package medium or the like, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. Further, the program can be received by the communication unit 509 and installed in the recording unit 508 via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM 502 or the recording unit 508.

The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in the present specification, in parallel, or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

Further, the embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

For example, this technology can take a cloud computing configuration in which one function is shared by multiple devices via a network and processed jointly.

In addition, each step described in the above flowchart can be executed by one device or shared by a plurality of devices.

Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

Furthermore, this technology can also have the following configurations.

(1)
A signal processing device that performs processing consisting of one or more arithmetic processing on an input data in frame units.
An arithmetic processing unit that performs a part of the arithmetic processing of a frame in the future from the predetermined frame based on a part or all of the input data of the predetermined frame and holds the processing result of the partial processing. A signal processing device.
(2)
The arithmetic processing unit obtains the processing result of the arithmetic processing of the future frame based on the processing result of the partial processing performed in the predetermined frame and the data of the future frame (1). ). The signal processing device.
(3)
It has a plurality of the arithmetic processing units, and has a plurality of the arithmetic processing units.
At least one of the plurality of arithmetic processing units performs the partial processing in the predetermined frame, and holds the processing result of the partial processing.
At least one of the plurality of arithmetic processing units has the arithmetic processing unit holding a part or all of the data in the predetermined frame, and a part or all of the data and the data in the future frame. The signal processing apparatus according to (1) or (2), which performs the arithmetic processing of the future frame based on the above.
(4)
The partial processing is performed in the predetermined frame and the processing result of the partial processing is retained, or a part or all of the data in the predetermined frame is retained, and a part or all of the data and the said. The signal processing apparatus according to any one of (1) to (3), further comprising a control unit for switching whether to perform the arithmetic processing of the future frame based on the data of the future frame. ..
(5)
The signal processing device according to (4), wherein the control unit performs the switching for each arithmetic processing unit.
(6)
The signal processing apparatus according to any one of (3) to (5), further comprising a memory for holding a processing result of the partial processing in the predetermined frame or a part or all of the data in the predetermined frame. ..
(7)
The signal processing apparatus according to any one of (1) to (6), wherein the one or more arithmetic processes include at least one of a convolution process, a pooling process, and a resilient process.
(8)
When the arithmetic processing of the future frame is a convolution processing targeting a data area including a part or all of the data of the predetermined frame and at least a part of the data of the future frame.
The arithmetic processing unit either performs a convolution process targeting only a part or all of the data of the predetermined frame in the data area as the partial processing, or the data of the future frame in the data area. The signal processing apparatus according to any one of (1) to (7), wherein the convolution process in which the portion of is zero data is performed as the partial process.
(9)
The signal processing apparatus according to any one of (1) to (8), which has a plurality of the arithmetic processing units and has a neural network configured by the plurality of arithmetic processing units.
(10)
The number of forward movements in the arithmetic processing, the number of taps of the filter consisting of the filter coefficient used in the arithmetic processing, the size of the data area targeted in the arithmetic processing, or the frame length of the input data is the data of the input data. The signal processing apparatus according to any one of (1) to (9), which is determined based on the shape.
(11)
The signal processing apparatus according to (10), wherein the data shape of the input data is determined by the number of dimensions of the input data and the number of data in each dimension.
(12)
The signal processing device according to (11), wherein when the input data is audio data, the dimension number is the number of channels of the audio data.
(13)
The signal processing device according to any one of (1) to (12), wherein the input data is data including at least one of audio data, sensor data, and video data.
(14)
A signal processing device that performs processing consisting of one or more arithmetic processing on an input data in frame units
A signal processing method that performs a part of the arithmetic processing of a frame in the future from the predetermined frame based on a part or all of the input data of the predetermined frame and retains the processing result of the part of the processing. ..
(15)
For a computer that controls a signal processing device that performs processing consisting of one or more arithmetic processing on a frame-by-frame basis for input data.
Includes a step of performing a part of the arithmetic processing of a frame after the predetermined frame based on a part or all of the input data of the predetermined frame and holding the processing result of the part of the processing. A program that executes processing.

11 Signal processing device, 21-1 to 21-9, 21 Layer processing unit, 22 Control unit, 31-1 to 31-5, 31 Memory

Claims (15)

A signal processing device that performs processing consisting of one or more arithmetic processing on an input data in frame units.
An arithmetic processing unit that performs a part of the arithmetic processing of a frame in the future from the predetermined frame based on a part or all of the input data of the predetermined frame and holds the processing result of the partial processing. A signal processing device. A claim that the arithmetic processing unit obtains a processing result of the arithmetic processing of the future frame based on the processing result of the partial processing performed in the predetermined frame and the data of the future frame. The signal processing apparatus according to 1. It has a plurality of the arithmetic processing units, and has a plurality of the arithmetic processing units.
At least one of the plurality of arithmetic processing units performs the partial processing in the predetermined frame, and holds the processing result of the partial processing.
At least one of the plurality of arithmetic processing units has the arithmetic processing unit holding a part or all of the data in the predetermined frame, and a part or all of the data and the data in the future frame. The signal processing apparatus according to claim 1, wherein the arithmetic processing of the future frame is performed based on the above. The partial processing is performed in the predetermined frame and the processing result of the partial processing is retained, or a part or all of the data in the predetermined frame is retained, and a part or all of the data and the said. The signal processing apparatus according to claim 1, further comprising a control unit for switching whether to perform the arithmetic processing of the future frame based on the data of the future frame. The signal processing device according to claim 4, wherein the control unit performs the switching for each arithmetic processing unit. The signal processing apparatus according to claim 3, further comprising a memory for holding a processing result of the partial processing in the predetermined frame or a part or all of the data in the predetermined frame. The signal processing apparatus according to claim 1, wherein the one or more arithmetic processes include at least one of a convolution process, a pooling process, and a resilient process. When the arithmetic processing of the future frame is a convolution processing targeting a data area including a part or all of the data of the predetermined frame and at least a part of the data of the future frame.
The arithmetic processing unit either performs a convolution process targeting only a part or all of the data of the predetermined frame in the data area as the partial processing, or the data of the future frame in the data area. The signal processing apparatus according to claim 1, wherein the convolution processing in which the portion of is zero data is performed as the partial processing. The signal processing apparatus according to claim 1, which has a plurality of the arithmetic processing units, and the neural network is configured by the plurality of arithmetic processing units. The number of forward movements in the arithmetic processing, the number of taps of the filter consisting of the filter coefficient used in the arithmetic processing, the size of the data area targeted in the arithmetic processing, or the frame length of the input data is the data of the input data. The signal processing device according to claim 1, which is determined based on the shape. The signal processing device according to claim 10, wherein the data shape of the input data is determined by the number of dimensions of the input data and the number of data in each dimension. The signal processing device according to claim 11, wherein when the input data is audio data, the number of dimensions is the number of channels of the audio data. The signal processing device according to claim 1, wherein the input data is data including at least one of audio data, sensor data, and video data. A signal processing device that performs processing consisting of one or more arithmetic processing on an input data in frame units
A signal processing method that performs a part of the arithmetic processing of a frame in the future from the predetermined frame based on a part or all of the input data of the predetermined frame and retains the processing result of the part of the processing. .. For a computer that controls a signal processing device that performs processing consisting of one or more arithmetic processing on a frame-by-frame basis for input data.
Including a step of performing a part of the arithmetic processing of a frame after the predetermined frame based on a part or all of the input data of the predetermined frame and holding the processing result of the part of the processing. A program that executes processing.

Images