JP6951913B2 - Classification model generator, image data classification device and their programs - Google Patents

Description

The present invention relates to a classification model generator that generates a classification model composed of a convolutional neural network for classifying image data, an image data classification device that classifies image data by the classification model, and a program thereof.

Conventionally, as a method for classifying image data, a method using a convolutional neural network (CNN) has been used (Non-Patent Documents 1, 2, etc.).
Here, an outline of an example of CNN will be described with reference to FIGS. 12 and 13. As shown in FIG. 12, the CNN is composed of an input layer I, a hidden layer H, and an output layer O. Each layer has a structure in which a plurality of nodes (units) are connected by edges. In FIG. 12, in order to simplify the explanation of CNN, the number of each layer is reduced and the size of the input image is reduced.

The input layer I is a layer for inputting image data (input image) to be classified.
The hidden layer H is provided via a plurality of convolution layers C (C ₁ , C ₂ , ...), a pooling layer P (P ₁ , P ₂ , ...), And a fully connected layer F (F ₁ , F ₂ , ...). This is a layer for extracting features (feature maps) from the input image. The hidden layer H is not limited to the configuration shown in FIG. 12, such as providing a convolution layer C continuously or providing a normalized layer.

The convolution layer C performs an image convolution calculation on the input image or the feature map that is the output of the previous layer by a plurality of convolution filters. In FIG. 12, for example, in the convolution layer C _{1 , a feature map M 1} (4 @ 20) of four 20 × 20 pixels is performed by performing a convolution operation on an input image of 24 × 24 pixels by four convolution filters. An example of generating × 20) is shown.

As shown in FIG. 13, the convolution layer C sequentially refers to the image of the previous layer (Lth layer) corresponding to the size of the convolution filter Cf (here, 3 × 3 pixels). Is moved to perform a convolution process, and an operation is performed by the activation function f (for example, the normalized linear function max (0, x)) to obtain the pixel value of the next layer (third (L + 1) layer). Here, an example is shown in which the convolution filters Cf are set to four and the feature maps of the four (L + 1) layers are generated from the image of the L layer.

The pooling layer P subsamples the feature map M generated by the convolution layer C. In FIG. 12, for example, in the pooling layer _{P 1,} with respect to the four 20 × 20 image feature map _M 1 (4 @ 20 × 20), by performing half the subsampling horizontally vertically, respectively, 4 _{An example of generating a feature map M 2} (4 @ 10 × 10) of two 10 × 10 images is shown.

The fully connected layer F is a multi-layer perceptron having a feature map generated through a plurality of convolution layers C and a pooling layer P as a one-dimensional vector. This fully connected layer F is composed of a plurality of layers (F ₁ , F ₂ , ...), And the nodes of each layer are all connected to the nodes of the next layer.
The output layer O is a layer that outputs the classification result of the input image as a probability value. This output layer O has the same number of nodes as the classification target to which all the outputs of the fully connected layer F are connected, and outputs the probability value for each node by the activation function (for example, the Sofmax function).
This CNN learns the parameters (network) of each layer from a plurality of image data whose classification is known in the learning stage, and classifies the image data whose classification is unknown by the learned parameters in the classification stage.

Quoc V Le, “Building high-level features using large scale unsupervised learning”, ICASSP, 2013 Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, “ImageNet classification with deep convolutional neural networks”, NIPS, 2012

The CNN described above performs a convolution process while moving the convolution filter in order to extract a feature amount from the image data. This convolution filter does not depend on the content of the image data, and the direction of the filter is always constant.
For example, as shown in FIG. 14, when the same object (for example, an image of “house”) is tilted in different image data, in FIGS. 14 (a), (b), and (c), When features are extracted by performing convolution processing with the convolution filter Cf in the same area of the object (image area of the "chimney part"), different features are extracted even in the same area of the same object. Become.
Therefore, the conventional CNN classifies as image data including different objects by tilting the objects even if the same objects are included in the image data of FIGS. 14 (a), (b), and (c). Will end up.

In order to recognize the objects in these image data as the same object, it is necessary to learn CNN by using the image data in which the objects are tilted in various directions as learning data.
As described above, in the conventional image data classification method using CNN, it is necessary to prepare image data including objects in various directions as learning data, and the amount of training data and the time required for learning become enormous. There is a problem that it will end up.

Therefore, in the present invention, it is possible to classify the image data by recognizing it as the same object regardless of the orientation of the objects in the image data only by learning the CNN (classification model) from the image data of the objects in one direction. An object of the present invention is to provide a classification model generator that generates a possible classification model, an image data classification device that classifies image data using the classification model, and a program thereof.

In order to solve the above problems, the classification model generator according to the present invention generates a classification model which is a convolutional neural network for classifying image data whose classification is unknown from a plurality of image data whose classification is known. The generator is configured to include a region-specific main direction estimation means and a classification model learning means.

In such a configuration, the classification model generator uses the region-specific main direction estimation means to apply the convolution filter of the first convolutional layer of the convolutional neural network from the image data whose classification is known, for each filter region of the image. Estimate the main direction (main direction). The main direction of the edge component can be estimated by using a Sobel filter or the like.

Then, the classification model generation device learns the convolutional neural network from the image data whose classification is known and the teacher data indicating the classification contents by the classification model learning means, and generates the classification model. At this time, in the classification model learning means, in the first convolution layer, the orientation of the predetermined reference of the filter region is constant with respect to the main direction of the edge component estimated by the region-specific main direction estimation means for each filter region. The filter area is rotated so as to be in the direction, and a convolution filter, which is a spatial filter, is applied to the rotated area to perform a convolution operation.
As a result, the classification model learning means can extract features that are almost invariant to the orientation of the objects in the image data in the first convolutional layer.
The classification model generation device can be operated by a classification model generation program for operating the computer as each of the above-mentioned means.

Further, in order to solve the above problems, the image data classification device according to the present invention is an image data classification device that classifies image data whose classification is unknown by using a classification model which is a convolutional neural network generated by the model generation device. Therefore, the configuration is provided with a region-specific main direction estimation means and a classification means.

In such a configuration, the image data classification device uses the region-specific main direction estimation means to apply the convolution filter of the first convolutional layer of the convolutional neural network from the image data whose classification is unknown, to the edge component of the image for each filter region. Estimate the main direction. The main direction of the edge component can be estimated by using a Sobel filter or the like.

Then, the image data classification device classifies the image data whose classification is unknown by the classification means and the convolutional neural network which is a classification model. At this time, in the first convolution layer, the classification means sets the orientation of the predetermined reference of the filter region as a constant direction with respect to the main direction of the edge component estimated by the region-specific main direction estimation means for each filter region. The filter area is rotated so as to be, and a convolution filter, which is a spatial filter, is applied to the rotated area to perform a convolution operation.
As a result, the classification means can extract features that are almost invariant to the orientation of the objects in the image data in the first convolution layer.
The image data classification device can be operated by an image data classification program for operating the computer as each of the above-mentioned means.

Further, in order to solve the above problems, the image data classification device according to the present invention generates a classification model which is a convolutional neural network for classifying image data whose classification is unknown from a plurality of image data whose classification is known. , An image data classification device for classifying image data whose classification is unknown, and may be configured to include a region-specific main direction estimation means, a classification model learning means, and a classification means.

The present invention has the following excellent effects.
According to the present invention, in the first convolution layer, the orientation of a predetermined reference of the filter region of the convolution filter is rotated so as to be a constant direction with respect to the main direction of the edge component of the image corresponding to the filter region. By performing the convolution operation, it is possible to extract a feature amount that is almost invariant with respect to the orientation of the object in the image data.
Thereby, in the present invention, the classification model may be learned by using the image data of the object in one direction as the learning data, and the amount of the learning data and the time required for the learning can be suppressed. Further, the present invention can classify the image data by recognizing the objects in the image data as the same object regardless of whether or not the objects are tilted.

It is a block block diagram which shows the structure of the image data classification apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the size and the movement amount of the filter area of the convolution filter applied in the convolution layer. It is a figure which shows the example of a sobel filter, (a) shows a vertical sovel filter, (b) shows a horizontal sovel filter. It is explanatory drawing for demonstrating the method of finding the main direction of an edge component, (a) is the figure which represented the gradient intensity and the gradient direction of the edge component of a filter region by a vector for each pixel, (b) is the gradient direction. Is a histogram of the cumulative gradient intensity. It is explanatory drawing for demonstrating the rotation direction of a filter area, (a) is a figure which shows the relationship between the reference direction of a filter area and the main direction of an edge component, (b) is a figure which rotated the filter area, (C) is a figure which shows the corresponding pixel of the convolution filter applied to the rotated filter area. It is explanatory drawing for demonstrating an example which performs a convolution process by moving a filter area while rotating a filter area. It is a figure for demonstrating the filter area of the first convolution layer of this invention, and (a)-(c) explain the example in which the feature quantity in substantially the same direction is extracted in the same area of the same object. It is explanatory drawing for this. It is a flowchart which shows the operation of the learning mode of the image data classification apparatus which concerns on embodiment of this invention. It is a flowchart which shows the operation of the classification mode of the image data classification apparatus which concerns on embodiment of this invention. It is a block block diagram which shows the structure of the classification model generation apparatus which concerns on other embodiment of this invention. It is a block block diagram which shows the structure of the image data classification apparatus which concerns on other embodiment of this invention. It is a network diagram which shows an example of the structure of a convolutional neural network. It is explanatory drawing for demonstrating the processing of the convolutional layer of a convolutional neural network. It is a figure for demonstrating the area of the conventional convolution filter, and (a)-(c) are the explanation for demonstrating the example in which the feature amount of a different direction is extracted in the same area of the same object. It is a figure.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Configuration of image data classification device>
First, the configuration of the image data classification device 1 according to the embodiment of the present invention will be described with reference to FIG.
The image data classification device 1 learns a convolutional neural network (CNN; hereinafter referred to as a classification model) for classifying image data by objects in the image data, and classifies the image data using the classification model. Is. The image data classification device 1 has two different operation modes: a mode for learning a classification model (hereinafter, referred to as “learning mode”) and a mode for classifying image data (hereinafter, referred to as “classification mode”).

In the learning mode, the image data classification device 1 learns the classification model by inputting a plurality of image data whose classification is known and teacher data indicating the classification contents as learning data. Here, the teacher data is, for example, information that uniquely identifies each person (for example, a person's name, etc.) if the classification target is a person.
In the classification mode, the image data classification device 1 inputs image data whose classification is unknown and outputs a result (classification result) of classification using the classification model.
Hereinafter, the configuration of the image data classification device 1 that operates in these two operation modes will be described in detail.

The image data classification device 1 includes a learning data input means 10, a classification data input means 11, a region-specific main direction estimation means 12, a region-specific main direction storage means 13, a classification model learning means 14, and a classification model. A storage means 15 and a classification means 16 are provided.

The learning data input means 10 inputs image data whose classification is known and teacher data indicating the classification contents as learning data. The learning data input means 10 outputs the input image data to the area-specific main direction estimation means 12 and the classification model learning means 14. Further, the learning data input means 10 outputs the input teacher data to the classification model learning means 14.

The classification data input means 11 inputs image data whose classification is unknown. The classification data input means 11 outputs the input image data to the area-specific main direction estimation means 12 and the classification means 16.

The area-specific main direction estimating means 12 estimates the main direction of the edge component of the image data for each image area (filter area) to which the convolution filter is applied in the convolution process performed in the first convolution layer of the classification model (CNN). Is. The region-specific main direction estimation means 12 inputs image data from the learning data input means 10 in the learning mode, and inputs image data from the classification data input means 11 in the classification mode.

FIG. 2 shows an example of the size and movement amount of the filter region applied in the convolution layer. Here, the size of the convolution filter (here, 3 × 3 pixels) is the same, and the filter areas R, R, which are moved by the movement width of the convolution filter (stride: here, 1 pixel in both the horizontal and vertical directions). ..., an example of R is shown. Of course, the size and movement width of the convolution filter are not limited to this.
The region-specific main direction estimating means 12 estimates the main direction of the edge component for each of the filter regions R illustrated in FIG. As a method for estimating the main direction of the edge component, a general method such as using a Sobel filter can be used.

Here, a method of estimating the main direction of the edge component of the filter region R using the Sobel filter will be briefly described.
First, the area-specific main direction estimating means 12 uses the sobel filter ((a) vertical sobel filter, (b) horizontal sobel filter) illustrated in FIG. 3, and is close to each pixel of the filter area R. The gradient strength and gradient direction of the edge component are calculated from the pixel value of the pixel.
Here, the values obtained by applying the vertical sobel filter of FIG. 3 (a) to the pixels of the (x, y) coordinates of the filter area R are _fx (x, y), and the horizontal saw of FIG. 3 (b). When the value to which the bell filter is applied is set to f _y (x, y), the region-specific main direction estimation means 12 uses the following equation (1) to determine the gradient intensity G of the edge component of the pixel at the (x, y) coordinate. (X, y) is obtained, and the gradient direction θ (x, y) of the edge component of the pixel at the (x, y) coordinate is obtained by the following equation (2).

Thereby, as shown in FIG. 4A, the gradient intensity (vector length) and the gradient direction (vector direction) of the edge component can be obtained for each pixel of the filter region R. Then, the region-specific main direction estimating means 12 quantizes the gradient direction of the edge component for each pixel shown in FIG. 4 (a) as shown in FIG. 4 (b) (for example, quantized in units of 5 °). Then, a histogram that accumulates the gradient intensities for each gradient direction is generated.

Then, the region-specific main direction estimating means 12 estimates the gradient direction at which the cumulative gradient intensity in the histogram shown in FIG. 4B peaks as the main direction of the edge component of the filter region R. If the region-specific main direction estimation means 12 cannot detect a clear peak, it is assumed that the main direction of the edge component does not exist, and for example, the main direction is set to 0 °. Here, whether or not a clear peak exists in the histogram may be determined, for example, when the ratio of the second largest cumulative value to the highest cumulative value of the gradient intensity is larger than a predetermined ratio.
Returning to FIG. 1, the configuration of the image data classification device 1 will be described.

The area-specific main direction estimation means 12 stores the main direction of the edge component for each of the filter areas R, R, ..., R (FIG. 2) in the area-specific main direction storage means 13 in association with the position of the filter area R. ..
The region-specific main direction estimation means 12 provides a "estimation completion notification" indicating that the estimation is completed at the stage where the main directions of the edge components are estimated for all the filter regions R of the image data, in the learning mode, as a classification model. Notify the learning means 14, and in the classification mode, notify the classification means 16.

The area-specific main direction storage means 13 stores the position of the filter area of the image data in association with the main direction of the edge component corresponding to the filter area estimated by the area-specific main direction estimation means 12. The area-specific main direction storage means 13 can be configured by a general storage medium such as a hard disk or a semiconductor memory.
The classification model learning means 14 refers to the main direction of the edge component for each filter region stored in the region-specific main direction storage means 13 in the learning mode, and the classification means 16 refers to the edge component in the classification mode.

The classification model learning means 14 is mainly composed of a plurality of learning data (image data, teacher data) input from the learning data input means 10 and edge components for each filter area stored in the area-specific main direction storage means 13. A convolutional neural network (CNN), which is a classification model for classifying image data whose classification is unknown, is learned by using directions. Initial values such as parameters of the classification model are stored in the classification model storage means 15, and the classification model learning means 14 updates the parameters of the classification model stored in the classification model storage means 15 by learning.

In the convolution process in the first convolution layer of the CNN, the classification model learning means 14 rotates each filter region of the image data by a predetermined angle according to the main direction of the edge component, and refers to the rotated filter region. , Performs a convolution operation.
As shown in FIG. 5A, for example, the main direction of the edge component of a certain filter region R in the image data is from a predetermined reference direction (here, the horizontal right direction [0 ° direction] of the image). In the case of the direction of 30 °, as shown in FIG. 5B, the classification model learning means 14 sets a region rotated by 30 ° with respect to the center O of the filter region R as a new filter region _RN . .. Note that the rotation of the filter area R only rotates the area to be convolved by a predetermined angle, and does not rotate the image in the area.

Then, as shown in FIG. 5C, the classification model learning means 14 replaces the pixel values of the pixel regions (a1, a2, ..., A9) of the filter region R before rotation with the filter region R after rotation. _A convolution filter is applied to the pixel values in the N pixel region (b1, b2, ..., B9) to perform a convolution operation. Strictly speaking, the filter region _{R N} of the pixel region after the rotation (b1, b2, ..., b9 ) and the pixel value of the pixel area of the filter region _{R N} (b1, b2, ..., b9 ) of each It is a pixel value of a pixel corresponding to the center position.
As a result, the classification model learning means 14 can make the direction in which the convolution filter is applied the same as the main direction regardless of the main direction of the edge component of the filter region R.

Then, as shown in FIG. 6, the classification model learning means 14 rotates the filter area according to the main direction of the edge component when the filter area of the image data is sequentially moved, and convolves the filter Cf into the rotated filter area. Is applied to perform the convolution process.
As described above, the classification model learning means 14 performs the convolution process in the first convolution layer of the CNN so that the direction of the convolution filter is constant with respect to the main direction of the edge component in all the filter regions R. As a result, regardless of whether the object in the image data is tilted or not, it can be propagated to the next layer of the CNN as a feature amount that is almost invariant for each filter region.
Returning to FIG. 1, the configuration of the image data classification device 1 will be described.

The classification model learning means 14 performs the convolution processing according to the main direction of the edge component only in the first convolution layer, and the subsequent processing (the second and subsequent convolution layers, the pooling layer, the fully connected layer, the output layer; 12) performs the same processing as the conventional CNN.
Then, the classification model learning means 14 has a direction of eliminating an error between the classification result output from the output layer corresponding to the input image data and the known classification result which is the training data (the value of the error function is "0"). The parameters of the classification model (convolution filter, weights between layers of fully connected layers [weight matrix], etc.) are updated using, for example, the error back propagation method. Since updating the parameters of this classification model is a general CNN method, detailed description thereof will be omitted here.
As will be described later, the convolution filter in which the filter region is rotated by a predetermined angle can be updated by the back-propagation method.

The classification model storage means 15 stores the classification model learned by the classification model learning means 14. The classification model storage means 15 can be configured by a general storage medium such as a hard disk or a semiconductor memory. The classification model after learning is referred to by the classification means 16.
The classification model storage means 15 stores in advance the structure of the classification model (structure of convolutional layer, pooling layer, fully connected layer, etc., size, number, movement width, etc. of the convolutional filter), and parameters of the classification model (convolutional model storage means 15). Store the initial values of the convolutional filter, the weight between the layers of the fully connected layer [weight matrix], etc.). The parameters of the classification model are updated by the classification model learning means 14 when the learning mode is operated.

The classification means 16 uses the main direction of the edge component for each filter area stored in the region-specific main direction storage means 13 and the classification model stored in the classification model storage means 15 to input data for classification. The image data input from 11 is classified.
In the convolution process in the first convolution layer of the classification model, the classification means 16 rotates each filter region of the image data by a predetermined angle according to the main direction of the edge component, and refers to the rotated filter region with respect to the rotation filter region. Performs a convolution operation.

The convolution process in the first convolution layer of this classification model is the classification model learning means 14 described with reference to FIGS. 5 and 6, except that the learned convolution coefficient stored in the classification model storage means 15 is used. Since it is the same as the process of, the description will be omitted.
Further, the classification means 16 propagates the features of the image data by using the classification model stored in the classification model storage means 15 after the convolution processing in the first convolutional layer, and is the highest in the node of the output layer of the classification model. Outputs the corresponding classification result of the node that becomes the probability value.

The configuration of the image data classification device 1 according to the embodiment of the present invention has been described above, but the image data classification device 1 is operated by a program (image data classification program) for operating the computer as each of the above-mentioned means. Can be done.

By configuring the image data classification device 1 as described above, the image data classification device 1 learns the classification model using the image data including the objects in one direction, regardless of the orientation of the objects. Image data can be classified with high accuracy.

For example, as shown in FIG. 7, when the same object (for example, an image of “house”) is in a tilted state in different image data, the image data classification device 1 uses FIGS. 7 (a) and 7 (b). ), (C), when the convolution filter Cf is applied in the same region of the object (the image region of the “chimney portion”), the convolution process is performed in the filter region that is in the same direction as the main direction of the edge component. Therefore, the image data classification device 1 can extract substantially the same feature amount in the same region of the same object, and may learn the classification model using the image data including the objects in one direction.

<Operation of image data classification device>
Next, the operation of the image data classification device 1 according to the embodiment of the present invention will be described with reference to FIGS. 8 and 9. Here, the operation of the image data classification device 1 will be described separately for a learning mode and a classification mode.

(Learning mode)
The operation of the learning mode of the image data classification device 1 will be described with reference to FIG. 8 (see FIG. 1 for the configuration as appropriate).

In step S1, the learning data input means 10 inputs image data whose classification is known and teacher data indicating the classification content as learning data.
Then, the region-specific main direction estimating means 12 estimates the main direction of the edge component for each filter region to which the convolution filter is applied in the image data input in step S1 by the following operations from step S2 to step S6.

In step S2, the area-specific main direction estimation means 12 sets the initial position (for example, the upper left of the image) of the filter area to which the convolution filter is applied with respect to the image data input in step S1.

In step S3, the region-specific main direction estimation means 12 estimates the main direction of the edge component of the image in the filter region. Specifically, the region-specific main direction estimation means 12 uses a Sobel filter to obtain the gradient intensity and the gradient direction of each pixel of the image in the filter region. Then, the region-specific main direction estimating means 12 quantizes the gradient direction, accumulates the gradient intensities for each of the quantized gradient directions, and sets the gradient direction at which the cumulative gradient intensity peaks as the main direction of the edge component. If the ratio of the second largest cumulative value to the highest cumulative value of the gradient intensity is larger than the predetermined ratio, it is assumed that the main direction does not exist (main direction = 0 °).

In step S4, the area-specific main direction estimation means 12 stores the position of the filter region and the main direction of the edge component estimated in step S3 in association with each other in the area-specific main direction storage means 13.
In step S5, the region-specific main direction estimating means 12 determines whether or not the main direction of the edge component has been estimated for the images of all the filter regions in the image data.

Here, when the main directions of the edge components have not yet been estimated for the images of all the filter regions (No in step S5), in step S6, the region-specific main direction estimation means 12 convolves the filter region. Move to a position according to the movement width of the filter. Then, the region-specific main direction estimating means 12 returns to step S3 and estimates the main direction of the edge component with respect to the image of the next filter region.
On the other hand, when the main direction of the edge component is estimated for the images of all the filter regions (Yes in step S5), the classification model learning means 14 performs the operations after step S7.

The classification model learning means 14 processes the first convolution layer by the following operations from step S7 to step S11.

In step S7, the classification model learning means 14 sets the initial position of the filter area to which the convolution filter is applied with respect to the image data input in step S1.
In step S8, the classification model learning means 14 reads out the main direction of the edge component corresponding to the position of the filter area from the area-specific main direction storage means 13, so that the predetermined reference direction of the filter area becomes the main direction. The rotated region is used as a new filter region.

In step S9, the classification model learning means 14 applies a convolution filter to the filter region rotated by a predetermined angle in the direction of the main direction in step S8 to perform a convolution operation.
In step S10, the classification model learning means 14 determines whether or not the convolution operation has been performed on all the filter areas in the image data.

Here, when the convolution operation has not yet been performed on all the filter areas (No in step S10), in step S11, the classification model learning means 14 positions the filter area according to the movement width of the convolution filter. Move to. Then, the classification model learning means 14 returns to step S8 and performs a convolution operation on the next filter area.

On the other hand, when the convolution operation is performed on all the filter areas (Yes in step S10), the classification model learning means 14 proceeds to step S12. Although not shown, when a plurality of convolution filters are used in the first convolution layer, the classification model learning means 14 executes the operations from step S7 to step S11 by the number of convolution filters.

In step S12, the classification model learning means 14 refers to the convolution layer, the pooling layer, the fully connected layer, and the output layer of the second and subsequent convolution layers in the subsequent stage with respect to the feature map generated by the first convolution layer in the operations up to step S11. Executes the processing of.
In step S13, the classification model learning means 14 updates the parameters of the classification model by using the error backpropagation method from the error between the classification result output from the output layer in step S12 and the teacher data input in step S1. Then, it is stored in the classification model storage means 15.

In step S14, the classification model learning means 14 determines whether or not the learning of the classification model has been completed. Here, the determination of learning of the classification model is a case where the error in step S13 becomes smaller than the predetermined threshold value.
Here, when the learning of the classification model is not completed (No in step S14), in step S1, the learning data input means 10 inputs new learning data, so that the classification model learning means 14 uses the classification model. Continue learning.
On the other hand, when the learning of the classification model is completed (Ye in step S14), the image data classification device 1 ends the operation.

By the above operation, when the image data classification device 1 learns the classification model of CNN, the convolution filter is set in a constant direction with respect to the main direction of the edge component of the filter region in the processing of the first convolution layer. Is applied to perform the convolution process.
As a result, the image data classification device 1 can perform learning by extracting features that are almost invariant to the inclination of the object in the image data, so that the image data in which the object is projected in various directions can be used as the training data. It is not necessary to do so, and the amount of training data and the training time can be reduced as compared with the conventional case.

(Classification mode)
Next, the operation of the image data classification mode of the image data classification device 1 will be described with reference to FIG. 9 (see FIG. 1 for the configuration as appropriate).

In step S20, the classification data input means 11 inputs image data whose classification is unknown.
Then, the region-specific main direction estimating means 12 estimates the main direction of the edge component for each filter region to which the convolution filter is applied in the image data input in step S20 by the operation of steps S21 to S25. Since the operations of steps S21 to S25 are the same as the operations of steps S2 to S6 described with reference to FIG. 8, the description thereof will be omitted.

Then, the classification means 16 processes the first convolution layer by the operations of steps S26 to S30. The operation of steps S26 to S30 is the same as the operation of steps S7 to S11 described with reference to FIG. 8 except that the action subject changes from the classification model learning means 14 to the classification means 16. Therefore, the description thereof will be omitted.

In step S31, the classification means 16 processes the convolution layer, the pooling layer, the fully connected layer, and the output layer of the second and subsequent convolution layers in the subsequent stage with respect to the feature map generated by the first convolution layer in the operations up to step S30. To execute.
In step S32, the classification means 16 outputs the corresponding classification result of the node having the highest probability value among the nodes of the output layer in step S31.

By the above operation, when the image data classification device 1 classifies the image data by the classification model of CNN, in the processing of the first convolution layer, the image data classification device 1 is in a constant direction with respect to the main direction of the edge component of the filter region. A convolution filter is applied to the convolution process.
As a result, the image data classification device 1 extracts a feature amount that is almost invariant to the inclination of the object of the image data. Therefore, even if the image data shows the same object in different directions, the image data is displayed as the same content. Can be classified.

<Modification example>
Although the configuration and operation of the image data classification device 1 according to the embodiment of the present invention have been described above, the present invention is not limited to this embodiment.
(Modification example 1)
The image data classification device 1 executes processing of two different operation modes, a mode for learning a classification model (learning mode) and a mode for classifying image data (classification mode), with one device. However, these processes may be performed by separate devices.

Specifically, the device for learning the classification model can be configured as the classification model generation device 2 shown in FIG.
As shown in FIG. 10, the classification model generation device 2 includes a learning data input means 10, a region-specific main direction estimation means 12, a region-specific main direction storage means 13, a classification model learning means 14, and a classification model storage. Means 15 and. In this configuration, the classification data input means 11 and the classification means 16 are deleted from the configuration of the image data classification device 1 described with reference to FIG.
The classification model generation device 2 only performs an operation of learning the classification model. The operation of the classification model generator 2 is the same as the operation described with reference to FIG.
The classification model generation device 2 can be operated by a program (classification model generation program) for operating the computer as each of the above-mentioned means.

(Modification 2)
Further, the device for classifying the image data using the classification model can be configured as the image data classification device 1B shown in FIG.
The image data classification device 1B includes a data input means 11 for classification, a main direction estimation means 12 for each area, a main direction storage means 13 for each area, a classification model storage means 15, and a classification means 16. In this configuration, the learning data input means 10 and the classification model learning means 14 are deleted from the configuration of the image data classification device 1 described with reference to FIG. The classification model stored in the classification model storage means 15 is generated by the classification model generation device 2 of FIG.
The image data classification device 1B only performs an operation of classifying image data. The operation of the image data classification device 1B is the same as the operation described with reference to FIG.
The image data classification device 1B can be operated by a program (image data classification program) for operating the computer as each of the above-mentioned means.

In this way, by operating the operation of learning the classification model and the operation of classifying the image data using the classification model by different devices (classification model generation device 2, image data classification device 1B), one classification is performed. The classifier model generated by the model generation device 2 can be used by a plurality of image data classification devices 1B.

(Modification example 3)
Further, here, the region-specific main direction estimation means 12 estimates the main direction of the edge component using a sobel filter, but the present invention is not limited to this.
For example, the region-specific main direction estimation means 12 may use the gradient intensity and gradient direction of the edge component, which is a feature amount of image data such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Up Robust Features). .. Alternatively, using the result of machine learning the image of the size of the convolution filter with a plurality of patterns in which the main direction of the edge component is known in advance, the region-specific main direction estimation means 12 determines the main direction of the input image data. It may be estimated.

<About updating the convolution filter>
Finally, in the classification model learning means 14 (FIG. 1), it will be described that the convolution filter in which the filter region is rotated by a predetermined angle can be updated (learned) by the error back propagation method.
Assuming that the output value (weighted sum) at the coordinates (i, j) of the Lth layer in the CNN is _uij ^L and the activation function is f, the activity (value of the activation function) z _ij ^L is the following equation ( It can be represented by 3).

Here, assuming that the coefficient of the convolution filter is h _{pq , the output value uij} ^L in the conventional convolution layer can be expressed by the following equation (4). Note that (p, q) indicates the coordinates of the convolution filter.

On the other hand, in the present invention, the coordinates (i + p, j + q) to be convoluted by the convolution filter are rotated by a predetermined angle according to the main direction of the edge component. This rotation angle is information known before the learning of the CNN in the classification model learning means 14 by the region-specific main direction estimating means 12. Here, assuming that the coordinates after rotation are ((i + p)', (j + q)'), the output value u _ij ^L of the first convolution layer in the classification model learning means 14 can be expressed by the following equation (5). can.

In the present invention, whether or not the convolution filter rotated by a predetermined angle by the error backpropagation method can be updated is synonymous with whether or not the error function is differentiable (the gradient of the error function can be obtained). be. Hereinafter, it is shown that the error function is differentiable in the present invention.
Here, let the error function be E. The gradient of the error function E can be expressed by the following equation (6) from the chain rule of partial differentiation.

Here, the result of partially differentiating the error function E with the weighted sum u _ij ^L _{is defined as δ ij} ^L shown in the following equation (7).

Then, from the equation (5), the equation (6) can be rewritten to the following equation (8).

_{The z (i + p)'and (j + q)'} ^L-1 of this equation (8) are the output values of the previous layer (the (L-1) layer), and the coordinate values after rotation are the edge components. Since the main direction has already been determined, it will be a fixed value. Therefore, in order to carry out error propagation, it is sufficient if _{δ ij} ^{L can be obtained.} Whether or not δ _ij ^L can be obtained can be obtained by the same method as in the conventional method regardless of whether or not the convolution filter is rotated by a predetermined angle.
First, according to the chain rule of partial differentiation, δ _ij ^L can be transformed as shown in the following equation (9). The weighted sum at the coordinates (s, t) is _ust ^L.

Here, from the above formula (3) and the above formula (5), (∂u _st ^{L + 1} / ∂u _ij ^L ) of the above formula (9) can be transformed into the following formula (10).

When the formula (9) is replaced with the formula (10), the following formula (11) is obtained.

Since ∂ (...) / ∂u _ij ^L in this equation (11) is partially differentiated with respect to u _ij ^L , it becomes us _{+ p, t + q} ^L = u _ij ^L , that is, s + p = i, t + q = j. Since only the combination of (s, t) and (p, q) needs to be considered (other values are "0"), the above equation (11) becomes the following equation (12).

Here, f'(...) is the derivative of the known activation function f, and δ _{ip and j-q} ^{L + 1} are values propagated from the subsequent layer, so that δ _ij ^L is obtained. be able to.
As described above, also in the present invention, the error function E is differentiable, and the classification model can be learned by performing the forward propagation and back propagation processing in the CNN.

1,1B Image data classification device 2 Classification model generation device 10 Learning data input means 11 Classification data input means 12 Area-specific main direction estimation means 13 Area-specific main direction storage means 14 Classification model learning means 15 Classification model storage means 16 Classification means

Claims (7)

A classification model generator that generates a classification model, which is a convolutional neural network for classifying image data whose classification is unknown, from a plurality of image data whose classification is known.
From the image data whose classification is known, a region-specific main direction estimating means for estimating the main direction of the edge component of the image for each filter region to which the convolution filter of the first convolutional layer of the convolutional neural network is applied.
A classification model learning means for learning the convolutional neural network and generating the classification model from image data whose classification is known and teacher data indicating the classification contents is provided.
In the classification model learning means, in the first convolution layer, for each of the filter regions, the orientation of the predetermined reference of the filter region is relative to the main direction of the edge component estimated by the region-specific main direction estimation means. A classification model generator characterized in that a convolution operation is performed by rotating a filter area so as to have a certain direction. The region-specific main direction estimating means calculates the gradient intensity and gradient direction of the edge component of the filter region from the pixel values of the nearby pixels close to the pixel for each pixel in the filter region, and for each quantized gradient direction. The classification model generation device according to claim 1, wherein the gradient angle having the largest accumulated gradient intensity is estimated as the main direction. An image data classification device for classifying image data whose classification is unknown by using a classification model which is a convolutional neural network generated by the model generation device according to claim 1 or 2.
A region-specific main direction estimation means for estimating the main direction of the edge component of the image for each filter region to which the convolution filter of the first convolutional layer of the convolutional neural network is applied from the image data whose classification is unknown.
The convolutional neural network, which is the classification model, is provided with a classification means for classifying image data whose classification is unknown.
In the first convolutional layer, the classification means has a predetermined reference orientation of the filter region for each of the filter regions in a fixed direction with respect to the main direction of the edge component estimated by the region-specific main direction estimation means. An image data classification device characterized in that a convolution operation is performed by rotating a filter area so as to become. The area-specific main direction estimating means calculates the gradient intensity and the gradient direction of the edge component from the pixel values of the nearby pixels for each pixel in the filter region, and the gradient intensity accumulated for each quantized gradient direction is the largest gradient. The image data classification device according to claim 3, wherein the angle is estimated as the main direction. An image data classification device that generates a classification model, which is a convolutional neural network for classifying image data whose classification is unknown, from a plurality of image data whose classification is known, and classifies the image data whose classification is unknown.
From the image data, for each filter region to which the convolution filter of the first convolution layer of the convolutional neural network is applied, a region-specific main direction estimation means for estimating the main direction of the edge component of the image, and a region-specific main direction estimation means.
A classification model learning means that learns the convolutional neural network and generates the classification model from image data whose classification is known and teacher data indicating the classification contents.
The convolutional neural network, which is the classification model, is provided with a classification means for classifying image data whose classification is unknown.
In the classification model learning means, in the first convolution layer, for each of the filter regions, the orientation of the predetermined reference of the filter region is relative to the main direction of the edge component estimated by the region-specific main direction estimation means. Rotate the filter area so that it is in a fixed direction, perform the convolution operation, and perform the convolution operation.
In the first convolutional layer, the classification means has a predetermined reference orientation of the filter region for each of the filter regions in a fixed direction with respect to the main direction of the edge component estimated by the region-specific main direction estimation means. An image data classification device characterized in that a convolution operation is performed by rotating a filter area so as to become. A classification model generation program for operating a computer as the classification model generation device according to claim 1 or 2. An image data classification program for causing a computer to function as the image data classification device according to any one of claims 3 to 5.

Images