CN105184303B - An Image Annotation Method Based on Multimodal Deep Learning - Google Patents

Abstract

The invention discloses an image labeling method based on multi-modal deep learning. The method comprises the following steps: firstly, using unlabeled images to train a deep neural network; secondly, using backpropagation to optimize each single mode; finally, using online The learned power gradient algorithm optimizes the weights across different modalities. The invention optimizes the parameters of the deep neural network by applying the convolutional neural network technology, and improves the labeling accuracy. Experiments on public datasets show that the invention can effectively improve the performance of image annotation.

Description

An Image Annotation Method Based on Multimodal Deep Learning

technical field

The invention relates to an image tagging method, in particular to an image tagging method based on multimodal deep learning, and belongs to the technical field of image processing.

Background technique

In recent years, with the rapid increase in the number of images, people urgently need to achieve efficient annotation of image content to achieve effective retrieval and management of large-scale images.

From the perspective of pattern recognition, the image annotation problem is regarded as assigning a set of labels to images according to the content, and how to select the appropriate features to represent the image content will greatly affect the annotation performance. Due to the well-known semantic gap problem, it is difficult for existing technologies to achieve satisfactory results when semantically annotating images. In recent years, Hinton et al. proposed to use deep neural networks to efficiently train features from the training set. Different types of deep neural networks have been successfully applied to various languages and information retrieval. These methods discover hidden data structures and effective representational features from training data through deep structure and deep learning to improve system performance.

Contents of the invention

The purpose of the present invention is to provide an image labeling method based on multimodal deep learning, which is applied to convolutional neural network technology, optimizes deep neural network parameters, and improves labeling accuracy. This method summarizes the basis of single-modal learning and realizes multi-modal learning, which includes not only studying the underlying features of the representation image, such as color, shape or texture, but also measuring the similarity function between the image and the label, such as linear similarity. sex, cosine similarity, and radial distance.

The technical scheme adopted by the present invention to solve the technical problem is: the present invention provides a method for image labeling based on multimodal deep learning, which comprises the following steps:

Step 1: Use the unlabeled image sample set to pre-train the node weights of the deep neural network.

Step 2: Use the backpropagation algorithm to optimize the weights of each single mode.

Step 3: Use the power gradient algorithm of online learning to optimize the weights between modal combinations.

The deep neural network described in step 1 of the present invention adopts eight layers of convolutional neural networks, wherein the first five layers are convolutional layers, and the remaining three layers are fully connected layers; the output of the fully connected layer is used as the input of the Softmax classifier, and Softmax The classifier generates 1000 labeled categories; both the pre-training and fine-tuning phases use the objective function of multinomial logistic regression.

In the above-mentioned convolutional layers of the present invention, the first layer, the second layer, and the fifth layer are all normalized layers, and in order to maintain invariance, all normalized layers use the maximum pooling technique. In addition, in all convolutional layers and fully connected layers, linear adjustment units are used as nonlinear activation functions;

In the convolutional neural network used above in the present invention, the size of all input images is unified to 256×256; next, the first two convolution filters are set to 7×7 and 5×5 respectively, and the step size is 2, using this This type of filter is to obtain all frequency band information, and the use of a small step size is to avoid the generation of "dead features" that affect the next layer of the network; then, the last three layers of the convolutional layer are connected in turn, and the filter size is set 3×3 with a stride of 1; finally, the output size of each fully connected layer is 4096. In the pre-training phase, the loss-of-signal ratio of the first two fully-connected layers is set to 0.6.

The backpropagation described in step 2 of the present invention optimizes each single-mode step, comprising:

①Single-modal pre-training:

Use the unlabeled training set to pre-train the convolutional neural network, realize the intermediate representation of the image object, and initialize the network at the same time. The specific process is described as follows: First, use the comparison difference to train the node weight W ₁ between the input layer and the first convolutional layer; then, use the conditional probability of the nodes in the first convolutional layer as the input of the second convolutional layer:

p(Γ|x ^j )=S(W ₁ ,x ^j ) (1)

Where x ^j is the jth feature vector, Γ is the label information, and S() is the similarity function shown in the following formula:

Then, the first convolutional layer and the second convolutional layer are combined to train the node weight W ₂ ; use the same method to train the node weights of the remaining 3 convolutional layers and 3 fully connected layers;

②Single-mode fine-tuning stage:

In the single-modal fine-tuning stage, the node weights are optimized by back-propagating annotation errors. From the perspective of pattern recognition, multi-label learning can be regarded as multi-task learning. Therefore, the overall labeling error of a convolutional neural network can be regarded as the sum of each labeling error. The following takes the lth labeling error as an example to illustrate the node weight optimization process;

First, for an image x, the probability that it contains the l-th label Γ _l in the j-th feature mode x ^j can be expressed by the posterior probability of the following formula:

where L represents the number of labels.

Then, minimize the KL difference between the predicted probability and the reference probability. Assuming that each image has multiple labels, it is represented by a vector y∈R ₁ ×c, where y _l = 1 means that the label set of image x contains the l-th label, and y _l = 0 means that the label set of image x does not contain This is the lth label. q _il represents the probability between the image x _i and the label l, then the error of correctly assigning the l-th label to the image is:

The distribution error for all labels is:

Finally, backpropagation is used to update the node weights of the other two layers of fully connected layers and the five layers of convolutional layers.

include:

For multimodal deep networks, another important task is to learn the best combined weights α=(α ₁ ,α ₂ ,…,α _n ,…,α _N ) among multimodal, where α _{n is} initially set into 1/N. The present invention adopts the power gradient algorithm of online learning to optimize the weight combination of multiple modes:

Where KL(.) represents the KL difference, and h(α) represents the hinge loss function:

where S _t is:

S _t ＝(S ₁ (x,Γ ⁺ )-S ₁ (x,Γ ^- ),...,S _N (x,Γ ⁺ )-S _N (x,Γ ^- )) ^T (8)

The labels Γ ⁺ and Γ ^- can better reflect the image content.

Perform a first-order Taylor expansion on the function h(α) at α _t to simplify the optimization problem, so equation (8) can be written as a first-order Taylor expansion:

If Γ+ and Γ- are not arranged correctly in order, the value of node weight α is automatically updated.

Beneficial effect:

1. The present invention optimizes the deep neural network parameters and improves the labeling accuracy.

2. The present invention better realizes the effectiveness of image labeling based on the deep neural network learning model.

3. The present invention can effectively improve the performance of image labeling.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention.

Fig. 2 is the deep neural network model of the present invention.

Fig. 3 is an example image of the natural scene graph library of the present invention.

Figure 4 is an image of the NUS-WIDE image library of the present invention.

FIG. 5 is an example image of the IAPRTC-12 image database of the present invention.

FIG. 6 is a schematic diagram of the results of different modality weight combinations in the three public image libraries of the present invention.

Detailed ways

The invention will be described in further detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the present invention provides a kind of image tagging method based on multimodal deep learning, and this method comprises: First, utilize unlabeled image training deep neural network; Finally, the power gradient algorithm of online learning is used to optimize the weights between different modalities.

The deep neural network in the present invention adopts a convolutional neural network, and its model structure is shown in FIG. 2 . The present invention evaluates the performance of the image labeling algorithm based on multimodal deep learning proposed by the present invention through a series of experiments.

Step 1: Introduce the dataset used to evaluate the performance of the algorithm.

The experiment uses three public image datasets, including the natural scene image library shown in Figure 3, the NUS-WIDE image library shown in Figure 4, and the IAPRTC-12 image library shown in Figure 5. The details of these three image libraries are described as follows:

The natural scene image library contains 2000 images, all of which contain the following 5 annotations: desert, mountain, sea, sunset and trees. More than 20% of images contain more than one annotation, with an average of 1.3 annotations per image. Figure 3 shows two sample images from the natural scene graph library, where Figure 3(a) is labeled as the sunset and the sea, and Figure 3(b) is labeled as mountains and trees.

The NUS-WIDE image library contains 30,000 images labeled with 31 annotations including boats, cars, flags, horses, sky, sun, towers, airplanes, zebras, etc. Figure 4 shows two images from the NUS-WIDE image library, where the labels in Figure 4(a) contain the sky and the plane, and the labels in Figure 4(b) contain the sea and the sunset.

The IAPRTC-12 image database contains 20,000 images with 291 annotations, and the average number of annotations per image is 5.7. Figure 5 shows two example images from the IAPRTC-12 image database. The annotations in Fig. 5(a) contain brown, human face, hair, man and woman, while the annotations in Fig. 5(b) contain ships, lakes, sky, trees.

Step 2: Give the visual features representing the image and the optimal parameters learned.

Feature selection has a great impact on system performance. The present invention selects the following global features and local features as descriptors for image representation:

Global features: (1) 128-dimensional HSV color histogram and 225-dimensional LAB color moment, (2) 37-dimensional edge direction histogram, (3) 36-dimensional pyramidal wavelet texture, (4) 59-dimensional local binary pattern feature descriptor , (5) 960-dimensional GIST feature descriptor.

Local features: Two different sampling methods and three different local descriptors are used to extract local texture features. The specific process includes the following description: First, perform dense sampling and Harris corner detection; then, extract SIFT features, CSIFT features , RGBSIFT features, and construct a codebook of 1000 categories for k-means clustering; next, use the two-level spatial pyramid mode to construct a 5000-dimensional vector for each image; finally, use the TF-IDF weight method to generate the final visual bag of words. All eigenvectors are normalized in the range [0,1] throughout the experiments.

For each query-label pair, three similarity measures are given in the above formula (4), and the marginal parameter μ is selected by cross-validation. After cross-validation, the μ value in the cosine similarity measure is 0.18; the μ value in the linear similarity measure is 1; the σ value in the RBF similarity measure is 2 and the μ value is 0.18.

Step 3: Test the performance of the proposed algorithm of the present invention through comparative experiments.

Algorithm comparison

The comparative experiment of the present invention is carried out between following three kinds of image classification methods:

Based on the lazy learning algorithm: first, for each test image, find the K most similar images in the training image library; then, count the characteristics of the K most similar images; finally, assign the labels of the test images according to the maximum posterior probability.

Based on deep representation and coding algorithm: use layered model to learn image pixel-level representation and realize image annotation

The method of the invention: realizing image labeling through a deep neural network.

Modal weight

In the method of the present invention, the combination weight α of different modes has a great influence on the system performance. Figure 5 shows the results of different modality weight combinations in three public image libraries. Figure 6(a): Combination weights of different modalities under the natural image library. Figure 6(b): Combination weights of different modalities under NUS-WIDE images. Figure 6(c): Weights of different modality combinations under IAPRTC-12 images.

From the results shown in Fig. 6, it can be easily seen that there is no significant difference in the proportions among the different modalities. This means that each modality is more or less helpful to different image categories, mainly because these three image libraries contain many different categories of natural scene images, which further verifies the optimal combination of different modalities importance.

performance comparison

Table 1 presents the experimental comparison results of several multi-label image annotation techniques using different methods.

Table 1: Experimental comparison results.

It can be seen from the results shown in Table 1 that the NDCG@w performance of the method proposed in the present invention is better than the other two existing methods, which verifies the effectiveness of image annotation based on the deep neural network learning model.

Claims (2)

1. a method for image labeling based on multimodal deep learning, is characterized in that, described method comprises the steps: Step 1: Use the unlabeled image sample set to pre-train the node weights of the deep neural network. The deep neural network uses an eight-layer convolutional neural network, of which the first five layers are convolutional layers, and the remaining three layers are fully connected layers; The output of the connection layer is used as the input of the Softmax classifier, and the Softmax classifier generates 1000 identified categories; both the pre-training and fine-tuning stages use the objective function of multinomial logistic regression; The first layer, the second layer, and the fifth layer of the convolutional layer are all normalized layers, and in order to maintain invariance, all normalized layers use the largest pooling technique; in all convolutional layers and fully connected layers In both, the linear adjustment unit is used as the nonlinear activation function; In the convolutional neural network used, the size of all input images is uniformly 256×256; next, set the first two convolution filters to 7×7 and 5×5 respectively, with a step size of 2, and use this type of filter In order to obtain all frequency band information, the small step size is used to avoid the generation of "dead features" that affect the next layer of the network; then, the last three layers of the convolutional layer are connected in turn, and the filter size is set to 3×3, The step size is 1; finally, the output size of each fully connected layer is 4096, and in the pre-training stage, the signal loss rate of the first two fully connected layers is set to 0.6; Step 2: Use the backpropagation algorithm to optimize the weight of each single mode. The backpropagation algorithm includes: ①Single-modal pre-training: Use the unlabeled training set to pre-train the convolutional neural network, realize the intermediate representation of the image target, and initialize the network at the same time, including: first, use the comparison difference to train the node weight W ₁ between the input layer and the first convolutional layer; Then, the conditional probability of the first convolutional layer node is used as the input of the second convolutional layer: p(Γ|x ^j )=S(W ₁ ,x ^j ) (1) Where x ^j is the jth feature vector, Γ is the label information, and S() is the similarity function shown in the following formula: Then, the first convolutional layer and the second convolutional layer are combined to train the node weight W ₂ ; use the same method to train the node weights of the remaining 3 convolutional layers and 3 fully connected layers; ②Single-mode fine-tuning stage: In the single-mode fine-tuning stage, the node weights are optimized by backpropagating labeling errors. From the perspective of pattern recognition, multi-labeling learning is regarded as multi-task learning; the overall labeling error of the convolutional neural network is regarded as the sum of each labeling error, The node weight optimization process is described by the lth labeling error, including: First, for an image x, in the j-th feature mode x ^j , the probability of containing the l-th label Γ _l is expressed by the posterior probability of the following formula: Where L represents the number of labels; Then, minimize the KL difference between the predicted probability and the reference probability; assuming that each image has multiple labels, y∈R _1c is represented by a vector, where y _l = 1 means that the label set of image x contains the l-th label, and y _l = 0 means that the label set of image x does not contain the l-th label, q _il indicates the probability between image x _i and label l, then the error of correctly assigning the l-th label to the image is: The distribution error for all labels is: Finally, use backpropagation to update the node weights of the other two layers of fully connected layers and five layers of convolutional layers; Step 3: Use the power gradient algorithm of online learning to optimize the weights between modal combinations; For multimodal deep networks, another important task is to learn the best combined weights α=(α ₁ ,α ₂ ,…,α _n ,…,α _N ) among multimodal, where α _{n is} initially set into 1/N; use the power gradient algorithm of online learning to optimize the weight combination of multi-modality, including: Where KL(.) represents the KL difference, and h(α) represents the hinge loss function: where S _t is: S _t ＝(S ₁ (x,Γ ⁺ )-S ₁ (x,Γ ^- ),...,S _N (x,Γ ⁺ )-S _N (x,Γ ^- )) ^T (8) Among them, the labels Γ ⁺ and Γ ^- can better reflect the image content; Perform a first-order Taylor expansion on the function h(α) at α _t to simplify the optimization problem, so equation (8) can be written as a first-order Taylor expansion: If Γ+ and Γ- are not arranged correctly in order, the value of node weight α is automatically updated. 2. The image labeling method based on multimodal deep learning according to claim 1, wherein the image labeling method based on multimodal deep learning is applied to a convolutional neural network.