CN112116685A - Image caption generation method with multi-attention fusion network based on multi-granularity reward mechanism - Google Patents

Abstract

The invention discloses an image caption generation method based on a multi-granularity reward mechanism and a multi-attention fusion network, which solves the problem that each generated word has different importance in the image caption generation method based on the reinforcement learning reward mechanism. The present invention firstly proposes a multi-attention fusion network based on a multi-granularity reward mechanism for image caption generation, which includes a multi-attention fusion model, a word importance re-evaluation network and a label retrieval network. A multi-attention fusion model is used as a baseline for reinforcement learning-based image captioning methods; a word importance re-evaluation network is used for reward re-evaluation by estimating the different importance of each word in the generated caption; a label retrieval network can The corresponding ground-truth labels are retrieved from the subtitles as the retrieval reward, and then the network is trained to generate better subtitles in a way that maximizes the reward. The present invention has been verified by a large number of experiments on the MSCOCO data set, and obtained very competitive evaluation results.

Description

Image caption generation with multi-attention fusion network based on multi-granularity reward mechanism method

technical field

The invention belongs to a method for automatically generating image captions, and relates to the technical fields of computer vision and natural language processing.

Background technique

The goal of image captioning is to automatically generate a natural language description of a given image. This task currently faces huge challenges. On the one hand, the computer must comprehensively understand the image content from the multi-level visual features; on the other hand, the image caption generation algorithm needs to gradually modify the rough semantic concepts into human-like natural language descriptions . In recent years, advances in deep learning related technologies (including attention mechanism and reinforcement learning) have significantly improved the quality of caption generation, among which the encoder-decoder framework is the mainstream method for image caption generation. Vinyals et al. used spatially merged CNN feature maps to generate subtitles, compress the entire image into a static representation, and then used an attention mechanism to improve the performance of subtitles by learning to adaptively focus on regions of the image, but only a single LSTM was used for visual information The handler as well as the language generator, which is weakened by the simultaneous visual handler. Peter Anderson et al. proposed a top-down architecture with two independent LSTM layers: the first LSTM layer acts as a top-down visual attention model, and the second LSTM layer acts as a language generator. All the image captioning methods mentioned above use the high-level visual features of the last convolutional layer of CNN as the image encoder, ignoring the low-level visual features, which in fact are also beneficial for understanding the image. Due to the complementarity between multi-layer features, image captioning can also be optimized by using multi-layer feature fusion. However, early fusion methods are not very effective, and how to integrate multi-level visual features into image captioning models is a problem worth considering. In general, training image captioning models is achieved by maximizing cross-entropy (XE), which makes image captioning models more sensitive to abnormal captions, rather than optimizing around human consensus on suitable captions for stable output. Furthermore, captioning models, such as BLEU, ROUGE, METEOR, and CIDEr, are usually evaluated by computing different metrics on the test set. The mismatch between objective function and evaluation metric can adversely affect image captioning models, a problem that can be addressed by reinforcement learning (RL) such as PolicyGradient and Actor-Critic. Reinforcement learning methods can optimize non-differentiable sequence-based evaluation metrics, and when using the Policy Gradient method, SCST authors apply CIDEr as a reward, producing captions that are more in line with human linguistic consensus.

In SCST, each word is given the same reward as a gradient weight. However, not all words should be given equal rewards in a sentence, and different words may have different importance. Yu et al. utilize Monte Carlo to introduce SeqGan to estimate the importance of each word, however, it must generate rich sentences, which leads to expensive time complexity. Based on the Actor-Critic strategy, Dzmitry Bahdanau et al. adopted a value evaluation network to evaluate words, but evaluation metrics (eg, CIDEr, BLEU) could not be directly optimized. In this paper, we propose to utilize word-level rewards to optimize an RL-trained image captioning model, aiming to address the different importance of each generated word.

Computing evaluation metrics (e.g., CIDEr, BLEU) as reward signals is an intuitive way in RL training to generate more human-like subtitles, however, these evaluation metrics are not the only criteria for judging the quality of generated subtitles. The quality of a can also be assessed by whether it can retrieve the corresponding tags in the retrieval system. From the perspective of information utilization, the traditional CIDEr reward makes full use of the matching label information, while the retrieval reward benefits from the additional label information, and the retrieval loss can also be used as a reward system.

In this paper, a Hierarchical Attention Fusion (HAF) model for image captioning is proposed, which integrates Resnet's multi-level feature maps with hierarchical attention to serve as a baseline for RL-based image captioning methods. Furthermore, multi-granularity rewards are presented in the RL stage to modify the proposed HAF. Specifically, the word importance re-evaluation network (REN) is used for reward re-evaluation by estimating the different importance of each word in the generated caption, where the reward for re-evaluation is obtained by weighting the CIDEr score, Different weights are computed from REN, and the re-evaluated reward can be viewed as a word-level reward. To benefit from the extra labels, a Label Retrieval Network (RN) is implemented to retrieve the corresponding labels from a batch of captions as a retrieval reward, which can be viewed as a sentence-level reward.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problem of the different importance of each generated word in the image caption generation method based on the reinforcement learning reward mechanism, so as to generate a sentence that is more in line with the consensus of human language, not all words should be in one sentence given equal rewards, different words may have different importance.

The technical scheme that the present invention takes for solving the above-mentioned technical problems is:

S1. Build a multi-attention fusion model.

S2. Construct a word importance re-evaluation network based on reinforcement learning reward mechanism.

S3. Combine the reinforcement learning reward mechanism to build a label retrieval network.

S4. Combine the model in S1, the word importance re-evaluation network in S2, and the label retrieval network in S3 to build a multi-attention fusion network architecture based on a multi-granularity reward mechanism.

S5. Training and caption generation of a multi-attention fusion network based on a multi-granularity reward mechanism.

Among them, the multi-attention fusion model (HAF), as the baseline for image captioning RL training, pays attention to the layered visual features of CNN and makes full use of the multi-layered visual information, in addition to using the last layer of convolutional representation of the image and adopting a single attention The model focuses outside a specific region of the image at each time step, and we also consider fusing an attention model for captioning and input attention-derived image features to the unit nodes of a linguistic LSTM. We adopt a classical network structure that produces normalized attention weights _{αt based on the LSTM hidden state ht at each time step t .} α _t is used to participate in the different spaces Att of the image features as the final representation of the image (A):

α _t =softmax(at ₎ (2)

是学习参数。Among them, W _a , U _a ,

is the learning parameter.

where h2 is the output of the ^second LSTM, which consists of the image information of the convolutional layer and the content of the generated sequence. ^The process of generating h2 can be given by:

Finally, the probability of the output word is given by a non-linear softmax function:

The word importance re-evaluation network is constructed based on a reinforcement learning reward mechanism to re-evaluate metric-based rewards by automatically estimating the importance of different words in the generated captions. First, REN takes the generated sentence S as input, then, the sentence is processed by the RNN with attention network and average pooling layer, the word embedding vector is composed of the sentence embedding vector with attention and the sentence embedding vector after pooling are connected as a comprehensive representation for generating subtitles, and then two fully connected layers and sigmoid transform are applied to obtain the weights W _t of different words. In particular, the caption model (rl-model) pretrained by the CIDEr reward mechanism acts as a baseline (b), reducing variance significantly without changing the expected gradient. We construct the word-level reward Wr _t to be 16, so only samples from the model that outperform the current test model (rl-model) are given positive weights, while inferior samples are suppressed. Mathematically, the loss function can be formalized as formula (11):

Wr _t =RW _t +Rb (10)

表示生成的句子的不同单词。where Wi is the output weight of _REN , θ is the parameter of the image captioning network,

Represent different words of the generated sentence.

In order to take advantage of the metric-based reward (CIDEr) and constrain the sentence space, after CIDEr optimization, word-level rewards are employed to fine-tune the captioning network, in addition, to optimize REN at the same time, we define the update of REN as another RL process with reward R-b . We observe that R-b is too small resulting in a weak gradient of REN, so the hyperparameter γ is set to enhance the gradient, and similarly, REN can be updated by a reinforcement learning algorithm with the following loss function:

The Label Retrieval Network (RN) is also constructed based on the reinforcement learning reward mechanism. In order to enhance the indicator-based reward (CIDEr) and utilize labels and other unmatched labels, a label retrieval network is introduced so that the generated captions should match their corresponding labels . Following a method called cross-media retrieval proposed by FartashFaghri et al., we reconstruct a sentence retrieval model with two LSTM networks. First, the RN is pre-trained with different labels of images to converge, since each image has five Different labels, we encode labels and generate captions for features in the same embedding space of RN:

s _i =LSTM(C _i ) (13)

g _j =LSTM(G _j ) (14)

where C and G represent the generated captions and labels, and S _i and _gi represent their respective embedding features. Compute the cosine similarity of the similarity between S and g:

指定匹配单词对的得分高于任何不匹配单词对的得分，RN的损失是通过铰链损失来计算的：

Specifying that matching word pairs have a higher score than any unmatched word pairs, the RN loss is computed via the hinge loss:

是正确的单词对，而

是不正确的。CIDEr的铰链损失在RL训练中充当句子级奖励，这鼓励字幕模型的生成字幕与给定的标签最佳匹配。in

is the correct word pair, and

is incorrect. The hinge loss of CIDEr acts as a sentence-level reward in RL training, which encourages the caption model to generate captions that best match the given labels.

Equation (17) is the loss function for β-optimized captioning models with sentence-level rewards, where β is a hyperparameter for balancing hinge loss and CIDEr. It is worth noting that the retrieval process is performed in each mini-batch, as it is time-consuming to retrieve the entire dataset.

The multi-attention fusion network based on the multi-granularity reward mechanism proposed by the present invention includes a multi-attention fusion model (HAF), a word importance re-evaluation network (REN) and a label retrieval network (RN).

Finally, the training method of the multi-attention fusion network based on the multi-granularity reward mechanism is as follows:

All models are pretrained with a cross-entropy loss and then trained to maximize different RL rewards. The encoder uses a pre-trained Resnet-101 to obtain the representation of the image, and for each image, we extract the outputs of the conv4 and conv5 convolutional layers from the Resnet, which are mapped to vectors of dimension 1024 as the input of the HAF. For HAF, the dimension of image feature embedding, the dimension of LSTM hidden state and word embedding are all set to 512. The baseline model is trained under the XE objective using the ADAM optimizer with an initial learning rate of 10 ⁻⁴ . At each iteration, we evaluate the model and select the best CIDEr as the baseline score. Reinforcement training starts at the 30th iteration epoch to optimize the CIDEr metric with a learning rate of 10 ⁻⁵ .

In the word-level reward training phase, the image captioning model is pre-trained with CIDEr rewards for 20 iterations, and reward-level rewards for 10 iterations. In sentence-level reward training, the RN is pre-trained for 10 epochs with different labels for each img. where the word embedding and LSTM hidden size are set to 512 and the joint embedding size is set to 1024, and the hyperparameter edge α is set to 0.2. In addition, the subtitle model of baseline (b) is trained for 30 epochs using cross-entropy, and the iteration epoch for sentence-level reward training is set to 30.

Compared with the prior art, the beneficial effects of the present invention are:

1. The present invention proposes a Hierarchical Attention Fusion (HAF) model as a baseline for image captioning RL training. HAF pays attention to the hierarchical visual features of CNN for many times, which can make full use of multi-level visual information.

2. The present invention proposes a word importance re-evaluation network (REN) for facilitating re-evaluation reward computation, which automatically assigns different importance to words generated in a sentence during the RL training phase.

3. The present invention proposes a label retrieval network (RN) to obtain sentence-level retrieval rewards. RN drives the generated captions to tend to match their corresponding labels rather than other sentences.

Description of drawings

Figure 1 is a schematic diagram of the structure of a multi-attention fusion network based on a multi-granularity reward mechanism.

Figure 2 is a schematic diagram of the Hierarchical Attention Fusion (HAF) model.

Figure 3 is a schematic diagram of the structure of the word importance re-evaluation network (REN).

FIG. 4 is a schematic diagram of the structure of a label retrieval network (RN).

Figure 5 is a comparison diagram of the subtitles generated by the multi-attention fusion network based on the multi-granularity reward mechanism and the subtitles generated by the top-down method, the subtitles generated by the hierarchical attention fusion model alone, and the real subtitles.

Detailed ways

The drawings are for illustrative purposes only and should not be construed as limiting the patent.

The present invention will be further elaborated below in conjunction with the accompanying drawings and embodiments.

Figure 1 is a schematic diagram of the structure of a multi-attention fusion network based on a multi-granularity reward mechanism. As shown in Figure 1, sentence-level reward and word-level reward, respectively, on the left, word-level reward is generated by adaptively re-evaluating the importance of words, and on the right, sentence-level reward is composed of retrieval loss, where retrieval The loss is calculated by the retrieval similarity S.

表示conv4和conv5的平均特征，X是输入字的one-hot编码，E是词汇表的词嵌入矩阵。我们采用的是一个经典网络结构，它根据每个时间步t的LSTM隐藏状态h_t产生归一化注意权重α_t，α_t用于参与图像特征的不同空间Att作为图像的最终表示(A)：Figure 2 is a schematic diagram of the Hierarchical Attention Fusion (HAF) model. as shown in picture 2,

represents the average feature of conv4 and conv5, X is the one-hot encoding of the input word, and E is the word embedding matrix of the vocabulary. We adopt a classical network structure that generates normalized attention weights α _t according to the LSTM hidden state h _t at each time step t, α _t is used to participate in different spaces of image features Att as the final representation of the image (A) :

α _t =softmax(at ₎ (2)

是学习参数。Among them, W _a , U _a ,

is the learning parameter.

where h2 is the output of the second LSTM, which consists of the image information of the convolutional layer and the content of the generated sequence. The process of producing h2 can be given by:

Finally, the probability of the output word is given by a non-linear softmax function:

Figure 3 is a schematic diagram of the structure of the word importance re-evaluation network (REN). As shown in Figure 3, the word importance re-evaluation network embeds the generated sentences and provides reward weights W, S is the sigmoid, and rl-model is the caption model pretrained by CIDEr. First, REN takes the generated sentence S as input, then, the sentence is processed by the RNN with attention network and average pooling layer, the word embedding vector is composed of the sentence embedding vector with attention and the sentence embedding vector after pooling are connected as a comprehensive representation for generating subtitles, and then two fully connected layers and sigmoid transform are applied to obtain the weights W _t of different words. Mathematically, the loss function can be formalized as 11:

Wr _t =RW _t +Rb (10)

表示生成的句子的不同单词。where Wi is the output weight of _REN , θ is the parameter of the image captioning network,

Represent different words of the generated sentence.

To exploit the metric-based reward (CIDEr) and constrain the sentence space, word-level rewards are employed to fine-tune the captioning network after CIDEr optimization. Furthermore, to simultaneously optimize REN, we define the update of REN as another RL process with reward R-b. We observe that R-b is too small resulting in a weak gradient of REN, so the hyperparameter γ is set to enhance the gradient. Similarly, REN can be updated by a reinforcement learning algorithm with the following loss function:

FIG. 4 is a schematic diagram of the structure of a label retrieval network (RN). As shown in Figure 4, through text-to-text retrieval, leveraging labels and unmatched labels to compose sentence-level rewards for RL training, we encode labels and generate captions for features in the same embedding space of RN:

s _i =LSTM(C _i ) (13)

g _j =LSTM(G _j ) (14)

where C and G represent the generated captions and labels, S _i and _gi represent their respective embedded features, and calculate the cosine similarity of the similarity between S and g:

指定匹配的单词对的得分高于任何不匹配的单词对的得分，RN的损失是通过铰链损失来计算的：

Specifying that matched word pairs have a higher score than any unmatched word pairs, the RN loss is computed via the hinge loss:

是正确的单词对，而

是不正确的单词对。CIDEr的铰链损失在RL训练中充当句子级奖励，这鼓励字幕模型的生成字幕与给定的标签最佳匹配。in

is the correct word pair, and

is an incorrect word pair. The hinge loss of CIDEr acts as a sentence-level reward in RL training, which encourages the caption model to generate captions that best match the given labels.

Equation (17) is the loss function used to β-optimize the caption model with sentence-level rewards, where β is a hyperparameter used to balance the hinge loss and CIDEr. It is worth noting that the retrieval process is performed in each mini-batch (small batches), as it is time-consuming to retrieve the entire dataset.

Figure 5 is a comparison diagram of the subtitles generated by the multi-attention fusion network based on the multi-granularity reward mechanism and the subtitles generated by the top-down method, the subtitles generated by the hierarchical attention fusion model alone, and the real subtitles. As shown in Figure 5, the sentences generated by the multi-attention fusion network based on the multi-granularity reward mechanism are more accurate and human-friendly than other models in the figure.

The invention proposes a word importance re-evaluation network and a label retrieval network based on a reinforcement learning reward mechanism, and on this basis, a multi-attention fusion network-based image caption generation method based on a multi-granularity reward mechanism is proposed. The network framework includes a A multi-attention fusion model (HAF), a word importance re-evaluation network (REN), and a label retrieval network (RN). The present invention proposes a Hierarchical Attention Fusion (HAF) model as the baseline for image captioning RL training. HAF pays attention to the hierarchical visual features of CNN for many times, and can make full use of multi-level visual information. At the same time, the word importance re-evaluation network ( REN) is used to facilitate re-evaluation reward computation, which automatically assigns different importance to words generated in sentences during the RL training phase. The Label Retrieval Network (RN) encourages the generated captions to match their corresponding labels rather than other sentences. Through training, the generated image captions can be expressed accurately and fluently, and can well reflect the content of the images.

Finally, the details of the above-mentioned examples of the present invention are only examples for explaining the present invention. For those skilled in the art, any modification, improvement and replacement of the above-mentioned embodiments should be included in the protection scope of the claims of the present invention. within.

Claims (6)

1. The method for generating the image captions of the multi-attention fusion network based on the multi-granularity reward mechanism is characterized by comprising the following steps:

s1, constructing a multi-attention fusion model.

And S2, constructing a word importance re-evaluation network based on a reinforcement learning reward mechanism.

And S3, constructing a label retrieval network by combining a reinforcement learning reward mechanism.

And S4, combining the model in S1, the word importance re-evaluation network in S2 and the label retrieval network in S3 to construct a multi-attention fusion network architecture based on a multi-granularity reward mechanism.

And S5, training and subtitle generation of a multi-attention fusion network based on a multi-granularity reward mechanism.

2. The method for generating image captions based on a multi-granular incentive mechanism and a multi-attention fusion network according to claim 1, wherein the specific process of S1 is as follows:

a classical network structure is adopted, which is based on LSTM hidden state h of each time step t_tGenerating a normalized attention weight α_t。α_tThe different spaces Att for participating in the image features as the final representation (a) of the image:

α_t＝softmax(a_t) (2)

wherein, W_a，U_a，

Are learning parameters.

Wherein h is²Is the output of the second LSTM, which consists of the image information of the convolutional layer and the content of the generated sequence. Generation of h²The process of (a) can be given by:

finally, the probability of the output word is given by the non-linear softmax function:

3. the method for generating image captions based on a multi-granular incentive mechanism and a multi-attention fusion network according to claim 1, wherein the specific process of S2 is as follows:

REN takes the generated sentence S as input, then the sentence is processed by RNN with attention network and average pooling layer, the word embedding vector is formed by connecting the sentence embedding vector with attention and the sentence embedding vector after pooling as comprehensive representation of generating caption, then two full connection layers and sigmoid transformation are applied to obtain the weight W of different words_t. Mathematically, the loss function can be formalized as (11):

Wr_t＝RW_t+R-b (10)

wherein, W_iIs the output weight of REN, theta is a parameter of the image caption network,

different words representing the generated sentence.

To take advantage of index-based rewards (CIDER) and constrain sentence space, after CIDER optimization, word-level rewards are employed to fine tune the caption network, and furthermore, to optimize REN simultaneously, we define the update of REN as another RL process with reward R-b. We observe that R-b is too small resulting in a weaker gradient for REN, so the hyper-parameter γ is set to enhance the gradient. Similarly, REN may be updated by a reinforcement learning algorithm with the following loss function:

4. the method for generating image captions based on a multi-granular incentive mechanism and a multi-attention fusion network according to claim 1, wherein the specific process of S3 is as follows:

the RN is pre-trained to converge with different labels for the images because each image has five different labels. We encode the tags and generate subtitles for the features in the same embedding space of the RN:

s_i＝LSTM(C_i) (13)

g_j＝LSTM(G_j) (14)

where C and G denote the generated subtitles and tags, S_iAnd g_iIndicating their respective embedded characteristics. Calculating cosine similarity of similarity between S and g:

the score of the designated matching word pair is higher than the score of any unmatched word pair, and the penalty of the RN is calculated by the hinge penalty:

wherein

Is the correct word pair, and

is incorrect. Hinge loss of CIDER acts as a sentence-level reward in RL training, which encourages the generation of captions for the caption model to best match a given tag.

Equation (17) is a loss function for beta optimizing the caption model by sentence-level rewards, where beta is a hyper-parameter for balancing hinge loss and CIDEr. It is noted that the retrieval process is performed in each mini-batch, since retrieval is time consuming in the entire data set.

5. The method for generating image captions based on a multi-granular incentive mechanism and a multi-attention fusion network according to claim 1, wherein the specific process of S4 is as follows:

the multi-attention fusion network based on the multi-granularity reward mechanism comprises a multi-attention fusion model (HAF), a reevaluation network (REN) and a Retrieval Network (RN).

6. The method for generating image captions based on a multi-granular incentive mechanism and a multi-attention fusion network according to claim 1, wherein the specific process of S5 is as follows:

the training and training method of the multi-attention fusion network based on the multi-granularity reward mechanism comprises the following steps:

all models were pre-trained with cross-entropy loss and then trained to maximize different RL rewards. The encoder uses the pre-trained Resnet-101 to obtain a representation of the images, and for each image we extract the outputs of the conv4 and conv5 convolutional layers from Resnet, which map to a vector of dimension 1024 as the input to the HAF. For the HAF, the image feature embedding dimension, LSTM hidden state and word embedding dimension are all set to 512. The baseline model was trained using an ADAM optimizer at XE goal with an initial learning rate of 10^-4. At each iteration cycle, we evaluate the model and select the best CIDER as the baseline score. The reinforced training starts from the 30 th iteration cycle to optimize the CIDER measurement, and the learning rate is 10^-5。

In the word-level reward training phase, the image subtitle model trains the CIDER reward of 20 iteration cycles and the reward level reward of 10 iteration cycles in advance. In sentence-level reward training, RN trains 10 iteration cycles in advance with different real labels per img, where word embedding and LSTM hidden sizes are set to 512 and joint embedding size is set to 1024, and the hyper-parameter edge α is set to 0.2. In addition, the subtitle model for baseline (b) was trained using cross entropy for 30 epochs, with the iteration period for sentence-level reward training set to 30.

Images