WO2020186484A1 - Automatic image description generation method and system, electronic device, and storage medium - Google Patents

Abstract

Disclosed are an automatic image description generation method and system, an electronic device, and a storage medium, for generating a text description for an image, and resolving the prior art issue in which a trained model does not meet an assessment criteria. The method comprises: using a loss function, a mobilenet convolutional neural network, and a long short-term memory network to construct an automatic image description generation model; recording a network parameter during construction of the model; inputting an image to the model; and the model outputting, according to the inputted image and the network parameter, a piece of text for describing the image. The automatic image description generation method provided by the present application develops a novel loss function by means of reinforced learning, such that model training standards meet the assessment criteria.

Description

Method, system, electronic device and storage medium for automatically generating and describing pictures Technical field

The present invention relates to the technical field of picture processing, and in particular to a method, system, electronic device and storage medium for automatically generating and describing pictures.

Background technique

Since the beginning of the 21st century, the Internet storage level and computer computing power have experienced a huge leap. The number of smartphone users has also increased to a large extent. Users use mobile terminals, PCs and other intelligent devices, every day Share a lot of picture data on the Internet. These huge picture data resources can more accurately reflect the objective world. The different visual changes they produce contain rich semantic information and provide a sufficient source of information for perceiving the real world.

With the development of neural networks and deep learning, image understanding has developed from primitive image processing based on low-level visual features to high-level image semantic information and semantic understanding. The existing neural network model is generally an encoder-decoder model, that is, features are extracted through CNN (convolutional neural network), and the extracted features are used as the initial state of LSTM (long and short-term memory neural network), and a segment can be generated by LSTM to describe The text of the image.

technical problem

Existing models mainly use the cross-entropy loss function when training. After the model training is completed, it is necessary to use evaluation indicators such as BLUE to evaluate the model. However, for models trained with the cross-entropy loss function, there are common training standards and The problem of inconsistent evaluation standards.

Technical solutions

The first aspect of the present invention provides a method for automatically generating descriptions of pictures, including: using a loss function, a mobilenet convolutional neural network, and a long and short-term memory neural network to construct a model for automatically generating descriptions of pictures; recording network parameters when constructing the model; The picture is input into the model; the model outputs a paragraph of text that can describe the picture according to the input picture and the network parameters.

Beneficial effect

Using the mobilenet convolutional neural network can extract image features, and the amount of parameters and calculations are small, which can effectively improve efficiency. After using the long- and short-term memory neural network, it can generate text information of image features, and use the loss function as a whole to reverse To propagate network parameters to update and improve the model, so that the data of the trained model can be more accurate when generating the picture description, thereby increasing the unity of the trained model and the evaluation standard.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic block diagram of the process of a method for automatically generating a description of pictures according to an embodiment of the present invention;

2 is a schematic diagram of matrix decomposition of the Mobilenet model in the method for automatically generating and describing pictures according to an embodiment of the present invention;

3 is a schematic block diagram of the structure of a system for automatically generating and describing pictures according to an embodiment of the present invention;

4 is a schematic block diagram of the structure of an electronic device according to an embodiment of the present invention.

The best mode of the invention

The first aspect of the present invention provides a method for automatically generating descriptions of pictures, including: using a loss function, a mobilenet convolutional neural network, and a long and short-term memory neural network to construct a model for automatically generating descriptions of pictures; recording network parameters when constructing the model; The picture is input into the model; the model outputs a paragraph of text that can describe the picture according to the input picture and the network parameters.

Further, the use of loss function, mobilenet convolutional neural network and long-short-term memory neural network to construct a model for automatically generating descriptions of pictures includes: adopting the idea of reinforcement learning to design and construct a loss function; using pre-trained mobilenet convolutional neural network to extract Picture feature vector; use matrix transformation to join the attention mechanism to fuse the feature vector with the original state vector extracted from the pre-trained long and short-term memory neural network to obtain a new feature vector; input the new feature vector into the long and short time A memory neural network, the long-short-term memory neural network generates text with picture information according to the new feature vector; derivates the loss function, and inputs the text into the derivated loss function; judges the Whether the loss function converges after receiving the text; if the loss function converges, save the network parameters during training; if the loss function does not converge, continue to use the mobilenet convolutional neural network, matrix transformation, and long and short-term memory The network generates text with picture information, and inputs the text into the derived loss function, until after the text is input into the loss function, the loss function converges.

Further, said extracting the picture feature vector using the pre-trained mobilenet convolutional neural network includes: after inputting the picture into the pre-trained mobilenet convolutional neural network, saving the output characteristics of the average pooling layer of the mobilenet convolutional neural network vector.

The second aspect of the present invention provides a system for automatically generating descriptions of pictures, including: a model training module for building a model for automatically generating descriptions of pictures using loss function, mobilenet convolutional neural network and long-short-term memory neural network; network parameter recording module , Used to record the network parameters when constructing the training model of the model training module; the picture receiving module, used to receive pictures input to the model trained by the model training module; the text generation module, used to make the model according to the picture The picture received by the receiving module and the network parameters recorded by the network parameter recording module output a paragraph of text that can describe the picture.

Further, the neural network pre-training module includes: a neural network construction unit, used to construct a mobilenet convolutional neural network; a neural network parameter update unit, used to construct the neural network construction unit based on an existing image data set The mobilenet convolutional neural network is pre-trained and the parameters of the mobilenet convolutional neural network are updated.

Further, the model training module includes: a loss function construction unit for designing and constructing a loss function using the idea of reinforcement learning; a feature vector extraction unit for extracting feature vectors of sample images using a pre-trained mobilenet convolutional neural network The attention mechanism introduction unit is used to introduce the attention mechanism using matrix transformation to fuse the feature vector with the original state vector extracted from the pre-trained long and short-term memory neural network to obtain a new feature vector; text generation unit, The new feature vector obtained by introducing the attention mechanism into the unit is used to input the long and short-term memory neural network, which generates text with picture information according to the new feature vector; the loss function derivation unit is used for Derivation of the loss function constructed by the loss function construction unit, and input the text into the derivative loss function; a loss function convergence judging unit for judging the loss after the derivation of the loss function derivation unit Whether the function is converged; a network parameter storage unit for saving the network parameters during training after the loss function convergence judging unit judges the convergence of the loss function; a loop unit for judging the loss function convergence judgment unit for the loss function convergence judging unit After that, continue to use the feature vector extraction unit, the attention mechanism introduction unit, and the text generation unit to generate text with picture information, and input the text into the loss function derivation unit to obtain the derivation loss In the function, the loss function converges until the text is input into the loss function.

Further, the feature vector extraction unit includes: a pooling layer output saving subunit for saving the output of the average pooling layer of the mobilenet convolutional neural network after the picture is input into the pre-trained mobilenet convolutional neural network Feature vector.

A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and capable of being run on the processor, wherein the processor executes the computer program When, implement any of the above methods.

A fourth aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the method described in any one of the above is implemented.

Embodiments of the invention

In order to make the objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the description The embodiments are only a part of the embodiments of the present invention, but not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present invention.

Please refer to Figure 1, a method for automatically generating descriptions for pictures, including: S1, using loss function, mobilenet convolutional neural network and long-short-term memory neural network to build a model for automatically generating descriptions for pictures; S2, recording the network when building the model Parameters; S3, input the picture into the model; S4, the model outputs a paragraph of text that can describe the picture according to the input picture and network parameters.

The use of loss function, mobilenet convolutional neural network and long-short-term memory neural network to build a model to automatically generate descriptions for pictures includes: adopting the idea of reinforcement learning to design and construct a loss function; use pre-trained mobilenet convolutional neural network to extract image feature vectors; use Matrix transformation is added to the attention mechanism to fuse the feature vector with the original state vector extracted from the pre-trained long and short-term memory neural network to obtain a new feature vector; input the new feature vector into the long- and short-term memory neural network, and the long and short-term memory neural network Generate text with picture information according to the new feature vector; derivate the loss function and input the text into the derivation loss function; determine whether the loss function converges after receiving the text; if the loss function converges, save the training time Network parameters; if the loss function does not converge, continue to use mobilenet convolutional neural network, matrix transformation and long-short-term memory neural network to generate text with picture information, and input the text into the derivation of the loss function until the text is input After the loss function is in, the loss function converges.

Using the pre-trained mobilenet convolutional neural network to extract the image feature vector includes: inputting the image into the pre-trained mobilenet convolutional neural network, and saving the feature vector output by the average pooling layer of the mobilenet convolutional neural network.

Mobilenet focuses on compressing the model while ensuring accuracy. The idea is to decompose a standard convolution into a deep convolution and a 1x1 ordinary convolution (also called pointwise convolution). A simple understanding is the factorization of the matrix. The specific steps are shown in Figure 2. Suppose that the size of the input image feature vector is DF * DF, the dimension is M, the size of the filter is DK * DK, the dimension is N, and the step size is assumed to be 1. In the original convolution operation, the number of matrix operations that need to be performed is DK*DK*M*N*DF*DF, and the convolution kernel parameter is DK *DK *N. The number of matrix operations required for convolution in Mobilenet is DK*DK*M*DF*DF + M *N *DF*DF, and the convolution kernel parameter is DK *DK *M+N. Since the process of convolution is mainly a process of reducing the spatial dimension and increasing the channel dimension, that is, N>M, so DK *DK *N> DK *DK *M+N. Therefore, the depth separable convolution performs a lot of compression in both the model size and the model calculation amount, so that the model is fast, the calculation overhead is low, and the accuracy is good.

Use the new state vector as the input of the next state of the long and short-term memory neural network, and use reinforcement learning and back propagation methods to retrain the pre-trained long and short-term memory neural network to obtain a new long and short-term memory neural network including: using pre-training The long-short-term memory neural network uses the sentence generated by the picture as the baseline, and the sentence composed of randomly selected words as the reward, and uses the baseline to subtract the reward to construct the loss function; the loss function and the new state vector are used, and the length is again calculated through backpropagation The time memory neural network is trained to obtain a new long and short time memory neural network, and the training parameters of the new long and short time memory neural network are updated.

Regarding the reinforcement learning method: regard the sequence problem as a reinforcement learning problem:

Agent: LSTM;

Environmental status: word and picture features;

Action: predict the next word;

State: the state of LSTM unit and hidden layer;

Reward: CIDEr score;

The training goal is to minimize the negative expectation L(θ):

L(θ)=−Ews∼pθ[r(ws)]L(θ)=−Ews∼pθ[r(ws)]=−∑r(ws)pθ(ws)=−∑r(ws)pθ( ws);

ws=(ws1,...,wsT)ws=(w1s,...,wTs) is the generated sentence.

In fact, ws can be randomly selected based on the probability of pθ (rather than choosing the one with the highest probability), L(θ) can be approximated as:

L(θ)≈−r(ws),ws∼pθ;

The gradient of L with respect to θ is:

∇θL(θ)=−Ews∼pθ[r(ws)∇θlogpθ(ws)];

Then introduce a baseline to reduce the variance:

∇θL(θ)=−Ews∼pθ[(r(ws)−b)∇θlogpθ(ws)];

The baseline can be any function, as long as it does not depend on the action ws, introducing it will not change the value of the gradient.

In fact, L(θ) can be approximated as:

∇θL(θ)≈−(r(ws)−b)∇θlogpθ(ws);

Applying the chain rule, the gradient can be expressed as:

∂L(θ)/∂st≈(r(ws)−b)(pθ(wt|ht)−1wst);

The idea of reinforcement learning is to use the reward of the word generated by the current model in the testing phase as the baseline, and the gradient becomes:

∂L(θ)/∂st≈(r(ws)−r(w^))(pθ(wt|ht)−1wst);

Among them, r(w^)=argmax _wt p(wt|ht) is to use greedy decoding to take the word with the highest probability to generate a sentence in the test phase; and r(ws) is a word randomly selected according to the probability, if the current probability The probability of the largest word is 60%, so there is a 60% probability of picking it, instead of 100% picking the highest probability like greedy decoding.

The meaning of the formula is: if the current random word is better than the word generated in the test phase, then in the dimension of the word this time, the value of the entire formula is negative (because the latter item must be negative), so the gradient It will rise, thereby increasing the score of this word; and for other words, the latter one is positive, the gradient will drop, thereby reducing the scores of other words.

Please refer to Figure 3, which is a system for automatically generating descriptions of pictures, including: model training module 1, network parameter recording module 2, picture receiving module 3, and text generation module 4; model training module 1 is used to use loss function and mobilenet volume Product neural network and long-short-term memory neural network build a model for automatically generating descriptions of pictures; network parameter recording module 2 is used to record network parameters when building model training module 1 training model; picture receiving module 3 is used to receive input model training module 1 training A picture of the model; the text generation module 4 is used to make the model output a paragraph of text that can describe the picture according to the picture received by the picture receiving module 3 and the network parameters recorded by the network parameter recording module 2.

Model training module 1 includes: loss function construction unit, feature vector extraction unit, attention mechanism introduction unit, text generation unit, loss function derivation unit, loss function convergence judgment unit, network parameter storage unit and loop unit; loss function construction unit It is used to design and construct the loss function using the idea of reinforcement learning; the feature vector extraction unit is used to extract the feature vector of the sample picture using the pre-trained mobilenet convolutional neural network; the attention mechanism introduction unit is used to introduce the attention mechanism using matrix transformation, The feature vector is fused with the original state vector extracted from the pre-trained long and short-term memory neural network to obtain a new feature vector; the text generation unit is used to introduce the attention mechanism into the unit to obtain the new feature vector into the long and short-term memory neural network, The long and short-term memory neural network generates text with picture information according to the new feature vector; the loss function derivation unit is used to derive the loss function constructed by the loss function construction unit, and input the text into the derivated loss function; loss function The convergence judgment unit is used to judge whether the loss function after the derivation of the loss function derivation unit has converged; the network parameter storage unit is used to save the network parameters during training after the loss function convergence judgment unit judges the convergence of the loss function; the loop unit is used in the After the loss function convergence judgment unit judges that the loss function does not converge, it continues to use the feature vector extraction unit, attention mechanism introduction unit and text generation unit to generate text with picture information, and input the text to the derivation obtained by the loss function derivation unit In the loss function, until the text is input into the loss function, the loss function converges.

The feature vector extraction unit includes: a pooling layer output saving subunit, which is used to save the feature vector output by the average pooling layer of the mobilenet convolutional neural network after the picture is input into the pre-trained mobilenet convolutional neural network.

An embodiment of the present application provides an electronic device. Please refer to 4. The electronic device includes: a memory 601, a processor 602, and a computer program stored in the memory 601 and running on the processor 602, and the processor 602 executes the computer program At the time, implement the method of automatically generating descriptions for pictures described in the foregoing.

Further, the electronic device further includes: at least one input device 603 and at least one output device 604.

The aforementioned memory 601, processor 602, input device 603, and output device 604 are connected via a bus 605.

Among them, the input device 603 may specifically be a camera, a touch panel, a physical button or a mouse, etc. The output device 604 may specifically be a display screen.

The memory 601 may be a high-speed random access memory (RAM, Random Access Memory) memory can also be non-volatile memory (non-volatile memory), such as disk memory. The memory 601 is used to store a group of executable program codes, and the processor 602 is coupled with the memory 601.

Further, the embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium may be the electronic device provided in each of the foregoing embodiments, and the computer-readable storage medium may be the aforementioned memory. 601. A computer program is stored on the computer-readable storage medium, and when the program is executed by the processor 602, the method for automatically generating a picture described in the foregoing embodiment is realized.

Further, the computer storage medium may also be a U disk, a mobile hard disk, a read-only memory 601 (ROM, Read-Only Memory), RAM, magnetic disks or optical disks and other media that can store program codes.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in each embodiment of the present invention may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules.

It should be noted that for the foregoing method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described sequence of actions. Because according to the present invention, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the involved actions and modules are not necessarily all required by the present invention.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

The above is a description of the method, system, electronic device, and storage medium for automatically generating descriptions of pictures provided by the present invention. For those skilled in the art, based on the ideas of the embodiments of the present invention, both in terms of specific implementation and scope of application There will be changes. In summary, the content of this specification should not be construed as limiting the present invention.

Industrial applicability

The invention provides a method, system, electronic device and storage medium for automatically generating and describing pictures, which solves the technical problem that the training model and the evaluation standard are not uniform in the prior art.

Claims (8)

A method for automatically generating descriptions of pictures, characterized in that it includes: Use loss function, mobilenet convolutional neural network and long-short-term memory neural network to build a model that automatically generates descriptions for pictures; Record the network parameters when constructing the model; Input pictures into the model; The model outputs a paragraph of text that can describe the picture according to the input picture and the network parameters. The method for automatically generating a description of a picture according to claim 1, wherein: The use of loss function, mobilenet convolutional neural network and long and short-term memory neural network to construct a model for automatically generating descriptions of pictures includes: Design and construct the loss function with the idea of reinforcement learning; Use pre-trained mobilenet convolutional neural network to extract image feature vectors; Use matrix transformation to join the attention mechanism to fuse the feature vector with the original state vector extracted from the pre-trained long and short-term memory neural network to obtain a new feature vector; Inputting the new feature vector into a long and short-term memory neural network, which generates text with picture information according to the new feature vector; Derivative of the loss function, and input the text into the derived loss function; Determine whether the loss function converges after receiving the text; If the loss function converges, save the network parameters during training; If the loss function does not converge, continue to use the mobilenet convolutional neural network, matrix transformation and long-short-term memory neural network to generate text with picture information, and input the text into the derivation of the loss function until the text After inputting the loss function, the loss function converges. The method for automatically generating descriptions for pictures according to claim 2, characterized in that, The extraction of image feature vectors using a pre-trained mobilenet convolutional neural network includes: After inputting the picture into the pre-trained mobilenet convolutional neural network, save the feature vector output by the average pooling layer of the mobilenet convolutional neural network. A system for automatically generating descriptions of pictures, characterized in that it includes: Model training module, used to use loss function, mobilenet convolutional neural network and long and short-term memory neural network to build a model for automatically generating descriptions of pictures; The network parameter recording module is used to record the network parameters when constructing the training model of the model training module; The picture receiving module is configured to receive pictures of the model trained by the model training module; The text generation module is used to enable the model to output a paragraph of text that can describe a picture based on the picture received by the picture receiving module and the network parameters recorded by the network parameter recording module. The system for automatically generating descriptions of pictures according to claim 4, wherein: The model training module includes: The loss function construction unit is used to design and construct the loss function using the idea of reinforcement learning; The feature vector extraction unit is used to extract the feature vector of the sample picture using the pre-trained mobilenet convolutional neural network; An attention mechanism introduction unit for introducing an attention mechanism using matrix transformation to fuse the feature vector with the original state vector extracted from the pre-trained long and short-term memory neural network to obtain a new feature vector; A text generation unit, configured to input the new feature vector obtained by the attention mechanism introduction unit into a long and short-term memory neural network, and the long and short-term memory neural network generates text with picture information according to the new feature vector; A loss function derivation unit, configured to derivate the loss function constructed by the loss function construction unit, and input the text into the derivated loss function; A loss function convergence judging unit for judging whether the loss function after the derivation of the loss function derivation unit converges; The network parameter saving unit is configured to save the network parameters during training after the loss function convergence judging unit judges the convergence of the loss function; The loop unit is configured to continue to use the feature vector extraction unit, the attention mechanism introduction unit, and the text generation unit to generate text with picture information after the loss function convergence judgment unit judges that the loss function does not converge, and The text is input into the derivated loss function obtained by the loss function derivation unit, and the loss function converges after the text is input into the loss function. The system for automatically generating descriptions of pictures according to claim 5, wherein: The feature vector extraction unit includes: The pooling layer output saving subunit is used to save the feature vector output by the average pooling layer of the mobilenet convolutional neural network after the picture is input into the pre-trained mobilenet convolutional neural network. An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor implements claim 1 when the processor executes the computer program To the method described in any one of 3. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in any one of claims 1 to 3 when the computer program is executed by a processor.