CN111753859A - Sample generation method, device and device - Google Patents

Abstract

The present invention provides a sample generation method, device and device. The sample generation method includes: acquiring a feature description vector of a specified standard word, where the feature description vector is used to indicate the content of the specified standard word; using the feature description vector and the specified standard word The non-standard feature vector converts a specified standard word into a target sample, and the style corresponding to the target sample is the same as that represented by the non-standard feature vector. A sample of the font style can be generated without collecting character images of the required font style, which improves the sample generation efficiency.

Description

Sample generation method, device and device

technical field

The present invention relates to the technical field of image processing, and in particular, to a sample generation method, device and device.

Background technique

With the development of science and technology, deep learning algorithms perform well in tasks such as classification, detection, and recognition. However, the achievement of this performance depends on many factors such as the improvement of computer computing power and a large number of training samples. Among them, the training samples as "fuel" are an indispensable part of algorithm development. In text recognition technology, a large number of samples containing characters are also required for training.

In the related sample generation method, the character image is pasted to the background image to synthesize the sample. In the real scene, the fonts of the text characters are diverse. In order to enable the algorithm to more accurately identify the text characters in the real scene, it is necessary to generate Samples of various font styles required for training. In this method, each time a font style sample is required, character images of the corresponding font style need to be collected to synthesize the required samples, and the sample generation efficiency is too low.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a sample generation method, device and device, which can generate samples of the font style without collecting character images of the required font style, thereby improving the sample generation efficiency.

A first aspect of the present invention provides a sample generation method, comprising:

Obtain the feature description vector of the specified standard word, and the feature description vector is used to indicate the content of the specified standard word;

The specified standard word is converted into a target sample by using the feature description vector and the specified non-standard feature vector, and the style corresponding to the target sample is the same as the style represented by the non-standard feature vector.

According to an embodiment of the present invention, the obtaining the feature description vector of the specified standard word includes:

The first image containing the specified standard word is input to the first neural network in the trained student network, so that the first neural network performs feature extraction on the inputted first image to obtain a feature description vector.

According to an embodiment of the present invention, the first neural network performs feature extraction on the inputted first image to obtain a feature description vector, including:

The first neural network performs feature extraction on the first image through at least a convolution layer for performing convolution processing and a first nonlinear transformation layer for performing nonlinear transformation processing to obtain a feature description vector.

According to an embodiment of the present invention, using the feature description vector and the specified non-standard feature vector to convert a specified standard word into a target sample, including:

inputting the feature description vector and the non-standard feature vector into a second neural network in the trained student network to fuse the feature description vector with the non-standard feature vector by the second neural network obtaining a fusion vector, and using the fusion vector to generate a second image;

The second image is determined as the target sample.

According to an embodiment of the present invention, the second neural network includes a fusion layer; the feature description vector has the same dimension as the non-standard feature vector;

The second neural network fuses the feature description vector and the non-standard feature vector to obtain a fusion vector, including:

The second neural network uses a fusion layer to perform superposition processing on the feature description vector and the non-standard feature vector to obtain the fusion vector.

According to an embodiment of the present invention, the second neural network includes a fully connected layer and a fusion layer; the dimension of the feature description vector is different from that of the non-standard feature vector;

The second neural network fuses the feature description vector and the non-standard feature vector to obtain a fusion vector, including:

The second neural network utilizes a fully connected layer to map the non-standard feature vector into a reference vector whose dimension is the same as that of the feature description vector;

The second neural network uses a fusion layer to perform superposition processing on the feature description vector and the reference vector to obtain the fusion vector.

According to an embodiment of the present invention, the second neural network includes a fusion layer;

The second neural network fuses the feature description vector and the non-standard feature vector to obtain a fusion vector, including:

The second neural network uses a fusion layer to combine the feature description vector and the non-standard feature vector to obtain the fusion vector.

According to an embodiment of the present invention, the second neural network further includes: a deconvolution layer for performing deconvolution processing, and a second nonlinear transformation layer for performing nonlinear transformation;

The second neural network using the fusion vector to generate the second image includes:

The second neural network uses the deconvolution layer and the second nonlinear transformation layer to generate a second image corresponding to the fusion vector.

According to an embodiment of the present invention, the student network is trained under the supervision of a trained teacher network;

The network parameters of at least one layer in the second neural network apply the network parameters of the corresponding layer in the teacher network.

A second aspect of the present invention provides a sample generation device, comprising:

a feature description vector acquisition module, used to obtain a feature description vector of a specified standard word, and the feature description vector is used to indicate the content of the specified standard word;

The target sample generation module is used to convert the specified standard word into a target sample by using the feature description vector and the specified non-standard feature vector, and the style corresponding to the target sample is the same as the style represented by the non-standard feature vector.

According to an embodiment of the present invention, the feature description vector obtaining module is specifically used for:

The first image containing the specified standard word is input to the first neural network in the trained student network, so that the first neural network performs feature extraction on the inputted first image to obtain a feature description vector.

According to an embodiment of the present invention, the first neural network performs feature extraction on the inputted first image to obtain a feature description vector, including:

The first neural network performs feature extraction on the first image through at least a convolution layer for performing convolution processing and a first nonlinear transformation layer for performing nonlinear transformation processing to obtain a feature description vector.

According to an embodiment of the present invention, the target sample generation module includes:

An image generation unit for inputting the feature description vector and the non-standard feature vector to a second neural network in the trained student network, so that the second neural network combines the feature description vector with the The non-standard feature vectors are fused to obtain a fusion vector, and the fusion vector is used to generate a second image;

A target sample determination unit, configured to determine the second image as the target sample.

According to an embodiment of the present invention, the second neural network includes a fusion layer; the feature description vector has the same dimension as the non-standard feature vector;

When the second neural network fuses the feature description vector and the non-standard feature vector to obtain a fusion vector, it is specifically used for:

The second neural network uses a fusion layer to perform superposition processing on the feature description vector and the non-standard feature vector to obtain the fusion vector.

According to an embodiment of the present invention, the second neural network includes a fully connected layer and a fusion layer; the dimension of the feature description vector is different from that of the non-standard feature vector;

When the second neural network fuses the feature description vector and the non-standard feature vector to obtain a fusion vector, it is specifically used for:

The second neural network utilizes a fully connected layer to map the non-standard feature vector into a reference vector whose dimension is the same as that of the feature description vector;

The second neural network uses a fusion layer to perform superposition processing on the feature description vector and the reference vector to obtain the fusion vector.

According to an embodiment of the present invention, the second neural network includes a fusion layer;

When the second neural network fuses the feature description vector and the non-standard feature vector to obtain a fusion vector, it is specifically used for:

The second neural network uses a fusion layer to combine the feature description vector and the non-standard feature vector to obtain the fusion vector.

According to an embodiment of the present invention, the second neural network further includes: a deconvolution layer for performing deconvolution processing, and a second nonlinear transformation layer for performing nonlinear transformation;

When the second neural network uses the fusion vector to generate the second image, it is specifically used for:

The second neural network uses the deconvolution layer and the second nonlinear transformation layer to generate a second image corresponding to the fusion vector.

According to an embodiment of the present invention, the student network is trained under the supervision of a trained teacher network;

The network parameters of at least one layer in the second neural network apply the network parameters of the corresponding layer in the teacher network.

A third aspect of the present invention provides an electronic device, including a processor and a memory; the memory stores a program that can be called by the processor; wherein, when the processor executes the program, the implementation is as described in the foregoing embodiments sample generation method.

A fourth aspect of the present invention provides a machine-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, the sample generation method described in the foregoing embodiments is implemented.

The embodiment of the present invention has the following beneficial effects:

In the embodiment of the present invention, a feature description vector used to indicate the content of a specified standard word and a non-standard feature vector representing a certain style can be used to convert the specified standard word into a target sample of the style, without collecting data in the style. Character images are used to synthesize samples, which improves the efficiency of sample generation, and can generate samples with different text contents in various styles according to needs, so as to realize the diversity of samples.

Description of drawings

1 is a schematic flowchart of a sample generation method according to an embodiment of the present invention;

2 is a structural block diagram of a sample generating apparatus according to an embodiment of the present invention;

3 is a block diagram of a connection structure of a first neural network and a second neural network according to an embodiment of the present invention;

4 is a schematic diagram of a training method of the first neural network and the second neural network according to an embodiment of the present invention;

5 is a schematic diagram of another training method of the first neural network and the second neural network according to an embodiment of the present invention;

FIG. 6 is a structural block diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with some aspects of the invention as recited in the appended claims.

The terminology used in the present invention is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in this specification and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used herein to describe various devices, this information should not be limited by these terms. These terms are only used to distinguish devices of the same type from one another. For example, a first device could be termed a second device, and similarly, a second device could be termed a first device, without departing from the scope of the present invention. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining."

In order to make the description of the present invention clearer and concise, some technical terms in the present invention are explained below:

Neural network: a technology abstracted by imitating the structure of the brain, which connects a large number of simple functions intricately to form a network system, which can fit extremely complex functional relationships, generally including convolution/inverse Convolution operations, activation operations, pooling operations, and operations such as addition, subtraction, multiplication and division, channel merging, and element rearrangement. Training the network with specific input and output data, and adjusting the connections, allows the neural network to learn to fit the mapping between input and output.

The sample generation method according to the embodiment of the present invention will be described in more detail below, but it should not be limited thereto. In one embodiment, referring to FIG. 1, a sample generation method may include the following steps:

S100: Obtain a feature description vector C of a specified standard word, and the feature description vector C is used to indicate the content of the specified standard word;

S200: Using the feature description vector C and the specified non-standard feature vector S to convert the specified standard word into a target sample, where the style corresponding to the target sample is the same as the style represented by the non-standard feature vector S.

The execution body of the sample generation method in the embodiment of the present invention may be an electronic device, and more specifically, may be a processor of the electronic device. The electronic device may be, for example, a computer device or an embedded device, and the specific type is not limited, as long as it has data processing capability.

In step S100, a feature description vector C of a specified standard word is obtained, and the feature description vector C is used to indicate the content of the specified standard word.

The font of the specified standard word can be Song, Hei, etc. The specific font style is not limited, as long as the content of the specified standard word is the text content required by the sample. Before acquiring the feature description vector C of the specified standard word, the specified standard word may be acquired from the font library of the corresponding font. After the specified standard word is obtained, feature extraction can be performed on the specified standard word to obtain a feature description vector C that describes the content of the specified standard word.

Feature extraction can be performed on the specified standard words through a feature extraction algorithm. The feature extraction algorithm is not limited, such as LBP feature extraction algorithm, HOG feature extraction algorithm, SIFT feature extraction operator, etc., and deep learning can also be used to achieve feature extraction.

The specified standard word is any standard word in the specified font library. Generally speaking, a common font font library is configured by default in a computer device, or a common font font library can also be downloaded from the network, and the specified font library can be any one of the common font font libraries. Take the designated font library as an example of the Song-style font. The Song-style font library contains more than 20,000 Song-style characters, and the specified standard character can be any of the more than 20,000 Song-style characters, depending on the text content required by the sample. If a corresponding sample is generated for each of more than 20,000 Song-style characters by using the embodiment of the present invention, more than 20,000 samples containing different text contents in a desired style can be generated.

If there are N1 standard characters in the specified font library, N1 samples containing different text contents in the required style can be converted and obtained, and the text contents in each sample are the same as the standard characters but in different styles. Therefore, in the embodiment of the present invention, it is possible to easily generate a plurality of samples of the required style, which can overcome the problem of few font style samples at present, such as font styles in calligraphy works.

In step S200, the specified standard word is converted into a target sample by using the feature description vector C and the specified non-standard feature vector S, and the style corresponding to the target sample is the same as the style represented by the non-standard feature vector S.

The style represented by the non-standard feature vector S (referred to as the target style) can be some commonly used or uncommon calligraphy styles, such as font styles such as Hei, Liu Zongyuan, or Mi Fu, and even, the target style can also be a certain person handwritten font. Generally speaking, there are some differences in everyone's writing style, and everyone's writing style can be used as a target style.

A variety of non-standard feature vectors representing different styles can be preset, and the non-standard feature vector S is one of them. If the total number of preset non-standard feature vectors is N2, if the conversion of the specified standard words is performed for each non-standard feature vector, N2 different styles of target samples containing the same text content can finally be generated, and the samples are more diverse. , which can reflect more real scenes. Using these samples to train the neural network can make the neural network text recognition results more accurate.

The encoding form of each non-standard feature vector is not limited, for example, encoding can be performed according to the total number of styles N2, and one-hot encoding (one-hot encoding) can be used to encode the vector.

Taking one-hot encoding as an example, it is assumed that there are N2 styles to be generated, such as Hei, Liu Zong Yuan, Mi Fu, etc. When a sample of Liu Zong Yuan needs to be generated, in the non-standard feature vector S, only the dimension corresponding to Liu Zong Yuan can be used. The value on the (2nd dimension) is coded as 1, and the value on the other dimensions is coded as 0, and the final non-standard feature vector S=[0,1,0…0]; when it is necessary to generate samples of Mi Fu body, non-standard In the feature vector S, only the numerical value on the corresponding Mi Fu body dimension (the third dimension) can be coded as 1, and the numerical value on the other dimensions can be coded as 0, and the final non-standard feature vector S=[0,0,1...0]; The same can be said for other styles.

Combined with the foregoing, in the case where there are N1 standard words in the specified font font library and N2 non-standard feature vectors are encoded, the total number of target samples that can be generated is the product of N1 and N2, where the total number of target samples is the product of N1 and N2. The total number of styles is N2, each style has N1 target samples and each target sample contains different text content.

In this embodiment of the present invention, a feature description vector C used to indicate the content of a specified standard word and a non-standard feature vector S representing a certain style can be used to convert a specified standard word into a target sample of the style without collecting the style. It can improve the efficiency of sample generation, and can generate samples containing different text content in various styles according to needs, so as to realize the diversity of samples.

In one embodiment, the above method flow can be executed by the sample generation apparatus 100 . As shown in FIG. 2 , the sample generation apparatus 100 may include two modules: a feature description vector acquisition module 101 and a target sample generation module 102 . The feature description vector obtaining module 101 is configured to execute the above step S100, and the target sample generation module 102 is configured to execute the above step S200.

In one embodiment, in step S100, the acquiring the feature description vector C of the specified standard word includes:

The first image containing the specified standard word is input to the first neural network in the trained student network, so that the first neural network performs feature extraction on the inputted first image to obtain a feature description vector C .

The student network is pre-trained and can be pre-stored in the electronic device or stored in an external device, and the electronic device calls the first neural network in the student network when the above method needs to be executed.

The first image may be obtained by collecting designated standard words in a real scene, or may be obtained by format conversion of designated standard words in a designated font library, and the specific manner is not limited. The first image may be preset in the electronic device, and the first image is acquired from the electronic device during execution.

There are more than 20,000 Song-style characters in the Song-style font library (the format is ttf), and the first image can be obtained according to the Song-style fonts existing in the Song-style font library. For example, the first image can be obtained by directly converting the Song-style characters in the Song-style font library from the ttf format to the image format; or, the first image can be generated by merging the Song-style characters in the Song-style font library with background data (such as background data representing a white background).

The first image is input to the first neural network, and after the first neural network performs feature extraction on the first image, a feature description vector C of the specified standard word can be obtained. This function of the first neural network can be provided by training.

Specifically, referring to FIG. 3 , the first image is, for example, an image with a size of 64*64, the specified standard word in the first image is, for example, "eye" in Song Dynasty, and the first neural network performs feature extraction on the first image to obtain the indication "" The 512-dimensional feature description vector C of "eye".

In one embodiment, the first neural network performs feature extraction on the inputted first image to obtain a feature description vector C, including:

The first neural network performs feature extraction on the first image through at least a convolution layer for performing convolution processing and a first nonlinear transformation layer for performing nonlinear transformation processing to obtain a feature description vector C.

The first neural network may include a multi-layer convolution layer, and the convolution layer performs a convolution operation, and may perform feature extraction on the first image to obtain a feature description vector, and output the feature description vector to the first nonlinear transformation layer. The first nonlinear transformation layer can enhance the fitting ability of the neural network, and the first nonlinear transformation layer outputs the fitted feature description vector as the feature description vector. Of course, the layer structure in the first neural network is not limited to this, and can also include other layers such as pooling layer. Pooling layer is a special downsampling layer, that is, the feature description vector obtained by convolution is processed. Dimensionality reduction.

For example, the first neural network can be implemented by using a convolutional neural network architecture such as VGG, Inception, and ResNet, but is not limited to this. Convolutional neural network is a feedforward neural network whose neurons can respond to surrounding units within a limited coverage area, and effectively extract the structural information of images through weight sharing and feature aggregation.

In one embodiment, in step S200, using the feature description vector C and the specified non-standard feature vector S to convert a specified standard word into a target sample, including:

S201 : Input the feature description vector C and the non-standard feature vector S to a second neural network in the trained student network, so that the second neural network combines the feature description vector C with the non-standard feature vector S The standard feature vector S is fused to obtain a fusion vector T, and the fusion vector T is used to generate a second image;

S202: Determine the second image as the target sample.

The student network is pre-trained and can be pre-stored in the electronic device or stored in an external device, and the electronic device calls the second neural network in the student network when the above method needs to be executed.

After the feature description vector C and the non-standard feature vector S are input into the second neural network, the second neural network will fuse the input feature description vector C and the non-standard feature vector S to obtain a fusion vector T, and use the fusion vector T. The vector T generates the second image. The second image is used as a target sample, and its style is consistent with the non-standard feature vector S and the contained text content is consistent with the content of the specified standard word.

Based on the training of the second neural network, the style corresponding to the second image can be specified by inputting different non-standard feature vectors, which can be applied to generate samples of different font styles. For example, if the style represented by the input non-standard feature vector S is Liu Zong Yuan body, then the style of the generated second image is Liu Zong Yuan body; the style represented by the input non-standard feature vector S is Mi Fu body, then the style of the generated second image is The style is Mi Fu body, etc.

Continuing to refer to Fig. 3, the first neural network performs feature extraction on the first image to obtain a 512-dimensional feature description vector C indicating "eye", and the feature description vector C is input into the second neural network. The vector S is input to the second neural network. The style represented by the non-standard feature vector S is, for example, a specified style, and the dimension of the non-standard feature vector S is, for example, 100 dimensions. The second neural network fuses the input 512-dimensional feature description vector C and 100-dimensional non-standard feature vector S into a fusion vector T, and uses the fusion vector T obtained by fusion to generate a second image, for example, the size of the second image is A 64*64 image, the second image contains the "eye" of the specified style, and the second image is used as the target sample. In step S201, the second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T in more than one implementation manner, for example, including the following three implementation manners:

In a first implementation manner, the second neural network includes a fusion layer; the feature description vector C has the same dimension as the non-standard feature vector S;

The second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T, including:

The second neural network uses a fusion layer to perform superposition processing on the feature description vector C and the non-standard feature vector S to obtain the fusion vector T.

In this method, the fusion layer is a computing layer for performing vector superposition processing, and the fusion vector T can be obtained by performing superposition processing on the feature description vector C and the non-standard feature vector S.

The superposition process may be a weighted superposition process, in which the values of the feature description vector C and the non-standard feature vector S in each dimension are correspondingly weighted and summed. For example, C=(a1, a2, a3,..., a512), S=(b1, b2, b3,..., b512), the fusion vector T=(a1*x1+b1*y1, a2*x2+b2*y2, a3*x3+b3*y3,...,a512*x512+b512*y512), where (x1, x2, x3,...,x512) is the feature description vector C in each dimension (y1, y2, y3, .

In the second implementation manner, the second neural network includes a fully connected layer and a fusion layer; the dimension of the feature description vector C is different from that of the non-standard feature vector S;

The second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T, including:

The second neural network utilizes a fully connected layer to map the non-standard feature vector S to a reference vector K having the same dimension as the feature description vector C;

The second neural network uses a fusion layer to perform superposition processing on the feature description vector C and the reference vector K to obtain the fusion vector T.

A fully connected layer is a computational layer that performs vector dimension mapping. For example, the dimension of the non-standard feature vector S is 100 dimensions, while the dimension of the feature description vector C is 512 dimensions. Through the full connection layer, the non-standard feature vector S can be mapped to a reference vector K with a dimension of 512 dimensions to achieve dimension expansion. . The fusion layer is a computing layer for performing vector superposition processing. The feature description vector C and the reference vector K can be superimposed to obtain a fusion vector T. The superposition processing method is similar to that in the first implementation, and will not be repeated here.

In a third implementation manner, the second neural network includes a fusion layer;

The second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T, including:

The second neural network uses a fusion layer to combine the feature description vector C and the non-standard feature vector S to obtain the fusion vector T.

This implementation is especially suitable for the case where the dimension of the feature description vector C and the non-standard feature vector S are different, of course, it is also applicable in the case of the same dimension.

In this method, the fusion layer is a computing layer for performing vector merging processing, and a new row fusion vector T can be obtained by performing merging processing on the feature description vector C and the non-standard feature vector S.

The merging of vectors is the splicing of two vectors in dimension, and the dimension of the merged vector is the sum of the dimensions of the two required vectors. For example, C=(a1, a2, a3,..., a512), S=(b1, b2, b3,..., b100), the combined T=(a1, a2, a3,..., a512, b1 , b2, b3, ..., b100).

In one embodiment, the second neural network further includes: a deconvolution layer for performing deconvolution processing, and a second nonlinear transformation layer for performing nonlinear transformation;

The second neural network using the fusion vector T to generate the second image includes:

The second neural network generates a second image corresponding to the fusion vector T by using the deconvolution layer and the second nonlinear transformation layer.

The second neural network may include a multi-layer deconvolution layer, the deconvolution layer performs a deconvolution operation, and may use the fusion vector T to generate a second image, and output the second image to the second nonlinear transformation layer. The second nonlinear transformation layer can also enhance the fitting ability of the neural network, and the second nonlinear transformation layer outputs the fitted second image. Of course, the layer structure of the second neural network is not limited to this, and may also include other layers such as a fully connected layer. Fully connected layers can also be replaced with convolutional layers.

In one embodiment, the student network is trained under the supervision of a trained teacher network;

The network parameters of at least one layer in the second neural network apply the network parameters of the corresponding layer in the teacher network.

By training a teacher network, the teacher network supervises the training of the first neural network and the second neural network. In the training method of this embodiment, the connection structure of the first neural network and the second neural network is regarded as a student network.

Referring to Figure 4, before training the student network, a teacher network needs to be trained first, and the teacher network includes a first neural network A1 and a second neural network A2. The layer structures of the first neural network A1 and the first neural network in the student network may be the same; the layer structures of the second neural network A2 and the second neural network in the student network may be similar, but do not need to be executed Fusion of vectors, so the fusion layer can be omitted. The training is divided into two steps:

First, the samples containing the specified style characters (non-Song-style characters) "eye" (which can be collected from the real scene) are used as the input and output of the teacher network, and the teacher network is trained. After training the teacher network, the first neural network A1 and The network parameters of each layer in the second neural network A2;

Next, take the network parameters of a certain layer or layers of the second neural network A2 in the trained teacher network as the network parameters of the corresponding layer of the second neural network in the student network, and then take the sample containing the Chinese character "eye" as the student network The input, the non-standard feature vector S representing the specified style is used as the input of the second neural network in the student network, and the sample containing the specified style character "eye" is used as the output of the student network (the feature description vector obtained by the first neural network). will be input to the second neural network), train the student network, and complete the training of the student network.

The following provides a method for training the first neural network and the second neural network.

The training of the first neural network and the second neural network is supervised by training a classifier capable of distinguishing the generated samples and the real samples. For the sake of concise description, in this training method, the connection structure of the first neural network and the second neural network is called a neural network EG.

Referring to Fig. 5, the real sample is a sample collected from a real scene for training the neural network EG, such as an image collected in a real scene that contains a specified style of text (non-Song font), and shown in the figure is a specified style of text. "Eye" images, in the actual training process, you can use more real samples containing different specified styles of text for training, you can get better network parameters, the training process is divided into two steps:

First, input the sample containing the Song-style character "eye" into the neural network to obtain the generated sample containing the specified style character "eye", and take the real sample containing the specified style character "eye" and the non-standard feature vector S representing the specified style as a set Input data, take the generated samples of the neural network EG and the non-standard feature vector S representing the specified style as another set of input data, respectively input the two sets of input data into the classifier, and train the classifier so that it can distinguish the generated samples from Real samples, and can calculate the deviation between the generated samples and the real samples, and complete the training of the classifier;

Next, the training of the network parameters of the neural network EG is supervised by the trained classifier, and the samples containing the Song-style character "eye" are input into the neural network EG to obtain a generated sample containing the specified style character "eye", and the generated sample and the representation The non-standard feature vector S of the specified style is input into the classifier. After the classifier calculates the deviation between the generated sample and the corresponding real sample, the loss function is used to calculate the loss value corresponding to the deviation, and the network of the neural network EG is adjusted according to the loss value. Parameters, the training of the network parameters of the neural network EG, the steps of inputting the samples containing the Song-style character "eye" into the neural network EG to obtain the generated samples containing the specified style character "eye" Continue to train the neural network EG until the loss value of the loss function decreases. To a reasonable range, complete the training of the neural network EG.

Wherein, in the above-mentioned training process, inputting the sample containing the Song-style character "eye" into the neural network EG to obtain a generated sample containing the designated style character "eye", including: inputting the sample containing the Song-style character "eye" into the first neural network, The first neural network performs feature extraction on the input samples to obtain a feature description vector C representing the content of the Song-style character "eye" and input C into the second neural network; the non-standard feature vector S representing the specified style is also input into the first neural network. In the second neural network, the second neural network fuses the feature description vector C with the non-standard feature vector S and uses the resulting vector to generate an image, and the generated image is used as a generated image containing the specified style of text "eye". sample.

In the above two training methods, the first neural network and the second neural network are trained together. Of course, the first neural network and the second neural network can also be trained separately.

The present invention also provides a sample generation device, referring to FIG. 2 , the sample generation device 100 includes:

The feature description vector acquisition module 101 is used to obtain the feature description vector C of the specified standard word, and the feature description vector C is used to indicate the content of the specified standard word;

The target sample generation module 102 is configured to convert the specified standard word into a target sample by using the feature description vector C and the specified non-standard feature vector S, and the style corresponding to the target sample is the same as the style represented by the non-standard feature vector S.

In one embodiment, the feature description vector obtaining module is specifically used for:

The first image containing the specified standard word is input to the first neural network in the trained student network, so that the first neural network performs feature extraction on the inputted first image to obtain a feature description vector C .

In one embodiment, the first neural network performs feature extraction on the inputted first image to obtain a feature description vector C, including:

The first neural network performs feature extraction on the first image through at least a convolution layer for performing convolution processing and a first nonlinear transformation layer for performing nonlinear transformation processing to obtain a feature description vector C.

In one embodiment, the target sample generation module includes:

The image generation unit is used to input the feature description vector C and the non-standard feature vector S to the second neural network in the trained student network, so that the feature description vector C is converted by the second neural network Perform fusion with the non-standard feature vector S to obtain a fusion vector T, and utilize the fusion vector T to generate a second image;

A target sample determination unit, configured to determine the second image as the target sample.

In one embodiment, the second neural network includes a fusion layer; the feature description vector C has the same dimension as the non-standard feature vector S;

When the second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T, it is specifically used for:

The second neural network uses a fusion layer to perform superposition processing on the feature description vector C and the non-standard feature vector S to obtain the fusion vector T.

In one embodiment, the second neural network includes a fully connected layer and a fusion layer; the dimension of the feature description vector C is different from the dimension of the non-standard feature vector S;

When the second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T, it is specifically used for:

The second neural network utilizes a fully connected layer to map the non-standard feature vector S to a reference vector K having the same dimension as the feature description vector C;

The second neural network uses a fusion layer to perform superposition processing on the feature description vector C and the reference vector K to obtain the fusion vector T.

In one embodiment, the second neural network includes a fusion layer;

When the second neural network fuses the feature description vector C and the non-standard feature vector S to obtain a fusion vector T, it is specifically used for:

The second neural network uses a fusion layer to combine the feature description vector C and the non-standard feature vector S to obtain the fusion vector T.

In one embodiment, the second neural network further includes: a deconvolution layer for performing deconvolution processing, and a second nonlinear transformation layer for performing nonlinear transformation;

When the second neural network uses the fusion vector T to generate the second image, it is specifically used for:

The second neural network generates a second image corresponding to the fusion vector T by using the deconvolution layer and the second nonlinear transformation layer.

In one embodiment, the student network is trained under the supervision of a trained teacher network;

The network parameters of at least one layer in the second neural network apply the network parameters of the corresponding layer in the teacher network.

For details of the implementation process of the functions and functions of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method, which will not be repeated here.

For the apparatus embodiments, since they basically correspond to the method embodiments, reference may be made to the partial descriptions of the method embodiments for related parts. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units.

The present invention also provides an electronic device, including a processor and a memory; the memory stores a program that can be called by the processor; wherein, when the processor executes the program, the sample as described in the foregoing embodiments is implemented generate method.

Embodiments of the sample generating apparatus of the present invention can be applied to electronic equipment. Taking software implementation as an example, a device in a logical sense is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for operation by the processor of the electronic device where the device is located. From a hardware perspective, as shown in FIG. 6 , FIG. 6 is a hardware structure diagram of an electronic device where the sample generating apparatus 100 is located according to an exemplary embodiment of the present invention, except for the processor 510 and the memory shown in FIG. 6 . 530 , the interface 520 , and the non-volatile memory 540 , the electronic device in which the apparatus 100 is located in the embodiment generally may also include other hardware according to the actual function of the electronic device, which will not be repeated here.

The present invention also provides a machine-readable storage medium on which a program is stored, and when the program is executed by a processor, implements the sample generation method described in any one of the foregoing embodiments.

The present invention may take the form of a computer program product embodied on one or more storage media having program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like. Machine-readable storage media includes both persistent and non-permanent, removable and non-removable media, and storage of information can be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of machine-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage , magnetic tape cartridges, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (11)

1. A method of generating a sample, comprising:

acquiring a feature description vector of a specified standard word, wherein the feature description vector is used for indicating the content of the specified standard word;

and converting the specified standard words into target samples by using the feature description vectors and the specified non-standard feature vectors, wherein the style corresponding to the target samples is the same as the style represented by the non-standard feature vectors.

2. The sample generation method of claim 1, wherein said obtaining a feature description vector specifying a standard word comprises:

inputting a first image containing the specified standard words into a first neural network in a trained student network, and performing feature extraction on the input first image by the first neural network to obtain a feature description vector.

3. The sample generation method of claim 2, wherein the first neural network performs feature extraction on the input first image to obtain a feature description vector, comprising:

the first neural network performs feature extraction on the first image at least through a convolution layer for performing convolution processing and a first nonlinear transformation layer for performing nonlinear transformation processing to obtain a feature description vector.

4. The sample generation method of claim 1, wherein converting a specified standard word into a target sample using the feature description vector and a specified non-standard feature vector comprises:

inputting the feature description vector and the non-standard feature vector into a second neural network in a trained student network, fusing the feature description vector and the non-standard feature vector by the second neural network to obtain a fusion vector, and generating a second image by using the fusion vector;

determining the second image as the target sample.

5. The sample generation method of claim 4, wherein the second neural network comprises a fusion layer; the feature description vector has the same dimension as the non-standard feature vector;

the second neural network fuses the feature description vector and the non-standard feature vector to obtain a fused vector, including:

and the second neural network utilizes a fusion layer to perform superposition processing on the feature description vector and the non-standard feature vector S to obtain the fusion vector.

6. The sample generation method of claim 4, wherein the second neural network comprises a fully connected layer and a fused layer; the feature description vector is different in dimension from the non-standard feature vector;

the second neural network fuses the feature description vector and the non-standard feature vector to obtain a fused vector, including:

the second neural network utilizes a fully connected layer to map the non-standard feature vector to a reference vector having dimensions that are the same as dimensions of the feature description vector;

and the second neural network utilizes a fusion layer to perform superposition processing on the feature description vector and the reference vector to obtain the fusion vector.

7. The sample generation method of claim 4, wherein the second neural network comprises a fusion layer;

the second neural network fuses the feature description vector and the non-standard feature vector to obtain a fused vector, including:

and the second neural network combines the feature description vector and the non-standard feature vector S by utilizing a fusion layer to obtain the fusion vector.

8. The sample generation method of any of claims 5 to 7, wherein the second neural network further comprises: a deconvolution layer for performing deconvolution processing, and a second nonlinear transformation layer for performing nonlinear transformation;

the second neural network generating a second image using the fused vector comprises:

the second neural network generates a second image corresponding to the fusion vector using the deconvolution layer and a second nonlinear conversion layer.

9. The sample generation method of claim 4, wherein the student network is trained under trained teacher network supervision;

the network parameters of at least one layer in the second neural network are applied to the network parameters of the corresponding layer in the teacher network.

10. A sample generation device, comprising:

the characteristic description vector acquisition module is used for acquiring a characteristic description vector of the specified standard word, and the characteristic description vector is used for indicating the content of the specified standard word;

and the target sample generation module is used for converting the specified standard words into target samples by using the feature description vectors and the specified non-standard feature vectors, and the style corresponding to the target samples is the same as the style represented by the non-standard feature vectors.

11. An electronic device comprising a processor and a memory; the memory stores a program that can be called by the processor; wherein the processor, when executing the program, implements the sample generation method of any of claims 1-7, 9.

Images