WO2022116161A1 - Portrait cartooning method, robot, and storage medium - Google Patents

Abstract

A portrait cartooning method, a robot, and a storage medium. The method comprises: obtaining an original portrait to be processed (102); recognizing a face in the original portrait, and performing face alignment according to recognized face key points to obtain an aligned standard portrait (104); performing portrait segmentation on the standard portrait, and filtering background information to obtain a target portrait (106); and using the target portrait as an input of a portrait cartoonlization model, and obtaining a cartooned portrait corresponding to the target portrait and outputted by the portrait cartoonlization model, the portrait cartoonlization model being obtained by training a generative adversarial network model (108). The target portrait does not contain external environment information and only contains a person, so that the processing of a subsequent portrait cartoonalization model is simplified, the method is suitable for being deployed at a robot end having limited computing power, and the obtained cartoon effect is good due to no background information.

Description

Portrait cartoonization method, robot and storage medium technical field

The present application relates to the field of computer technology, and in particular, to a method for cartoonizing a portrait, a robot and a storage medium.

Background technique

With the continuous development of Generative Adversarial Networks, more and more AI applications based on Generative Adversarial Networks have emerged. Such as black and white painting coloring, image stylization, portrait cartoonization and so on. Among them, the cartoonization of portraits has become a research hotspot. The current portrait cartoon models tend to have poor cartoon effects when the external environment is diverse, and if a good effect is desired, the complexity of the model is often increased.

Due to the limited computing power on the robot side, complex portrait cartoon models are difficult to use on the robot side. Therefore, there is an urgent need for a portrait cartoon method that can be used on the robot side.

SUMMARY OF THE INVENTION

Based on this, it is necessary to address the above problems, and to propose a method for cartoonizing a portrait, a robot and a storage medium, which has a good cartoon effect and is suitable for use on the robot side.

A method of cartoonizing a portrait, comprising:

Get the original character image to be processed;

Recognizing the face in the original character image, and aligning the face according to the key points of the face obtained by the identification, to obtain an aligned standard character image;

Perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image;

The target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model. of.

A robot includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:

Get the original character image to be processed;

Recognizing the face in the original character image, and aligning the face according to the key points of the face obtained by the identification, to obtain an aligned standard character image;

Perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image;

The target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model. of.

A computer-readable storage medium storing a computer program, when executed by a processor, the computer program causes the processor to perform the following steps:

Get the original character image to be processed;

Recognizing the face in the original character image, and aligning the face according to the key points of the face obtained by the identification, to obtain an aligned standard character image;

Perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image;

The target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model. of.

In the above-mentioned portrait cartoonization method, robot and storage medium, firstly, face alignment is performed on the face in the original character image to obtain the aligned standard character image, and then the standard character image is segmented, and the background information is filtered out to obtain the target. Character images, there is no external environment information in the target character image, only characters are included, so the processing of subsequent portrait cartoon models is simplified, which is suitable for deployment on robots with limited computing power, and because there is no background information, the obtained cartoon effect is good , that is, a good cartoon effect can be achieved on the robot side.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

in:

Fig. 1 is the flow chart of the portrait cartoonization method in one embodiment;

Fig. 2 is a schematic diagram of portrait alignment in one embodiment;

Fig. 3 is the schematic diagram before and after cartoonization of single person portrait in one embodiment;

Fig. 4 is the training flow chart of generative adversarial network model in one embodiment;

Fig. 5 is the schematic diagram before and after single person portrait segmentation in one embodiment;

Fig. 6 is the structural block diagram of the portrait cartoonization device in one embodiment;

Fig. 7 is a structural block diagram of a training module in one embodiment;

FIG. 8 is a diagram of the internal structure of the robot in one embodiment.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

As shown in FIG. 1 , a method for cartoonizing a portrait is proposed, and the method for cartoonizing a portrait can be applied to a terminal. This embodiment is illustrated by being applied to a robot terminal. The portrait cartoonization method specifically includes the following steps:

Step 102, acquiring the original character image to be processed.

Among them, the original character image includes cartoon character images. The proportion of the human body in different original character images is often inconsistent. For example, in some original human images, the proportion of the human body is marked on the stomach area, some of the human body is marked on the legs, and some are marked on the shoulders, etc. . If the character images in these original character images are directly cartoonized, the cartoonized portrait model must be adapted to various situations, and the relative model design will be particularly complicated, resulting in a large amount of calculation. Not suitable for use on the robot side. The original character image may be an image directly captured by a camera, or an image obtained from an album. In one embodiment, the terminal is a robot terminal.

Step 104 , identify the face in the original character image, and perform face alignment according to the identified face key points to obtain an aligned standard character image.

Among them, the face key points refer to the feature points that reflect the facial features of the face, including: the feature points of eyebrows, eyes, nose, mouth and facial contour. In order to facilitate the subsequent segmentation, the face alignment method is used to align the original person images to obtain the aligned standard person images.

The standard person image refers to a preset normalized person image. For example, the hair in the standard task image can be set as the starting position, and the shoulder area is the ending position. The process of face alignment is equivalent to equidistant transformation + uniform scale scaling, and its effect has the invariant characteristics of angle, parallelism and verticality. The face in the aligned standard human image is positive, and the proportion of the human body conforms to the preset proportion rule. As shown in FIG. 2 , it is a schematic diagram of portrait alignment in one embodiment.

Specifically, the goal of face alignment is to map five key points in the face (left eye, right eye, nose, mouth left, and mouth right) to specified positions in the target space, while other parts undergo undistorted changes. The role of the 5 key points is to map the face to the frontal face, and then the other parts are mapped to the target space accordingly. The selection of the target space is based on the proportion of the human body. If the proportion of the human body is small, the corresponding target space is also relatively small.

Step 106, perform portrait segmentation based on the standard person image, filter out the background information, and obtain the target person image.

Among them, since the standard person images have a unified and single feature, it is simpler to perform portrait segmentation based on annotated person images, and only a lightweight portrait segmentation model can be used. After the portrait is segmented, the background information is filtered out, so that the target person image without the external environment information is obtained, so that the cartoon effect will not be poor due to the influence of the external environment during subsequent cartoonization. As shown in FIG. 3 , it is a schematic diagram before and after segmentation of a single person portrait in one embodiment.

In step 108, the target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model.

Among them, since the target person image has a uniform body proportion and does not have any background information interference, this greatly simplifies the training of the portrait cartoon model and helps to improve the cartoon effect. As shown in FIG. 4 , it is a schematic diagram of characters before and after cartoonization in one embodiment.

In the above-mentioned portrait cartoonization method, firstly, face alignment is performed on the face in the original character image to obtain the aligned standard character image, then the standard character image is segmented, and the background information is filtered out to obtain the target character image, the target character image. There is no external environment information in the image, only characters are included, so the processing of subsequent portrait cartoon models is simplified, which is suitable for deployment on the robot side with limited computing power, and because there is no background information, the obtained cartoon effect is good, that is, on the robot side A good cartoon effect can be achieved.

In one embodiment, the generative adversarial network model includes: a forward generator, a reverse generator, a first discriminator and a second discriminator; the output of the forward generator is connected to the first discriminator , the output of the reverse generator is connected to the second discriminator; the forward generator is used to convert the input character image into a character cartoon image, and the reverse generator is used to convert the input character cartoon image into is a character image, the first discriminator is used to calculate the probability that the input image belongs to a character cartoon image, and the second discriminant network model is used to calculate the probability that the input image belongs to a character image.

Among them, the portrait cartoon model is obtained by training the generative adversarial network model. Since the portrait cartoon model is trained under unsupervised conditions, other models are needed to assist the training. The generative adversarial network model includes: two generators, called "forward generator" and "reverse generator" for distinction, and then two discriminators, called "first discriminator" and "first discriminator" respectively. Second Discriminator". The final trained forward generator is the above-mentioned portrait cartoon model.

Specifically, the forward generator is connected to the first discriminator, and the reverse generator is connected to the second discriminator. The function of the forward generator is to convert the input character image into a character cartoon image, and the function of the reverse generator is to convert the input character cartoon image into a character image. The function of the first discriminator is to calculate the probability that the input image belongs to a cartoon image, and the function of the second discriminator network model is to calculate the probability that the input image belongs to a character image. That is to say, the purpose of the first discriminator is to judge the input real character cartoon image as true, and judge the image output by the forward generator as false, if the output image of the forward generator is enough to fool the first judger, It shows that the image output by the forward generator has the characteristics of cartoon image, so that the first discriminant network model cannot identify the authenticity. That is, the forward generator and the first discriminator adopt the way of learning against each other. In the same way, the reverse generator and the second discriminator also adopt the way of learning against each other.

As shown in Figure 5, in one embodiment, the training step of the generative adversarial network model includes:

Step 502 , acquiring a training image set, where the training image set includes: a training character image set and a training character cartoon image set.

Among them, the generative adversarial network model is an unsupervised training method. Since the training images are not labeled, two image sets are required, the training character image set and the training character cartoon image set, the training character image set and the training character cartoon image set. Sets are asymmetric.

Step 504 , input the training person image into the forward generator to obtain the first output image, and input the first output image into the first discriminator to obtain the first probability of the corresponding output.

The training character image is obtained from the training character image set, and the training character image is used as the input of the forward generator to obtain the outputted first output image. The purpose of the forward generator is to convert the input training character image into a character cartoon image. Since there is no labeling, it is necessary to use the first output image as the input of the first discriminator to obtain the first probability of the output. The first probability refers to Determine the probability that the obtained first output image belongs to the cartoonized image of the character. The purpose of the first discriminator is to identify that the first output image is a fake character cartoon image.

Step 506: Input the cartoon image of the training character into the reverse generator to obtain the second output image, and input the second output image into the second discriminator to obtain the second probability of the corresponding output.

The training character cartoon image is obtained from the training character cartoon image set, and the training character cartoon image is used as the input of the reverse generator to obtain the output second output image. The purpose of the reverse generator is to convert the input training character cartoon image into a character image, and then use the second discriminator to identify it, and the purpose of the second discriminator is to output that the second output image is a fake character image.

Step 508: Input the first output image into the reverse generator to obtain the third output image; input the second output image into the forward generator to obtain the fourth output image.

Among them, in order to keep the content of the original image as much as possible during the image conversion process, it is hoped that after the character image is converted into a character cartoon image through the forward generator, the original image can be obtained through the reverse generator. Therefore, the smaller the difference between the images after the cycle, the better, that is, the smaller the difference between the training character image and the third output image, the better, and the smaller the difference between the training character cartoon image and the fourth output image, the better.

Step 510: Calculate the first difference value between the training character image and the third output image, and calculate the second difference value between the training character cartoon image and the fourth output image.

Among them, the smaller the difference value, the greater the similarity between the images. So we can calculate the difference value by calculating the image similarity. That is, the similarity (less than 1) between the training person image and the third output image is calculated, and then the similarity is subtracted from 1 to obtain the first difference value. Similarly, calculate the similarity between the cartoon image of the training character and the fourth output image, and then subtract the similarity from 1 to obtain the second difference value.

Step 512: Calculate and obtain a first loss function value according to the first probability, the second probability, the first difference value and the second difference value.

The first probability can be used to calculate the adversarial loss value of the forward generator and the first discriminator, the second probability can be used to calculate the adversarial loss value of the reverse generator and the second discriminator, the first difference value and the second The difference value is used to calculate the cycle loss value. The three are combined to form the first loss function value. The first loss function value is also a conventional loss function value.

Step 514 , taking the training person image as the input of the facial feature extraction model, and acquiring the first facial feature extracted by the facial feature extraction model.

Among them, in order to make the generated cartoon face closer to the real face, that is, to have more of the same face features as the real face. A face feature extraction model is innovatively introduced in the embodiment of the present application. First, the training person image is used as the input of the face feature extraction model to extract the first face feature. The facial feature extraction model is used to identify and extract the facial features in the training character images. In one embodiment, the facial feature extraction model can be implemented by using a lightweight mobilefacenet.

Step 516 , using the first output image as the input of the face feature extraction model, and obtain the second face feature extracted by the face feature extraction model.

Wherein, the first output image is also used as the input of the face feature extraction model, and the extracted second face feature is obtained, which is convenient for subsequent comparison with the first face feature, so that the first face feature and the second face feature are Closer.

Step 518: Calculate the similarity between the first face feature and the second face feature, and calculate the second loss function value according to the similarity.

Among them, in order to make the second face feature output by the trained model more similar to the first face feature, in the training process, the similarity between the first face feature and the second face feature is calculated, and then according to the similarity The second loss function value is obtained by the degree calculation, which facilitates the subsequent adjustment of the model according to the second loss function value. In one embodiment, the second loss function value is calculated using the following formula:

loss ₂ =1-cosine(fea_realA,fea_fakeA2B)

Among them, cosine() represents similarity, fea_realA represents the first face feature, and fea_fakeA2B represents the second face feature.

Step 520: Calculate a total loss value according to the first loss function value and the second loss function value, and the total loss value is inversely correlated with the similarity.

Among them, both the first loss function value and the second loss function value are taken as part of the total loss value, and then the parameter weights in the model are updated according to the total loss value, so that the total loss value is updated in a decreasing direction until convergence is achieved condition. Specifically, the total loss function value is expressed as: loss=loss ₁ +loss ₂ . loss ₁ represents the first loss function value, and loss ₂ represents the second loss function value.

Step 522: Update the weight parameters in the forward generator, the reverse generator, the first discriminator and the second discriminator according to the total loss value, and repeat this cycle until the total loss value reaches the convergence condition, and the positive result obtained after training is completed. To the generator as a cartoonized model of a portrait.

Among them, after calculating the total loss value, update the weight parameters in the forward generator, the reverse generator, the first discriminator and the second discriminator, so that the total loss value changes in the direction of decreasing, and then repeat this cycle , until the final total loss value reaches the convergence condition, the training is completed, and the forward generator obtained after the training is used as the portrait cartoon model.

In one embodiment, the forward generator includes an encoder and a decoder, and the encoder and the decoder are composed of Hourglass modules.

Among them, in order to make the generated cartoon images of characters more realistic, in the training phase, the encoder and decoder in the forward generator are implemented by Hourglass modules respectively. The Hourglass module helps to provide feature abstraction and reconstruction capabilities of the model. The Hourglass module is composed of the residual module (Residual). The feature of the Hourglass (funnel) module is to perform maximum pooling first, and then perform upsampling. Through maximum pooling, the feature map can be reduced, and then upsampling is performed to expand. Experiments show that implementing the functions of encoder and decoder through Hourglass module can greatly improve the cartoon-like fidelity of the model.

In one embodiment, the process of recognizing the face in the original character image, aligning the face according to the key points of the face obtained by identification, and obtaining the aligned standard character image, includes: according to the preset human body occupancy Determine the target coordinate position of the face key point in the preset space; map the face key point to the target coordinate position in the preset space to obtain a standard human image aligned in the preset space.

Among them, the traditional face alignment is only to align the face, and the preset space is often relatively small, for example, the size is 112X112, and then in this limited space, set the coordinate positions of five key points after mapping. For example, the coordinate positions of the 5 key points (left eye, right eye, nose, mouth left, mouth right) are {[38.2946, 51.6963], [73.5318, 51.5014], [56.0252, 71.7366], [41.5493, 92.3655] , [70.7299, 92.2041]}. If the standard person image needs to contain not only the face area, but also needs to be extended to other parts, for example, to the shoulders. Then the corresponding space needs to be expanded. For example, the size needs to be set to 256X256, and the coordinate positions of the corresponding 5 key points also need to be changed, so that the hair above the face and the parts from the bottom of the face to the shoulders can be displayed in the standard character image. middle.

Specifically, first determine the target coordinate position where the key points of the face are mapped to the preset space according to the preset proportion of the human body in the image, and the target coordinate position is the designated position. For example, if the preset proportion of the human body starts from the hair to the shoulders In order to reserve the mapping space for the hair part and the part below the head when aligning the face, it is necessary to map the coordinates of the key points of the face to the middle part of the image as much as possible. Taking the lower left corner of the image as the origin coordinate for illustration, compared with the traditional face alignment, the ordinate of the left eye and the right eye can be lowered, and the abscissa of the left eye and the abscissa of the right eye can be moved closer to the middle of the image, that is Expand the abscissa of the left eye and decrease the abscissa of the right eye. In this way, space can be reserved for the upper part of the hair, and space is reserved for the left and right parts of the face. The same principle, the abscissa of the nose remains unchanged, the ordinate decreases, and space is reserved for the lower part of the face, while the left side of the mouth and the mouth The abscissa on the right moves closer to the middle of the image, and the ordinate increases.

In one embodiment, the performing portrait segmentation based on the standard human image, filtering out background information, and obtaining the target human image includes: using the standard human image as an input of a portrait segmentation model, and the portrait segmentation model is used for A person image is segmented from the standard person image; background information is filtered out based on the person image output by the portrait segmentation model, and the target person image is generated.

The portrait segmentation model is used to segment the target person image in the standard person image to obtain the target person image. The portrait segmentation model is implemented by a lightweight convolutional neural network mobilenetv2. Based on the filtered background information of the human image output by the portrait segmentation model, a uniform white background can be added to the segmented human image to generate the target human image.

In one embodiment, the portrait segmentation model is obtained by training a convolutional neural network, and includes a plurality of convolutional layers, and the convolutional layers are used to perform feature extraction on images; before using the convolutional layers for feature extraction , and further comprising: performing edge amplification on the image, so that the resolution of the image obtained after the convolution is consistent with the resolution of the input standard person image.

Among them, in order to keep the resolution of the image before and after the convolution unchanged, before the convolution, the edge of the image is enlarged, that is, the image is enlarged, and then the convolution operation is performed based on the enlarged image, so that the result obtained after the convolution operation is obtained. The resolution of the standard character image is consistent with the resolution of the original input character image, so as to ensure the accuracy of the cartoonization of the portrait.

As shown in FIG. 6 , in one embodiment, a device for cartoonizing a portrait is proposed, including:

an acquisition module 602, configured to acquire the original character image to be processed;

Alignment module 604, configured to identify the human face in the original character image, and perform face alignment according to the identified face key points to obtain an aligned standard character image;

A segmentation module 606, configured to perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image;

The cartoonization module 608 is configured to use the target character image as the input of the portrait cartoonization model, and obtain the character cartoonization image corresponding to the target character image output by the portrait cartoonization module, and the portrait cartoonization model is It is obtained by training a generative adversarial network model.

In one embodiment, the generative adversarial network model includes: a forward generator, a reverse generator, a first discriminator and a second discriminator; the output of the forward generator is connected to the first discriminator , the output of the reverse generator is connected to the second discriminator; the forward generator is used to convert the input character image into a character cartoon image, and the reverse generator is used to convert the input character cartoon image into is a character image, the first discriminator is used to calculate the probability that the input image belongs to a cartoon image, and the second discriminant network model is used to calculate the probability that the input image belongs to a character image.

In one embodiment, the above-mentioned device for cartoonizing a portrait further includes: a training module.

As shown in Figure 7, the training module 601 includes:

A training acquisition module 601A, configured to acquire a training image set, the training image set includes: a training character image set and a training character cartoon image set;

The first input module 601B is used for inputting the training character image into the forward generator to obtain the first output image, and inputting the first output image into the first discriminator to obtain the first probability of the corresponding output;

The second input module 601C is used to input the cartoon image of the training character into the reverse generator to obtain the second output image, and input the second output image to the second discriminator to obtain the second probability of the corresponding output;

The third input module 601D is configured to input the first output image into the reverse generator to obtain a third output image; input the second output image into the forward generator to obtain a fourth output image;

A first calculation module 601E, configured to calculate the first difference value between the training character image and the third output image, and calculate the second difference value between the training character cartoon image and the fourth output image;

The second calculation module 601F is configured to calculate and obtain the first loss function value according to the first probability, the second probability, the first difference value and the second difference value;

The first extraction module 601G is used to use the training character image as the input of the facial feature extraction model, and obtain the first facial feature extracted by the facial feature extraction model;

The second extraction module 601H is configured to use the first output image as the input of the facial feature extraction model, and obtain the second facial feature extracted by the facial feature extraction model;

The third calculation module 601I is used to calculate the similarity between the first face feature and the second face feature, and calculate the second loss function value according to the similarity;

a fourth calculation module 601J, configured to calculate and obtain a total loss value according to the first loss function value and the second loss function value, where the total loss value is inversely correlated with the similarity;

The updating module 601K is configured to update the weight parameters in the forward generator, the reverse generator, the first discriminator and the second discriminator according to the total loss value, and repeat this cycle until The total loss value reaches the convergence condition, and the forward generator obtained after training is used as the portrait cartoon model.

In one embodiment, the forward generator includes an encoder and a decoder, and the encoder and the decoder are composed of Hourglass modules.

In one embodiment, the alignment module is further configured to determine the target coordinate position of the key points of the face in the preset space according to the preset proportion of the human body; map the key points of the face to the target in the preset space Coordinate position, get the standard human image aligned in the preset space.

In one embodiment, the segmentation module is further configured to use the standard person image as an input of a portrait segmentation model, and the portrait segmentation model is used to segment a person image from the standard person image; output based on the portrait segmentation model The background information of the person image is filtered out to generate the target person image.

In one embodiment, the portrait segmentation model is obtained by training a convolutional neural network, and includes a plurality of convolutional layers, and the convolutional layers are used to perform feature extraction on the image; the above-mentioned apparatus further includes: an amplification module for Edge augmentation is performed on the image, so that the resolution of the image obtained after the convolution is consistent with the resolution of the input standard person image.

Figure 8 shows a diagram of the internal structure of the robot in one embodiment. As shown in Figure 8, the robot includes a processor, memory, camera, and network interface connected through a system bus. Wherein, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the robot stores an operating system, and also stores a computer program. When the computer program is executed by the processor, the processor can realize the above-mentioned method for cartoonizing a portrait. A computer program may also be stored in the internal memory, and when the computer program is executed by the processor, the processor can execute the above-mentioned method for cartoonizing a portrait. Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the robot to which the solution of the present application is applied. More or fewer components are shown in the figures, either in combination or with different arrangements of components.

In one embodiment, a robot is proposed, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the above-mentioned method for cartoonizing a portrait. step.

In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, causes the processor to perform the steps of the above-mentioned method for cartoonizing a portrait.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium , when the program is executed, it may include the flow of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (21)

A method for cartoonizing a portrait, comprising: Get the original character image to be processed; Recognizing the face in the original character image, and aligning the face according to the key points of the face obtained by the identification, to obtain an aligned standard character image; Perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image; The target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model. of. The method according to claim 1, wherein the generative adversarial network model comprises: a forward generator, a reverse generator, a first discriminator and a second discriminator; the output of the forward generator is the same as the The first discriminator is connected, and the output of the reverse generator is connected to the second discriminator; The forward generator is used to convert the input character image into a character cartoon image, the reverse generator is used to convert the input character cartoon image into a character image, and the first discriminator is used to calculate that the input image belongs to a character cartoon. The probability of the image, the second discriminant network model is used to calculate the probability that the input image belongs to the person image. The method according to claim 2, wherein the training step of the generative adversarial network model comprises: Acquiring a training image set, the training image set includes: a training character image set and a training character cartoon image set; Input the training character image into the forward generator to obtain the first output image, and input the first output image into the first discriminator to obtain the first probability of the corresponding output; Input the training character cartoon image into the reverse generator to obtain the second output image, and input the second output image into the second discriminator to obtain the second probability of the corresponding output; Inputting the first output image into the reverse generator to obtain a third output image; inputting the second output image into the forward generator to obtain a fourth output image; Calculate the first difference value between the training character image and the third output image, and calculate the second difference value between the training character cartoon image and the fourth output image; The first loss function value is obtained by calculating according to the first probability, the second probability, the first difference value and the second difference value; Using the training character image as the input of the facial feature extraction model, obtain the first facial feature extracted by the facial feature extraction model; Using the first output image as the input of the facial feature extraction model, obtain the second facial feature extracted by the facial feature extraction model; Calculate the similarity between the first face feature and the second face feature, and calculate the second loss function value according to the similarity; A total loss value is calculated according to the first loss function value and the second loss function value, and the total loss value is inversely correlated with the similarity; Update the weight parameters in the forward generator, the reverse generator, the first discriminator and the second discriminator according to the total loss value, and repeat this cycle until the total loss value reaches Convergence condition, the forward generator obtained after training is used as the portrait cartoon model. The method according to claim 2, wherein the forward generator comprises: an encoder and a decoder, and the encoder and the decoder are composed of Hourglass modules. The method according to claim 1, wherein the identifying the face in the original character image, aligning the face according to the identified face key points, and obtaining the aligned standard character image, comprising: : Determine the target coordinate position of the key point of the face in the preset space according to the preset proportion of the human body; The face key points are mapped to the target coordinate position in the preset space to obtain a standard person image aligned in the preset space. The method according to claim 1, wherein the performing portrait segmentation based on the standard person image, filtering out background information, and obtaining the target person image, comprising: Using the standard person image as the input of a portrait segmentation model, the portrait segmentation model is used to segment a person image from the standard person image; The target person image is generated by filtering out background information based on the person image output by the portrait segmentation model. The method according to claim 6, wherein the portrait segmentation model is obtained by training with a convolutional neural network, and includes a plurality of convolutional layers, and the convolutional layers are used to perform feature extraction on the image; Before using the convolution layer to perform feature extraction, the method further includes: performing edge enhancement on the image, so that the resolution of the image obtained after the convolution is consistent with the resolution of the input standard person image. A robot includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps: Get the original character image to be processed; Recognizing the face in the original character image, and aligning the face according to the key points of the face obtained by the identification, to obtain an aligned standard character image; Perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image; The target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model. of. The robot according to claim 8, wherein the generative adversarial network model comprises: a forward generator, a reverse generator, a first discriminator and a second discriminator; the output of the forward generator is the same as the The first discriminator is connected, and the output of the reverse generator is connected to the second discriminator; The forward generator is used to convert the input character image into a character cartoon image, the reverse generator is used to convert the input character cartoon image into a character image, and the first discriminator is used to calculate that the input image belongs to a character cartoon. The probability of the image, the second discriminant network model is used to calculate the probability that the input image belongs to the person image. The robot according to claim 9, wherein the training step of the generative adversarial network model comprises: Acquiring a training image set, the training image set includes: a training character image set and a training character cartoon image set; Input the training character image into the forward generator to obtain the first output image, and input the first output image into the first discriminator to obtain the first probability of the corresponding output; Input the training character cartoon image into the reverse generator to obtain the second output image, and input the second output image into the second discriminator to obtain the second probability of the corresponding output; Inputting the first output image into the reverse generator to obtain a third output image; inputting the second output image into the forward generator to obtain a fourth output image; Calculate the first difference value between the training character image and the third output image, and calculate the second difference value between the training character cartoon image and the fourth output image; The first loss function value is obtained by calculating according to the first probability, the second probability, the first difference value and the second difference value; Using the training character image as the input of the facial feature extraction model, obtain the first facial feature extracted by the facial feature extraction model; Using the first output image as the input of the facial feature extraction model, obtain the second facial feature extracted by the facial feature extraction model; Calculate the similarity between the first face feature and the second face feature, and calculate the second loss function value according to the similarity; A total loss value is calculated according to the first loss function value and the second loss function value, and the total loss value is inversely correlated with the similarity; Update the weight parameters in the forward generator, the reverse generator, the first discriminator and the second discriminator according to the total loss value, and repeat this cycle until the total loss value reaches Convergence condition, the forward generator obtained after training is used as the portrait cartoon model. The robot according to claim 9, wherein the forward generator comprises: an encoder and a decoder, and the encoder and the decoder are composed of Hourglass modules. The robot according to claim 8, wherein the face recognition in the original character image is performed, and face alignment is performed according to the key points of the face obtained by the recognition, and an aligned standard character image is obtained, comprising: : Determine the target coordinate position of the key point of the face in the preset space according to the preset proportion of the human body; The face key points are mapped to the target coordinate position in the preset space to obtain a standard person image aligned in the preset space. The robot according to claim 8, wherein the performing portrait segmentation based on the standard person image, filtering out background information, and obtaining the target person image, comprising: Using the standard person image as an input of a portrait segmentation model, the portrait segmentation model is used to segment a person image from the standard person image; The target person image is generated by filtering out background information based on the person image output by the portrait segmentation model. The robot according to claim 13, wherein the portrait segmentation model is obtained by training with a convolutional neural network, and includes a plurality of convolutional layers, and the convolutional layers are used to perform feature extraction on images; Before using the convolution layer to perform feature extraction, the method further includes: performing edge enhancement on the image, so that the resolution of the image obtained after the convolution is consistent with the resolution of the input standard person image. A computer-readable storage medium storing a computer program, when executed by a processor, the computer program causes the processor to perform the following steps: Get the original character image to be processed; Recognizing the face in the original character image, and aligning the face according to the key points of the face obtained by the identification, to obtain an aligned standard character image; Perform portrait segmentation based on the standard person image, filter out background information, and obtain a target person image; The target character image is used as the input of the portrait cartoon model, and the character cartoon image corresponding to the target character image output by the portrait cartoon model is obtained, and the portrait cartoon model is obtained by training a generative confrontation network model. of. The storage medium according to claim 15, wherein the generative adversarial network model comprises: a forward generator, a reverse generator, a first discriminator and a second discriminator; the output of the forward generator connected with the first discriminator, and the output of the reverse generator is connected with the second discriminator; The forward generator is used to convert the input character image into a character cartoon image, the reverse generator is used to convert the input character cartoon image into a character image, and the first discriminator is used to calculate that the input image belongs to a character cartoon. The probability of the image, the second discriminant network model is used to calculate the probability that the input image belongs to the person image. The storage medium according to claim 16, wherein the step of training the generative adversarial network model comprises: Acquiring a training image set, the training image set includes: a training character image set and a training character cartoon image set; Input the training character image into the forward generator to obtain the first output image, and input the first output image into the first discriminator to obtain the first probability of the corresponding output; Input the training character cartoon image into the reverse generator to obtain the second output image, and input the second output image into the second discriminator to obtain the second probability of the corresponding output; Inputting the first output image into the reverse generator to obtain a third output image; inputting the second output image into the forward generator to obtain a fourth output image; Calculate the first difference value between the training character image and the third output image, and calculate the second difference value between the training character cartoon image and the fourth output image; The first loss function value is obtained by calculating according to the first probability, the second probability, the first difference value and the second difference value; Using the training character image as the input of the facial feature extraction model, obtain the first facial feature extracted by the facial feature extraction model; Using the first output image as the input of the facial feature extraction model, obtain the second facial feature extracted by the facial feature extraction model; Calculate the similarity between the first face feature and the second face feature, and calculate the second loss function value according to the similarity; A total loss value is calculated according to the first loss function value and the second loss function value, and the total loss value is inversely correlated with the similarity; Update the weight parameters in the forward generator, the reverse generator, the first discriminator and the second discriminator according to the total loss value, and repeat this cycle until the total loss value reaches Convergence condition, the forward generator obtained after training is used as the portrait cartoon model. The storage medium according to claim 16, wherein the forward generator comprises: an encoder and a decoder, and the encoder and the decoder are composed of Hourglass modules. The storage medium according to claim 15, wherein the identifying the face in the original character image, aligning the face according to the identified face key points, and obtaining the aligned standard character image, include: Determine the target coordinate position of the key point of the face in the preset space according to the preset proportion of the human body; The face key points are mapped to the target coordinate position in the preset space to obtain a standard person image aligned in the preset space. The storage medium according to claim 15, wherein the performing portrait segmentation based on the standard person image, filtering out background information, and obtaining the target person image, comprising: Using the standard person image as an input of a portrait segmentation model, the portrait segmentation model is used to segment a person image from the standard person image; The target person image is generated by filtering out background information based on the person image output by the portrait segmentation model. The storage medium according to claim 15, wherein the portrait segmentation model is obtained by training a convolutional neural network, and comprises a plurality of convolutional layers, and the convolutional layers are used to perform feature extraction on images; Before using the convolution layer to perform feature extraction, the method further includes: performing edge enhancement on the image, so that the resolution of the image obtained after the convolution is consistent with the resolution of the input standard person image.

Images