TWI712032B - Voice conversion method for virtual face image - Google Patents

Abstract

A method for voice-converting virtual facial images is executed by a computer system and includes the following steps: (A) A phoneme conversion model for converting speech fragments into phonemes is used to continuously capture a voice capture unit. One of the examinee’s current speech fragments is converted into a target phoneme; (B) According to the target phoneme, it is obtained from the stored multiple frames of mouth images corresponding to a variety of different mouth shapes when a digital character emits multiple different phonemes. A target mouth shape image corresponding to the target phoneme, wherein each mouth shape image corresponds to one of the phonemes; and (C) obtaining at least one virtual face related to the digital character according to the target mouth shape image image.

Description

Voice conversion method for virtual face image

The invention relates to a conversion method, in particular to a method for voice conversion to a virtual face image.

Industries such as animation, games, and movies often create virtual characters. One of the important technologies in creating virtual characters is to enable the virtual characters to have natural and smooth mouth movements that are synchronized with the sound. Synchronization, the designer needs to adjust the mouth configuration on the timeline according to the audio content, which is very time-consuming and labor-intensive. Moreover, the animation is produced in advance and cannot be changed according to the scene atmosphere. Therefore, many existing manufacturers are committed to the development of real-time voice drive The virtual character talking technology.

However, the existing real-time voice-driven virtual character speech technology mostly converts the voice to text, and then converts the text to the virtual character's mouth image. The conversion process of first converting to text takes too long to obtain the real-time effect. .

Therefore, the purpose of the present invention is to provide a method that can instantly convert voices into virtual facial images.

Therefore, the method for voice-converting virtual facial images of the present invention is executed by a computer system, and the computer system stores multiple frames of mouth images corresponding to multiple different mouth shapes when a digital character emits multiple different phonemes, each The mouth shape image corresponds to one of the phonemes. The computer system includes a voice capturing unit for continuously capturing a current voice segment of a subject. The method of voice converting a virtual facial image includes a Step (A), one step (B), and one step (C).

In this step (A), the computer system uses a phoneme conversion model for converting a speech segment into a phoneme to convert the current speech segment into a target phoneme.

In the step (B), the computer system obtains a target mouth shape image corresponding to the target phoneme from the mouth shape images according to the target phoneme.

In the step (C), the computer system obtains at least one virtual face image related to the digital character according to the target mouth shape image.

The effect of the present invention is that after the computer system captures the current speech segment, the phoneme conversion model is used to quickly obtain the target mouth shape image corresponding to the target phoneme, and to obtain the target mouth shape image in real time according to the target mouth shape image The at least one virtual face image can achieve the effect of rapid conversion.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are represented by the same numbers.

Referring to FIG. 1, an embodiment of the method for voice conversion of a virtual face image of the present invention is executed by a computer system 1. The computer system 1 includes a storage unit 11, a voice capture unit 12, and an electrical connection to the storage unit 11 And the processing unit 13 of the voice capture unit 12.

The storage unit 11 stores multiple frames of lip-shape images, multiple conversations, multiple audiovisual data, and multiple frames related to the digital character when it emits multiple different phonemes. A variety of facial expression images of the character. It is worth noting that in this embodiment, the phonemes are, for example, OO, IY, EE, AA, WW, LL, ER, UU, FV, MM, CH, and DD, etc., and each mouth image corresponds to the One of the isophones (as shown in Figure 2), but not limited to this. Each conversation includes multiple conversation fragments, and each audiovisual data includes multiple frames of facial images related to a trainer’s speech and multiple audio fragments corresponding to the facial images. Each facial expression corresponds to an indicator For the expression parameters of the facial expression, each expression image corresponds to the expression parameter corresponding to one of the facial expressions.

The voice capturing unit 12 is used for continuously capturing a current voice segment of a subject, and the current voice segment includes at least one voice sub-segment. In this embodiment, the voice capturing unit 12 is, for example, a microphone, but it is not limited to this.

This embodiment of the method for voice conversion of virtual facial images of the present invention includes a phoneme conversion model creation program 2, an expression conversion model creation program 3, and a voice conversion virtual facial image program 4.

Referring to FIGS. 1 and 3, the phoneme conversion model establishment program 2 includes steps 21 to 23. The steps of the phoneme conversion model establishment program 2 shown in FIG. 3 are detailed below.

In step 21, the processing unit 13 extracts the features of the phonemes from an audio dictionary.

In step 22, for each conversation, the processing unit 13 generates a phoneme sequence including a plurality of permutations and combinations of the phonemes of the conversation according to the characteristics of the phonemes and the conversation, and each conversation segment of the conversation corresponds to the phoneme One of the phonemes in the sequence.

In step 23, the processing unit 13 performs machine learning calculations, such as Convolutional Neural Network (CNN), on each conversation segment and its corresponding phoneme of each conversation, to create a voice A phoneme conversion model where segments are converted to phonemes.

Referring to FIGS. 1 and 4, the expression conversion model establishment procedure includes steps 31 to 34, and each step of the expression conversion model establishment procedure shown in FIG. 4 will be described in detail below.

In step 31, for each face image of the video data stored in the storage unit 11, the processing unit 13 obtains the eyebrow portion of the face image corresponding to the face of the trainer.

In step 32, for each face image of the video and audio data stored in the storage unit 11, the processing unit 13 obtains an eyebrow feature according to the eyebrow portion in the face image. It is worth noting that in this embodiment, in steps 31 and 32, the processing unit 13 obtains the eyebrows and the features of the eyebrows. In other embodiments, the parts and features of other facial organs can also be obtained, and it is not limited thereto.

In step 33, for each facial image of the video and audio data stored in the storage unit 11, the processing unit 13 performs expression recognition on the image according to the eyebrow characteristics corresponding to the facial image to obtain the facial image. Corresponds to the expression recognition result of the trainer. It is worth noting that in this embodiment, each expression recognition result includes one of happiness, anger, sadness, and no expression. In other embodiments, each expression recognition result may include other expressions instead of Is limited.

In step 34, the processing unit 13 performs machine learning calculations on the facial expression recognition results and audio segments corresponding to each facial image to establish an expression conversion model for converting voice sub-segments into expression parameters. It is worth noting that, in this embodiment, the time length of the audio segment corresponding to each face image is equal to the time length of each voice sub-segment, that is, the time for playing one frame of image.

Referring to FIGS. 1 and 5, the voice conversion virtual face image program 4 includes steps 41 to 45. The steps of the expression conversion model establishment program shown in FIG. 5 are described in detail below.

In step 41, the processing unit 13 uses the phoneme conversion model to convert the current speech segment into a target phoneme.

In step 42, the processing unit 13 obtains a target mouth shape image corresponding to the target phoneme from the mouth shape images according to the target phoneme.

In step 43, for each of the at least one voice sub-segment of the current voice segment, the processing unit 13 uses the expression conversion model to convert the voice sub-segment into a target expression parameter.

In step 44, for each target expression parameter, the processing unit 13 obtains a target expression image corresponding to the target expression parameter from the expression images according to the target expression parameter.

It should be noted that, in this embodiment, the expression conversion model includes a double exponential smoothing method, that is, voice sub-segments at different times have different weights, and the closer the weights of the voice sub-segments at the current time are The larger is, in step 43, for each of the at least one voice sub-segment of the current voice segment, the processing unit 13 performs prediction to convert according to all the voice segments before the voice sub-segment and the voice sub-segment The target expression parameters, using the quadratic exponential smoothing method, can make the target expression image obtained in step 44 more consistent with the target expression image obtained in the previous period, because the characteristics of the present invention are not known to those skilled in the art. The sub-exponential smoothing method, so for the sake of brevity, the details of the second exponential smoothing method are omitted here.

It should be particularly noted that in this embodiment, steps 41 and 42 are performed simultaneously with steps 43 and 44. In other embodiments, steps 41 and 42 can be performed before or after steps 43 and 44. limit.

In step 45 after step 42 and step 44, the processing unit 13 obtains at least one virtual face image related to the digital character according to the target mouth image and the at least one target expression image.

It should be noted that when the current voice segment only contains a voice sub-segment, in step 43, the processing unit 13 converts the voice sub-segment into a target expression parameter, and in step 44, the processing unit 13 Obtain a target expression image corresponding to the target expression parameters from the expression images. Finally, in step 45, the processing unit 13 obtains a virtual face image according to the target mouth shape image and the target expression image; and when the current When the voice segment includes multiple voice sub-segments, in step 43, the processing unit 13 converts the voice sub-segments into multiple target expression parameters, and in step 44, the processing unit 13 obtains the expression images from the expression images. A plurality of target expression images corresponding to the target expression parameters, and finally in step 45, the processing unit 13 obtains multiple frames of virtual face images according to the target mouth shape image and the target expression images, representing the virtual faces The images have the same mouth shape.

In summary, the method for voice conversion of virtual facial images of the present invention uses the processing unit 13 to perform machine learning calculations on the speech and phoneme sequences to establish the phoneme conversion model and correspond to the facial images The expression recognition result and the audio fragments are subjected to machine learning calculations to establish the expression conversion model, so that after the voice capture unit 12 captures the current voice fragment, the processing unit 13 uses the phoneme conversion model and the expression conversion The model quickly obtains the target mouth shape image and the at least one target expression image corresponding to the target phoneme, and obtains the at least one virtual face image in real time according to the target mouth shape image and the at least one target expression image to achieve rapid The effect of conversion, in addition, the phoneme conversion model is not limited to languages, and can support mouth shape conversion in various languages, so it can indeed achieve the purpose of the invention.

However, the above are only examples of the present invention. When the scope of implementation of the present invention cannot be limited by this, all simple equivalent changes and modifications made in accordance with the scope of the patent application of the present invention and the content of the patent specification still belong to Within the scope of the patent for the present invention.

1: Computer system 11: storage unit 12: Voice capture unit 13: processing unit 21~23: Steps 31~34: Steps 41~45: Steps

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which: FIG. 1 is a block diagram illustrating a computer system for implementing the method of voice conversion virtual face image of the present invention; FIG. 2 is a schematic diagram illustrating the mouth shape image corresponding to each phoneme in the embodiment of the method of voice conversion virtual face image of the present invention; 3 is a flowchart illustrating a phoneme conversion model establishment procedure of an embodiment of the method for voice conversion to virtual facial images of the present invention; Fig. 4 is a flowchart illustrating an expression conversion model establishment procedure of this embodiment; and Fig. 5 is a flowchart illustrating a voice conversion virtual face image procedure of the embodiment.

41~45: Steps

Claims (5)

A method for voice-converting virtual facial images, executed by a computer system, the computer system stores multiple frames of mouth-shaped images and multiple conversations corresponding to multiple different mouth shapes when a digital character emits multiple different phonemes, and A plurality of phoneme sequences corresponding to the conversations, each mouth shape image corresponds to one of the phonemes, and the phoneme sequence corresponding to each conversation includes a plurality of permutations and combinations of phonemes of the conversation, and each conversation includes multiple phonemes. A conversation segment, each conversation segment corresponds to one of the phonemes in the corresponding phoneme sequence, the computer system includes a voice capturing unit for continuously capturing a current voice segment of a subject, The method for voice-converting virtual facial images includes the following steps: (A) Perform machine learning calculations on each conversation segment and its corresponding phoneme of each conversation to establish a phoneme conversion for converting the speech segment into phoneme Model; (B) using the phoneme conversion model to convert the current speech segment into a target phoneme; (C) according to the target phoneme, obtain a target mouth shape image corresponding to the target phoneme from the mouth shape images; and ( D) Obtain at least one virtual face image related to the digital character according to the target mouth shape image. According to the method for voice-converting virtual facial images in claim 1, the computer system also stores multiple frames of facial expression images related to the digital character, each facial expression corresponds to an indication of the face Expression parameters of facial expressions, each expression image corresponds to one of the facial expressions The current voice segment captured by the voice capture unit includes at least one voice sub-segment, and the following steps are also included before step (D): (F) For the at least one voice segment of the current voice segment Each of a voice sub-segment uses an expression conversion model for converting the voice sub-segment into an expression parameter to convert the voice sub-segment into a target expression parameter; and (G) for each target expression parameter, according to the The target expression parameter is to obtain a target expression image corresponding to the target expression parameter from the expression images; wherein, in step (D), the at least one virtual facial image is also obtained according to the target expression image obtained in step (G) . According to the method for voice-converting virtual facial images according to claim 2, the computer system also stores multiple pieces of audio-visual data, and each audio-visual data includes multiple frames of facial images related to a trainer’s speech and multiple corresponding The audio segment of the facial image includes the following steps before step (F): (H) For each facial image of the audio-visual data, obtain one of the organs in the facial image corresponding to the face of the trainer (I) For each facial image of the audio-visual data, obtain an organ feature according to the organ part in the facial image; (J) For each facial image of the audio-visual data, according to the The organ features corresponding to the facial image perform expression recognition on the facial image to obtain the expression recognition result of the trainer corresponding to the facial image; and (K) the expression recognition result corresponding to each facial image and The audio segment is subjected to machine learning calculations to establish the expression conversion model. The method for voice-converting a virtual face image according to claim 3, wherein, in step (H), each organ part is the eyebrow corresponding to the face of the trainer. The method for voice-converting virtual facial images according to claim 3, wherein in step (J), each facial expression recognition result includes one of happy, angry, sad, and expressionless.

Images