CN111652165B - Mouth evaluation method, equipment and computer storage medium - Google Patents

Abstract

Embodiments of the present application provide a mouth shape evaluation method, equipment, and computer storage medium. The mouth shape evaluation method includes: acquiring data to be evaluated of a target object, the data to be evaluated includes a sequence of image frames, and the sequence of image frames includes an image representing the mouth shape of the target pronunciation. at least one continuous image frame; determine the feature matrix of the image frame sequence according to the image frame sequence; obtain first evaluation data according to the feature matrix of the image frame sequence and the preset model, and the first evaluation data is used to indicate the feature matrix according to the preset model. The evaluation result; the lip shape evaluation information of the target object is generated according to the evaluation data, the evaluation data includes the first evaluation data, and the lip shape evaluation information is used to indicate the evaluation result of the target pronunciation lip shape of the target object. It can evaluate whether the pronunciation mouth shape is accurate, and the specific accuracy of the pronunciation mouth shape.

Description

Mouth evaluation method, equipment and computer storage medium

technical field

The embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to a mouth shape evaluation method, device, and computer storage medium.

Background technique

Artificial intelligence technology is widely used in various fields of life. In some application scenarios, for example, in the field of language teaching, artificial intelligence technology can be used to evaluate students' pronunciation; another example, in the field of vocal music teaching buildings, artificial intelligence technology can be used. Evaluate students' singing pronunciation. Of course, this is just an example. However, in the process of speech evaluation, using audio for evaluation can only ensure that the continuous pronunciations are roughly similar, and cannot accurately evaluate the accuracy of the pronunciation of the mouth shape.

SUMMARY OF THE INVENTION

In view of this, one of the technical problems solved by the embodiments of the present invention is to provide a mouth shape evaluation method, device and computer storage medium to overcome the inability to accurately evaluate the accuracy of pronunciation mouth shapes in the prior art.

The embodiment of the present application provides a mouth shape evaluation method, which includes:

Acquiring data to be evaluated of the target object, the data to be evaluated includes a sequence of image frames, and the sequence of image frames includes at least one continuous image frame representing the mouth shape of the target pronunciation;

Determine the feature matrix of the image frame sequence according to the image frame sequence;

Obtain first evaluation data according to the feature matrix of the image frame sequence and the preset model, where the first evaluation data is used to indicate the result of evaluating the feature matrix according to the preset model;

The mouth shape evaluation information of the target object is generated according to the evaluation data, the evaluation data includes the first evaluation data, and the mouth shape evaluation information is used to indicate the evaluation result of the target pronunciation mouth shape of the target object.

Optionally, in an embodiment of the present application, determining a feature matrix of the image frame sequence according to the image frame sequence, including:

In the image frames included in the image frame sequence, determine the coordinates of at least one key point of the mouth;

According to the number of image frames in the image frame sequence, the number of mouth key points in each image frame, and the dimension of the coordinates, a feature matrix of the image frame sequence is generated in time sequence, and the feature matrix of the image frame sequence includes at least one mouth key point. coordinate.

Optionally, in an embodiment of the present application, the preset model includes a standard pronunciation curve model; the first evaluation data is obtained according to the feature matrix of the image frame sequence and the preset model, including:

Input the coordinates of at least one key point of the mouth into the standard pronunciation curve model to obtain the score of at least one key point of the mouth; obtain the mouth shape score of the image frame according to the score of the at least one key point of the mouth; The scores are normalized and the first evaluation data is generated.

Optionally, in an embodiment of the present application, the preset model includes a preset transpose matrix, and the first evaluation data is obtained according to the feature matrix of the image frame sequence and the preset model, including:

The feature matrix of the image frame sequence is dot-multiplied with the preset transposed matrix to obtain the pronunciation matrix;

The first evaluation data is obtained by comparing the pronunciation matrix with the preset standard pronunciation matrix.

Optionally, in an embodiment of the present application, the method further includes:

Perform mouth shape evaluation on the image frame sequence, and obtain second evaluation data, where the second evaluation data is used to represent the similarity between the target pronunciation mouth shape in the image frame and the preset standard pronunciation mouth shape;

Generate lip-synch evaluation information of the target object according to the evaluation data, including:

A weighted operation is performed on the first evaluation data and the second evaluation data to generate mouth shape evaluation information, where the evaluation data includes the first evaluation data and the second evaluation data.

Optionally, in an embodiment of the present application, the method further includes:

Perform mouth shape recognition on at least one image frame in the sequence of image frames;

The similarity between the target pronunciation mouth shape and the preset standard pronunciation mouth shape is determined according to the recognition result, and the second evaluation data is generated.

Optionally, in an embodiment of the present application, before determining the similarity between the target pronunciation mouth shape and the preset standard pronunciation mouth shape according to the recognition result, the method further includes:

The image frame sequence is aligned with the frame number of the reference frame sequence, and the reference frame sequence is a preset frame sequence representing the standard pronunciation mouth shape.

Optionally, in an embodiment of the present application, aligning the frame numbers of the image frame sequence with the reference frame sequence includes:

The image frame sequence is down-sampled or linearly interpolated, so that the image frame sequence is aligned with the frame numbers of the reference frame sequence.

Optionally, in an embodiment of the present application, the data to be evaluated further includes audio data corresponding to the sequence of image frames, and the method further includes:

The audio data is evaluated according to the preset reference audio to generate third evaluation data, where the third evaluation data is used to represent the pronunciation similarity between the audio data and the reference audio;

Generate lip-synch evaluation information of the target object according to the evaluation data, including:

A weighted operation is performed on the first evaluation data and the third evaluation data to generate mouth shape evaluation information, where the evaluation data includes the first evaluation data and the third evaluation data.

Optionally, in an embodiment of the present application, the audio data is evaluated according to a preset reference audio to generate third evaluation data, including:

Extending the start time of the audio data forward by a first preset time length, and extending the start time of the audio data backward by a second preset time length, to obtain the extended audio data;

The extended audio data is evaluated according to the preset reference audio to generate third evaluation data.

An embodiment of the present application provides a mouth shape evaluation device, including: an acquisition module, a matrix module, an evaluation module, and a synthesis module;

Wherein, the obtaining module is used to obtain the data to be evaluated of the target object, the data to be evaluated includes a sequence of image frames, and the sequence of image frames includes at least one continuous image frame representing the mouth shape of the target pronunciation;

The matrix module is used to determine the feature matrix of the image frame sequence according to the image frame sequence;

The evaluation module is configured to obtain first evaluation data according to the feature matrix of the image frame sequence and the preset model, and the first evaluation data is used to indicate the evaluation result of the feature matrix according to the preset model;

The comprehensive module is used for generating mouth shape evaluation information of the target object according to the evaluation data, the evaluation data includes the first evaluation data, and the mouth shape evaluation information is used to indicate the evaluation result of the target pronunciation mouth shape of the target object.

An embodiment of the present application provides an electronic device, including: a processor and a memory, the processor is connected to the memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the mouth shape evaluation method described in any embodiment of the present application .

An embodiment of the present application provides a computer storage medium, including: the computer storage medium stores a computer program, and when the processor executes the computer program, the mouth shape evaluation method described in any embodiment of the present application is implemented.

In the embodiment of the present application, the feature matrix of the image frame sequence is determined according to at least one continuous image frame in the image frame sequence, and the first evaluation data is obtained according to the feature matrix and the preset model, so as to evaluate the accuracy of the pronunciation mouth shape. Assume that the model represents the standard pronunciation mouth shape, and the points on the preset model represent the change process of the standard pronunciation mouth shape. Using the preset model to evaluate the feature matrix of the true image sequence, the changes in the pronunciation mouth shape can be determined. In , the gap between the pronunciation mouth shape represented by the image frame sequence and the standard pronunciation mouth shape can be used to evaluate whether the pronunciation mouth shape is accurate and the specific accuracy of the pronunciation mouth shape.

Description of drawings

Hereinafter, some specific embodiments of the embodiments of the present application will be described in detail by way of example and not limitation with reference to the accompanying drawings. The same reference numbers in the figures designate the same or similar parts or parts. It will be understood by those skilled in the art that the drawings are not necessarily to scale. In the attached picture:

1 is a flowchart of a mouth shape evaluation method provided in Embodiment 1 of the present application;

2 is a flowchart of a mouth shape evaluation method provided in Embodiment 2 of the present application;

2a is a schematic diagram of a waveform of audio data according to Embodiment 2 of the present application;

2b is a schematic diagram of a key point of a human face provided in Embodiment 2 of the present application;

FIG. 2c is a schematic diagram of a phoneme decomposition effect provided in Embodiment 2 of the present application;

FIG. 3 is a structural diagram of a mouth shape evaluation device provided in Embodiment 3 of the present application;

FIG. 4 is a structural diagram of an electronic device according to Embodiment 4 of the present application.

Detailed ways

The specific implementation of the embodiments of the present invention is further described below with reference to the accompanying drawings of the embodiments of the present invention.

Embodiment 1.

Embodiment 1 of the present application provides a mouth shape evaluation method, as shown in FIG. 1 , which is a flowchart of a mouth shape evaluation method provided by an embodiment of the present application. The mouth shape evaluation method includes the following steps:

Step 101: Acquire the data to be evaluated of the target object.

It should be noted that the mouth shape evaluation method provided in this application is used to evaluate the accuracy of the mouth shape of the target object when making the target sound. The target object can be any person or any animal, etc., which is not limited in this application. , the target sound can be a word, or a sentence, or a paragraph of a certain language, or a piece of music from a certain piece of music, or a sentence or a paragraph of a song. Of course, this is just an exemplary illustration. It does not mean that the present application is limited to this.

The data to be evaluated includes a sequence of image frames, the sequence of image frames includes at least one continuous image frame representing the target pronunciation mouth shape, and the data to be evaluated may also include audio data corresponding to the sequence of image frames. It should be noted that the target pronunciation mouth shape is the mouth shape when the target object emits the target sound. In this application, the target is only used to represent the singular without any limitation, and it can be any person or any animal. The mouth shape at the time of the sound, of course, is only an example here. The sequence of image frames includes at least one continuous image frame when the target object emits the target sound. It should be noted that the continuous at least one image frame here only indicates that the image frames are arranged in chronological order, and does not represent two consecutive image frames. The time difference between image frames is very small. For example, the image frame sequence includes 10 image frames. According to the time sequence, the (n+1)th image frame is sorted after the nth image frame, and n is an integer greater than 0 and less than 10. . As for the time interval between two image frames, the time interval between every two adjacent image frames may be the same or different, and may be 3 ms, 5 ms, or 1 s, which is not limited in this application.

Step 102: Determine a feature matrix of the image frame sequence according to the image frame sequence.

It should be noted that the feature matrix of the image frame sequence may represent the shape feature of the mouth in each image frame of the image frame sequence. The feature matrix of the image frame sequence may be composed of feature vectors of the image frame. For example, one image frame corresponds to one feature vector, and the feature vector of at least one image frame, that is, at least one feature vector constitutes the feature matrix of the image frame sequence. Of course, this is only an exemplary description, which does not mean that the present application is limited thereto.

Optionally, in an embodiment of the present application, determining a feature matrix of the image frame sequence according to the image frame sequence, including:

In the image frames included in the image frame sequence, the coordinates of at least one mouth key point are determined; according to the number of image frames in the image frame sequence, the number of mouth key points in each image frame, and the dimension of the coordinates, images are generated in time sequence The feature matrix of the frame sequence, the feature matrix of the image frame sequence includes the coordinates of at least one mouth key point. Specifically, the feature matrix of the image frame sequence includes the feature vector of each image frame in the image frame sequence, and the feature vector of one image frame may include a set of coordinates of key points of the mouth. For example, if there are n key points of the mouth, then one The feature vector of the image frame may include the coordinates of n mouth keypoints.

In this application, because it is a mouth shape evaluation, the key points of the mouth can be selected, that is, the lip region of the image frame can be selected. Because the coordinates of the key points of the mouth include horizontal and vertical coordinates, the dimension of the coordinates can be 2, and the feature matrix can be expressed as S(N,k,2), where S represents the feature matrix and N represents the number of image frames , k represents the number of keypoints. Of course, this is only an exemplary description, which does not mean that the present application is limited thereto.

Optionally, in an embodiment of the present application, the method further includes: aligning the frame numbers of the image frame sequence with the reference frame sequence, where the reference frame sequence is a preset frame sequence representing standard pronunciation mouth shapes. Further optionally, the image frame sequence may be down-sampled or linearly interpolated, so that the image frame sequence is aligned with the frame numbers of the reference frame sequence.

Because the reference frame sequence represents the standard pronunciation mouth shape, of course, the standard pronunciation mouth shape and the target pronunciation mouth shape represented by the image frame sequence are both uttering the same sound, for example, they are both uttering the word "congratulation" Word, the target pronunciation mouth shape is the mouth shape of the target object when the word "congratulation" is pronounced, and the standard pronunciation mouth shape is the mouth shape of the teacher or a more professional language expert when the word "congratulation" is issued. The frame sequence is evaluated to achieve the purpose of lip-synch evaluation. Before the evaluation, the image frame sequence is aligned with the reference frame sequence, so that the number of frames is the same, so that the evaluation accuracy is higher. Perform down-sampling to reduce the number of frames of the image frame sequence; when the number of frames of the image frame sequence is less than that of the reference frame sequence, perform linear interpolation on the image frame sequence to increase the number of frames of the image frame sequence; If the number is equal to the number of frames in the reference frame sequence, it indicates that the number of frames has been aligned and the evaluation can begin.

Step 103: Obtain first evaluation data according to the feature matrix of the image frame sequence and the preset model.

The first evaluation data is used to indicate the result of evaluating the feature matrix according to the preset model. With reference to the description in step 102, the preset model may be trained according to the feature matrix of the reference frame sequence. For example, there are J types of standard pronunciation mouth shapes in the database, and each standard pronunciation mouth shape is used to make a sound, which can be the standard pronunciation of sentences, words, and the like. A feature matrix is constructed for the reference frame sequence of each standard pronunciation mouth shape, and a one-dimensional model is constructed. Specifically, the coordinates of the k lip key points of each reference frame in a reference frame sequence can be used as input, the manually marked score can be used as output, and at least one neuron can be used to fit a curve model of the standard pronunciation mouth shape , a curve model can be fitted to the reference frame sequence of each standard pronunciation mouth shape, and there can be J curve models in the database.

Here, two specific examples are given to illustrate how to obtain the first evaluation data.

Optionally, in the first example of the present application, the preset model includes a standard pronunciation curve model; the first evaluation data is obtained according to the feature matrix of the image frame sequence and the preset model, including:

Input the coordinates of at least one key point of the mouth into the standard pronunciation curve model to obtain the score of at least one key point of the mouth; obtain the mouth shape score of the image frame according to the score of the at least one key point of the mouth; The scores are normalized and the first evaluation data is generated.

The feature vector represents the coordinates of the key points of the mouth of the image frame, and the standard pronunciation curve model is used to score the feature vector, that is, the key points of the mouth of each image frame are scored, and then the mouth shape of each image frame is scored. , the score is used to express the accuracy of the vocal lip. The standard pronunciation curve model can be trained by using the feature vectors of the pronunciation mouth shapes of multiple samples and the manually annotated scores.

Optionally, in the second embodiment of the present application, the preset model includes a preset transpose matrix, and the first evaluation data is obtained according to the feature matrix of the image frame sequence and the preset model, including:

Dot-multiply the feature matrix of the image frame sequence and the preset transposition matrix to obtain the pronunciation matrix; compare the pronunciation matrix with the preset standard pronunciation matrix to obtain the first evaluation data.

Step 104 , generating lip shape evaluation information of the target object according to the evaluation data.

The evaluation data includes the first evaluation data, and the mouth shape evaluation information is used to represent the evaluation result of the target pronunciation mouth shape of the target object.

The mouth shape evaluation information can also be evaluated in combination with more data. Here, three specific examples are listed as follows:

Optionally, in the first example, the method further includes: performing mouth shape evaluation on the image frame sequence, and obtaining second evaluation data, where the second evaluation data is used to represent the target pronunciation mouth shape in the image frame and the preset mouth shape. The similarity of standard pronunciation mouth shapes;

Generating the mouth shape evaluation information of the target object according to the evaluation data includes: performing a weighted operation on the first evaluation data and the second evaluation data to generate the mouth shape evaluation information, and the evaluation data includes the first evaluation data and the second evaluation data. The first evaluation data can represent the evaluation results of the target object in the process of continuous pronunciation, and the second evaluation data can represent the evaluation results of the target object's pronunciation and mouth shape in each image frame, combining the first evaluation data and the second evaluation data. The lip evaluation of the target object is more accurate and comprehensive.

Further optionally, it is described how to obtain the second evaluation data. In an embodiment of the present application, the method further includes: performing mouth shape recognition on at least one image frame in the image frame sequence; determining the target pronunciation according to the recognition result The similarity between the mouth shape and the preset standard pronunciation mouth shape, and generate the second evaluation data.

Optionally, in conjunction with the description of step 102, before determining the similarity between the target pronunciation mouth shape and the preset standard pronunciation mouth shape according to the recognition result, the method may also include: aligning the frame number of the image frame sequence with the reference frame sequence, The reference frame sequence is a preset frame sequence representing the standard pronunciation mouth shape. Further optionally, aligning the frame numbers of the image frame sequence with the reference frame sequence includes: performing down-sampling or linear interpolation on the image frame sequence, so that the image frame sequence is aligned with the frame numbers of the reference frame sequence. Align the image frame sequence with the reference frame sequence so that the number of frames is the same, so that the evaluation accuracy is higher.

Optionally, in the second example, the data to be evaluated further includes audio data corresponding to the image frame sequence, and the method further includes: evaluating the audio data according to a preset reference audio to generate third evaluation data, and the third evaluation The data is used to represent the pronunciation similarity between the audio data and the reference audio;

Generating the mouth shape evaluation information of the target object according to the evaluation data includes: performing a weighted operation on the first evaluation data and the third evaluation data to generate the mouth shape evaluation information, and the evaluation data includes the first evaluation data and the third evaluation data. The third evaluation data is used to represent the result of evaluating the audio data of the target object, and the mouth shape evaluation of the target object is more accurate and comprehensive by combining the first evaluation data and the third evaluation data.

Optionally, the audio data is evaluated according to the preset reference audio to generate third evaluation data, including:

Extend the start time of the audio data forward by a first preset duration, and extend the start time of the audio data backward by a second preset duration to obtain the extended audio data; The audio data is evaluated to generate third evaluation data. Extending the time before and after the audio data plays a certain role in noise reduction and regularization, which can improve the evaluation accuracy.

Optionally, in the third example, the lip shape evaluation information of the target object is generated according to the evaluation data, including:

Perform a weighted operation on the first evaluation data, the second evaluation data, and the third evaluation data to generate mouth shape evaluation information. The evaluation data includes the first evaluation data, the second evaluation data and the third evaluation data, and the second evaluation data is used to represent The similarity between the target pronunciation mouth shape in the image frame and the preset standard pronunciation mouth shape, and the third evaluation data is used to indicate the pronunciation similarity between the audio data and the reference audio.

In the embodiment of the present application, the feature matrix of the image frame sequence is determined according to at least one continuous image frame in the image frame sequence, and the first evaluation data is obtained according to the feature matrix and the preset model, so as to evaluate the accuracy of the pronunciation mouth shape. Assume that the model represents the standard pronunciation mouth shape, and the points on the preset model represent the change process of the standard pronunciation mouth shape. Using the preset model to evaluate the feature matrix of the true image sequence, the changes in the pronunciation mouth shape can be determined. In , the gap between the pronunciation mouth shape represented by the image frame sequence and the standard pronunciation mouth shape can be used to evaluate whether the pronunciation mouth shape is accurate and the specific accuracy of the pronunciation mouth shape.

Embodiment two,

Based on the mouth shape evaluation method described in the first embodiment, the second embodiment of the present application further provides a mouth shape evaluation method, which is a specific implementation of the method in the first embodiment. Referring to FIG. 2 , FIG. 2 is the present application. A flow chart of a mouth shape evaluation method provided by the second embodiment, the method comprises the following steps:

Step 201: Acquire data to be evaluated.

In this embodiment, the data to be evaluated includes a pronunciation video, and an image frame sequence and audio data can be cut out from the pronunciation video. In one application scenario, for example, in the classroom, students can turn on the camera and microphone throughout the whole process, and start to intercept the pronunciation video of students reading letters, words or sentences when entering the lip-synch evaluation. After the lip-evaluation process, Stop the acquisition, and the acquired video contains image frame sequence and audio data.

Step 202: Separate audio data from the data to be evaluated.

Step 203 , perform noise reduction processing on the audio data.

For example, Wiener filtering can be used to perform noise reduction processing on audio data to reduce the signal-to-noise ratio, thereby improving the accuracy of audio data recognition and the accuracy of audio data scoring.

Step 204: Cut out the audio data segment of the target sound from the audio data.

For example, the first prominent peak can be found in the waveform of the audio data, and the first prominent peak can be the starting moment when the student begins to speak; The pronunciation ends, and the last wave peak is determined as the end time of the pronunciation. As shown in FIG. 2a, FIG. 2a is a schematic diagram of a waveform of audio data according to Embodiment 2 of the present application. Because the reference audio (that is, the audio data of the standard pronunciation) is recorded in advance, in order to improve the evaluation accuracy, it is necessary to align the audio data segment with the reference audio. For example, the reference audio extends the start of the utterance forward by 100 milliseconds, i.e., extends the end moment back by 100 milliseconds, to make the reference audio easier to discern, so the audio data segment can do the same to align with the standard pronunciation , to improve the evaluation accuracy.

It should be noted that the start time and end time of pronunciation can be detected by voice activity detection (English: Voice Activity Detection, VAD) technology, and the time of all pronunciation can be detected by voice activity detection technology, for example, through voice activity detection The technology obtains the pronunciation judgment results of multiple times. If the pronunciation is marked as 1 and the unpronounced is marked as 0, the times corresponding to the first 1 and the last 1 are taken as the start and end times of the pronunciation. Of course, this is only an exemplary description, which does not mean that the present application is limited thereto.

Step 205 , comparing the audio data segment with the reference audio to obtain third evaluation data.

The third evaluation data represents the pronunciation similarity between the audio data and the reference audio. Of course, the third evaluation data may also represent the difference between the audio data and the reference audio, and in short, it can be used to evaluate the pronunciation accuracy of the audio data.

Step 206: Capture a sequence of image frames in the pronunciation video according to the start time and end time of the audio data segment.

According to the start time and end time of the audio data segment, the corresponding image frame sequence can be cut out. Similarly, the start time of the image frame sequence can be extended forward by 100ms, and the end time of the image frame sequence can be extended backward by 100ms , which can not only align with the reference audio, but also play a certain role in noise reduction and regularization.

Step 207: Perform face detection on the image frames in the image frame sequence.

The image frame is represented by f, and the image frame sequence can be represented as f ₀ , f ₁ , f ₂ ...... f _n , and face detection is performed frame by frame to determine the face area. Before face detection, the image frame needs to be preprocessed, including converting the original channel RGB image frame to BGR, scaling the image size to 512×512 pixels, etc. Input the preprocessed image frame into the face detection model for detection and determine the face area. For example, P(x ₁ , y ₁ , x ₂ , y ₂ ) can be used to represent the face area, (x ₁ , y ₁ ) and (x ₂ , y ₂ ) can respectively represent the vertex coordinates of the two opposite corners of the face region.

Step 208: Perform key point detection on the face region in the image frame to obtain key point information.

In this application, the key points may include the key points of the mouth. Of course, the key points may also include the key points of other regions, as long as they can be combined to evaluate the pronunciation mouth shape. The keypoint information may include coordinates of at least one mouth keypoint. Before the key point detection, the face area can be extended to the outside. The expansion can increase the robustness. If it is not extended, some points will be predicted to the edge of the picture, causing a large error. After the expansion, you can Better return to the correct face. Specifically, x ₁ =x ₁ -(x ₂ -x ₁ )×0.2, x ₂ =x ₂ +(x ₂ -x ₁ )×0.2, y ₁ =y ₁ -(y ₂ -y ₁ )×0.2 , y ₂ =y ₂ +(y ₂ -y ₁ )×0.2, after the above expansion operation, the face is cut out, and the cut face is scaled to the preset size in the model, that is, to adapt to the size of the model The size, for example, can be to scale the face to a size of 112×112, and then divide the scaled image by 256. After the above processing, the processed image is input into the face key point detection model, and finally 106 key points of the outline of the face organs are obtained. Of course, this is only an exemplary illustration, and the number of key points can also be 100 or 50, which is not limited in this application. As shown in FIG. 2b, FIG. 2b is a schematic diagram of a face key point according to Embodiment 2 of the present application.

Step 209 , extract a mouth image from the face region of the image frame according to the key point information, and perform mouth shape recognition.

Taking the English pronunciation as an example, according to the definition of the English international pronunciation standard, mouth shapes are divided into 48 categories. The mouth shape recognition model can be trained by using the mouth shape image samples and the category data marked on the samples. The previous category does not belong to all other categories of mouth shapes, for a total of 49 categories.

The mouth image is scaled to a size of 64×64 and divided by 256, and then the mouth image is input into the mouth shape recognition model to obtain the mouth shape of the mouth in each image frame in the image frame sequence.

Step 210: Align the image frame sequence with the reference frame sequence.

It should be noted that, after step 209, the image frames in the image frame sequence may only retain the mouth image, or may retain the complete image, which is not limited in this application. Each image frame corresponds to a mouth shape category, and the image frame sequence can also be replaced with a mouth shape category sequence. Of course, its function is the same, which is not limited in this application. It should be noted that the alignment of the image frame sequence and the reference frame sequence can be performed after step 206, that is, after the image frame sequence is intercepted in the pronunciation video, the image frame sequence is aligned with the reference frame sequence, or the image frame sequence can be aligned in step 209. After extracting the mouth image, the image frame sequence is aligned with the reference frame sequence, which is not limited in this application, as long as the similarity between the target pronunciation mouth shape and the preset standard pronunciation mouth shape is determined, the image frame sequence is aligned with the reference frame. Align the sequences.

Because different students spend different time reading the same word or sentence, the number of frames is different from the reference frame sequence in the database (that is, the number of standard pronunciation frames), and the frame-by-frame comparison of mouth shapes cannot be done. Downsampling or linear interpolation is performed to ensure the same number of frames as the reference frame sequence.

Since a word or sentence is likely to be read fast in the first half and slow in the second half, uniform downsampling or linear difference will result in too few frames in the fast part, which will lead to The recognition performance becomes poor, so non-uniform downsampling or linear difference can be used.

For example, the standard pronunciation in the database and the collected pronunciation of the target object can be phoneme-decomposed. For example, the word "congratulation" will be divided into con'gra'tu'la'tion according to the phoneme after the phoneme decomposition. Fig. 2c is a schematic diagram of a phoneme decomposition effect provided in the second embodiment of the present application, where the pronunciation phoneme of the target object is aligned with the phoneme of the standard pronunciation.

For example, if the phoneme of con pronounced by the target object is 10 frames, and the standard pronunciation is 8 frames, the image frame sequence of the target object is down-sampled; if the target object is pronounced 8 frames, the standard pronunciation is 10 frames, the target object The sequence of image frames is linearly interpolated to keep the number of frames exactly the same.

Step 211: Determine the similarity between the target pronunciation mouth shape and the preset standard pronunciation mouth shape, and generate second evaluation data.

For example, the Euclidean distance can be used to calculate the similarity between a sequence of image frames and a sequence of reference frames. For each image frame, it is compared with the reference frame in the reference frame sequence, and the similarity is calculated, that is, each image frame in the image frame sequence is evaluated, and the similarity between each image frame and the reference frame is used. The second evaluation data can be generated.

Step 212: Determine a feature matrix of the image frame sequence according to the image frame sequence.

The feature matrix of the sequence of image frames may consist of feature vectors of at least one image frame. E.g,

After the mouth key points are extracted, the mouth key points in each image frame can be expressed as (20, 2), where 20 is the number of key points, and 2 means that each key point has two coordinates from x and y. The dimensional description is equivalent to 20 mouth coordinates, and the coordinates of the key points of the mouth in an image frame can form the feature vector of an image frame. Of course, only 20 key points are used as an example for description, which does not mean that the present application is limited to this.

The feature vectors of all image frames are combined into a feature matrix in order, which is the feature matrix of the image frame sequence. If the number of image frames in the image frame sequence is N, the feature matrix can be expressed as S(N, 20, 2), where N represents the number of image frames; if the reference frame sequence has M reference frames, the feature matrix of the reference frame sequence The matrix can be represented as T(M, 20, 2), where M is the number of reference frame sequences.

Further, for example, there are J standard pronunciations of sentences and words in the database, a feature matrix is constructed for each standard pronunciation reference frame sequence, and a one-dimensional model is constructed. Specifically, the coordinates of 20 key points of the mouth of a standard pronunciation can be used as input, the score of manual annotation can be used as output, and 10 neurons can be used to fit a standard pronunciation curve model of the standard pronunciation. The standard pronunciation curve The model can both be used to detect the accuracy of the standard pronunciation of this type of mouth shape. There can be J standard pronunciation curve models in the database.

Step 213: Input the feature vector of the image frame into the standard pronunciation curve model to obtain the mouth shape score of the image frame, and perform normalization processing to generate the first evaluation data.

The first evaluation data can represent the evaluation result of the continuously changing pronunciation mouth shape.

Step 214: Perform a weighted operation on the first evaluation data, the second evaluation data, and the third evaluation data to generate mouth shape evaluation information.

The third evaluation data obtained in step 205, the second evaluation data obtained in step 211, and the first evaluation data obtained in step 213 are weighted to obtain the final mouth shape evaluation result. The mouth shape of one frame is evaluated for the continuous mouth shape changes, and the mouth shape evaluation information obtained by the comprehensive evaluation can more accurately evaluate the accuracy of the pronunciation mouth shape. It should be noted that the first evaluation data, the second evaluation data, and the third evaluation data may be multiplied by the same weight, or may be multiplied by different weights according to user needs, which is not limited in this application. The sum of the evaluation data, the second evaluation data, and the third evaluation data is used to generate mouth shape evaluation information, that is, the weight of each evaluation data is 1.

In addition to calculating the first evaluation data by building a one-dimensional model, it can also be calculated in the following ways:

The feature vectors of all image frames are combined into a feature matrix in order, which is the feature matrix of the image frame sequence. If the number of image frames in the image frame sequence is N, the feature matrix can be expressed as S(N, 20, 2), where N represents the number of image frames; if the reference frame sequence has M reference frames, the feature matrix of the reference frame sequence The matrix can be represented as T(M, 20, 2), where M is the number of reference frame sequences. Preset multiple transposed matrices. The transposed matrix can unify the two matrices to be compared to the same dimension. For example, a transposed matrix M×N multiplied by S(N, 20, 2) can be obtained by multiplying with T(M , 20, 2) Matrix of the same dimension, then compare the difference between these two matrices, multiply the feature matrix point of the image frame sequence by the transposed matrix, and then compare it with the feature matrix of the reference frame sequence to obtain the evaluation result. It should be noted that, by comparing the differences between the two matrices, the Euclidean distance of the two matrices can be calculated, and the similarity of the two matrices can be determined according to the Euclidean distance. Of course, this is only an exemplary description, which does not mean that the present application is limited thereto.

Embodiment three,

An embodiment of the present application provides a mouth shape evaluation device, including: an acquisition module 301, a matrix module 302, an evaluation module 303, and a synthesis module 3014;

Wherein, the obtaining module 301 is used to obtain the data to be evaluated of the target object, the data to be evaluated includes a sequence of image frames, and the sequence of image frames includes at least one continuous image frame representing the mouth shape of the target pronunciation;

a matrix module 302, configured to determine a feature matrix of the image frame sequence according to the image frame sequence;

An evaluation module 303, configured to obtain first evaluation data according to a feature matrix of the image frame sequence and a preset model, where the first evaluation data is used to indicate a result of evaluating the feature matrix according to the preset model;

The synthesis module 304 is used for generating lip shape evaluation information of the target object according to the evaluation data, the evaluation data includes first evaluation data, and the lip shape evaluation information is used to represent the evaluation result of the target pronunciation lip shape of the target object.

Optionally, in an embodiment of the present application, the matrix module 302 is further configured to determine the coordinates of at least one key point of the mouth in the image frames included in the image frame sequence; according to the number of image frames in the image frame sequence , the number of key points of the mouth in each image frame and the dimension of the coordinates, and the feature matrix of the sequence of image frames is generated in time sequence, and the feature matrix of the sequence of image frames includes the coordinates of at least one key point of the mouth.

Optionally, in an embodiment of the present application, the preset model includes a standard pronunciation curve model; the evaluation module 303 is further configured to input the coordinates of at least one key point of the mouth into the standard pronunciation curve model to obtain at least one mouth Scoring of key points; obtaining a mouth shape score of the image frame according to the score of at least one mouth key point; normalizing the mouth shape score of at least one image frame to generate first evaluation data.

Optionally, in an embodiment of the present application, the preset model includes a preset transpose matrix, and the evaluation module 303 is further configured to perform a dot product between the feature matrix of the image frame sequence and the preset transpose matrix to obtain Pronunciation matrix; compare the pronunciation matrix with the preset standard pronunciation matrix to obtain the first evaluation data.

Optionally, in an embodiment of the present application, the evaluation module 303 is further configured to perform mouth shape evaluation on the image frame sequence, and obtain second evaluation data, where the second evaluation data is used to represent the target pronunciation mouth in the image frame. The similarity between the shape and the preset standard pronunciation mouth shape;

The synthesis module 304 is further configured to perform a weighted operation on the first evaluation data and the second evaluation data and generate mouth shape evaluation information, where the evaluation data includes the first evaluation data and the second evaluation data.

Optionally, in an embodiment of the present application, the evaluation module 303 is further configured to perform mouth shape recognition on at least one image frame in the image frame sequence; determine the target pronunciation mouth shape and the preset standard pronunciation according to the recognition result. The similarity of the mouth shape, and generate the second evaluation data.

Optionally, in an embodiment of the present application, the evaluation module 303 aligns the frame numbers of the image frame sequence with the reference frame sequence, where the reference frame sequence is a preset frame sequence representing standard pronunciation mouth shapes.

Optionally, in an embodiment of the present application, the evaluation module 303 is further configured to perform downsampling or linear interpolation on the image frame sequence, so that the image frame sequence is aligned with the frame numbers of the reference frame sequence, and the reference frame sequence is: A preset sequence of frames representing standard pronunciation mouth shapes.

Optionally, in an embodiment of the present application, the evaluation module 303 is further configured to evaluate the audio data according to the preset reference audio to generate third evaluation data, where the third evaluation data is used to represent the audio data and the reference audio. pronunciation similarity;

The synthesis module 304 is further configured to perform a weighted operation on the first evaluation data and the third evaluation data and generate mouth shape evaluation information, where the evaluation data includes the first evaluation data and the third evaluation data.

Optionally, in an embodiment of the present application, the evaluation module 303 is further configured to extend the start time of the audio data forward by a first preset duration, and extend the start time of the audio data backward by a second time. A preset time length is used to obtain extended audio data; and third evaluation data is generated by evaluating the extended audio data according to a preset reference audio.

Embodiment four,

An embodiment of the present application provides an electronic device, including: as shown in FIG. 4 , the electronic device includes:

At least one processor (processor) 402 and memory (memory) 404.

in:

The memory 404 is used to store the program 410 . Memory 404 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory. Specifically, the program 410 may include program code including computer operation instructions.

The processor 402 is configured to execute the program 410, and specifically may execute the relevant steps in the methods described in the first embodiment and the second embodiment.

Optionally, in an embodiment of the present application, the electronic device may further include a bus 406 and a communications interface (Communications Interface) 408 .

The processor 402 , the communication interface 408 , and the memory 404 can communicate with each other through the communication bus 406 .

A communication interface 408 for communicating with other devices.

The processor 402 may be a central processing unit CPU, or a special integrated circuit ASIC

(Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the electronic device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

Embodiment five,

An embodiment of the present application provides a computer storage medium, including: the computer storage medium stores a computer program, and when the processor executes the computer program, the mouth shape evaluation method described in any embodiment of the present application is implemented.

In the embodiment of the present application, the feature matrix of the image frame sequence is determined according to at least one continuous image frame in the image frame sequence, and the first evaluation data is obtained according to the feature matrix and the preset model, so as to evaluate the accuracy of the pronunciation mouth shape. Assume that the model represents the standard pronunciation mouth shape, and the points on the preset model represent the change process of the standard pronunciation mouth shape. Using the preset model to evaluate the feature matrix of the true image sequence, the changes in the pronunciation mouth shape can be determined. In , the gap between the pronunciation mouth shape represented by the image frame sequence and the standard pronunciation mouth shape can be used to evaluate whether the pronunciation mouth shape is accurate and the specific accuracy of the pronunciation mouth shape.

The mouth shape evaluation devices of the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones (eg iPhone), multimedia phones, functional phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as iPads.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players (eg iPod), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Other electronic devices with data interaction function.

So far, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain embodiments, multitasking and parallel processing may be advantageous.

In the 1990s, improvements in a technology could be clearly differentiated between improvements in hardware (eg, improvements to circuit structures such as diodes, transistors, switches, etc.) or improvements in software (improvements in method flow). However, with the development of technology, the improvement of many methods and processes today can be regarded as a direct improvement of the hardware circuit structure. Designers almost get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware entity modules. For example, a Programmable Logic Device (PLD) (eg, Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. It is programmed by the designer to "integrate" a digital system on a PLD without having to ask the chip manufacturer to design and manufacture a dedicated integrated circuit chip. And, instead of making integrated circuit chips by hand, these days, much of this programming is done using software called a "logic compiler", which is similar to the software compiler used in program development and writing, but before compiling The original code also has to be written in a specific programming language, which is called Hardware Description Language (HDL), and there is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language) , AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (RubyHardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be clear to those skilled in the art that a hardware circuit for implementing the logic method process can be easily obtained by simply programming the method process in the above-mentioned several hardware description languages and programming it into the integrated circuit.

The controller may be implemented in any suitable manner, for example, the controller may take the form of eg a microprocessor or processor and a computer readable medium storing computer readable program code (eg software or firmware) executable by the (micro)processor , logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers, examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicon Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in the form of pure computer-readable program code, the controller can be implemented as logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded devices by logically programming the method steps. The same function can be realized in the form of a microcontroller, etc. Therefore, such a controller can be regarded as a hardware component, and the devices included therein for realizing various functions can also be regarded as a structure within the hardware component. Or even, the means for implementing various functions can be regarded as both a software module implementing a method and a structure within a hardware component.

The systems, devices, modules or units described in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or A combination of any of these devices.

For the convenience of description, when describing the above device, the functions are divided into various units and described respectively. Of course, when implementing the present application, the functions of each unit may be implemented in one or more software and/or hardware.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture, or device that includes the element.

It will be appreciated by those skilled in the art that the embodiments of the present application may be provided as a method, a system or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular transactions or implement particular abstract data types. The application may also be practiced in distributed computing environments where transactions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for related parts, please refer to the partial descriptions of the method embodiments.

The above descriptions are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (11)

1. A method for mouth shape evaluation, comprising:

acquiring data to be evaluated of a target object, wherein the data to be evaluated comprises an image frame sequence, and the image frame sequence comprises at least one continuous image frame representing a target pronunciation mouth shape;

determining a feature matrix of the image frame sequence from the image frame sequence;

obtaining first evaluation data according to a feature matrix of the image frame sequence and a preset model, wherein the first evaluation data is used for indicating a result of evaluation of the feature matrix according to the preset model;

performing mouth shape evaluation on the image frame sequence to obtain second evaluation data, wherein the second evaluation data is used for representing the similarity between a target pronunciation mouth shape in the image frame and the preset standard pronunciation mouth shape;

evaluating audio data according to a preset reference audio to generate third evaluation data, wherein the third evaluation data is used for representing the pronunciation similarity of the audio data and the reference audio;

generating mouth shape evaluation information of the target object according to evaluation data, wherein the evaluation data comprise the first evaluation data, and the mouth shape evaluation information is used for expressing an evaluation result of a target pronunciation mouth shape of the target object;

generating mouth shape evaluation information of the target object according to evaluation data, wherein the mouth shape evaluation information comprises the following steps:

and performing weighted operation on the first evaluation data and the second evaluation data to generate the mouth shape evaluation information, or performing weighted operation on the first evaluation data and the third evaluation data to generate the mouth shape evaluation information, or performing weighted operation on the first evaluation data, the second evaluation data and the third evaluation data to generate the mouth shape evaluation information.

2. The method of claim 1, wherein determining a feature matrix for the sequence of image frames from the sequence of image frames comprises:

determining coordinates of at least one mouth key point in image frames contained in the image frame sequence;

and generating a feature matrix of the image frame sequence according to the number of the image frames in the image frame sequence, the number of the mouth key points in each image frame and the dimension of the coordinates in a time sequence, wherein the feature matrix of the image frame sequence comprises the coordinates of the at least one mouth key point.

3. The method of claim 2, wherein the preset model comprises a standard pronunciation curve model; the obtaining of the first evaluation data according to the feature matrix of the image frame sequence and a preset model comprises:

inputting the coordinates of the at least one mouth key point into the standard pronunciation curve model to obtain the score of the at least one mouth key point;

obtaining a mouth shape score of the image frame according to the score of the at least one mouth key point;

and normalizing the mouth shape score of at least one image frame and generating the first evaluation data.

4. The method according to claim 1, wherein the preset model includes a preset transpose matrix, and the obtaining of the first evaluation data according to the feature matrix of the image frame sequence and the preset model includes:

performing dot multiplication on the feature matrix of the image frame sequence and a preset transposed matrix to obtain a pronunciation matrix;

and comparing the pronunciation matrix with a preset standard pronunciation matrix to obtain the first evaluation data.

5. The method of claim 1, further comprising:

performing mouth shape recognition on at least one image frame in the image frame sequence;

and determining the similarity between the target pronunciation mouth shape and a preset standard pronunciation mouth shape according to the recognition result, and generating second evaluation data.

6. The method according to claim 5, wherein before determining the similarity between the target pronunciation mouth shape and the preset standard pronunciation mouth shape according to the recognition result, the method further comprises:

aligning the image frame sequence with the frame number of a reference frame sequence, wherein the reference frame sequence is a preset frame sequence representing a standard pronunciation mouth shape.

7. The method of claim 6, wherein aligning the sequence of image frames with the number of frames of a sequence of reference frames comprises:

down-sampling or linearly interpolating the sequence of image frames to align the sequence of image frames with the number of frames of a sequence of reference frames.

8. The method according to claim 1, wherein evaluating the audio data according to a predetermined reference audio to generate third evaluation data comprises:

extending the initial time of the audio data forwards for a first preset time length, and extending the initial time of the audio data backwards for a second preset time length to obtain extended audio data;

and evaluating the extended audio data according to a preset reference audio to generate third evaluation data.

9. An oral evaluation device, comprising: the system comprises an acquisition module, a matrix module, an evaluation module and a comprehensive module;

the system comprises a fetching module, a judging module and a judging module, wherein the fetching module is used for acquiring data to be evaluated of a target object, the data to be evaluated comprises an image frame sequence, and the image frame sequence comprises at least one continuous image frame representing a target pronunciation mouth shape;

the matrix module is used for determining a characteristic matrix of the image frame sequence according to the image frame sequence;

the evaluation module is used for obtaining first evaluation data according to a feature matrix of the image frame sequence and a preset model, and the first evaluation data is used for indicating a result of evaluating the feature matrix according to the preset model;

the evaluating module is further configured to evaluate the mouth shape of the image frame sequence to obtain second evaluating data, where the second evaluating data is used to represent the similarity between a target pronunciation mouth shape in the image frame and the preset standard pronunciation mouth shape, and evaluate audio data according to a preset reference audio to generate third evaluating data, and the third evaluating data is used to represent the pronunciation similarity between the audio data and the reference audio;

the comprehensive module is used for generating mouth shape evaluation information of the target object according to evaluation data, the evaluation data comprise the first evaluation data, and the mouth shape evaluation information is used for representing an evaluation result of a target pronunciation mouth shape of the target object;

the comprehensive module is further configured to perform a weighted operation on the first evaluation data and the second evaluation data to generate the mouth shape evaluation information, or perform a weighted operation on the first evaluation data and the third evaluation data to generate the mouth shape evaluation information, or perform a weighted operation on the first evaluation data, the second evaluation data, and the third evaluation data to generate the mouth shape evaluation information.

10. An electronic device, comprising: a processor and a memory, said processor being connected to said memory, said memory storing a computer program, said processor being adapted to execute said computer program to implement the method for mouth shape assessment according to any of claims 1-7.

11. A computer storage medium, comprising: the computer storage medium stores a computer program which, when executed by a processor, implements a method of mouth shape evaluation according to any one of claims 1-7.

Images