CN100508587C - A News Video Retrieval Method Based on Speech Classification Recognition - Google Patents

Abstract

This invention relates to a news video search method based on phone sort identification, which divides all phone fragments of standard phones automatically in news video then identifies the standard phones by a phone identification system, since the standard phone can express the main content of the video, it's easy to realize the news searches from the context to the video.

Description

A News Video Retrieval Method Based on Speech Classification Recognition

technical field

The invention belongs to the technical field of computer voice recognition and video retrieval, and in particular relates to a news video retrieval method based on voice classification recognition.

Background technique

At present, speech recognition technology has a wide range of applications, not only in the field of audio, but also in the field of video, because the video also contains audio information. If the speech content in the video can be recognized through speech recognition technology, it can provide powerful support for video retrieval and realize the retrieval from speech text to video content. Existing video retrieval technologies generally extract low-level features such as color and texture from videos, and then perform video retrieval based on these features. However, this method brings the following two problems: (1) When people retrieve videos, they retrieve them based on human high-level semantic features such as football matches, Iraq war, bird flu, etc., which are different from the underlying features of videos described by computers, such as Features such as color and texture have great contradictions, and the two cannot be consistent; (2) The existing video retrieval methods cannot realize the retrieval from text to video well, and the query method is not suitable for people's usual methods, and the application is very inappropriate. convenient. The existing video retrieval method is: generally, the user submits a query shot or query segment to the system, and then the system returns a result similar to the query example. However, at the same time, the problem is: how does the user get the query example? In addition, the query method that most users are accustomed to is to enter query text, and then the system returns video materials related to the query text. For example, the user enters the query text "Iraq War" and hopes that the system can return video materials related to "Iraq War". Similar to current search engines such as Google and Baidu, but different from these search engines, the input is text, but the result of retrieval is video data.

In order to realize the above retrieval from speech text to video content, it is necessary to obtain the text information in the video, and in order to obtain the text information in the video, a feasible method is to use speech recognition technology to recognize the speech text in the video. However, in the existing speech recognition system, in order to recognize the speech of different people, it is often necessary to train the speech recognition system by the speaker first, and then the speech recognition system recognizes the speech of the speaker. This method is difficult to apply to speech clips that include many people, because it is difficult to find everyone to train the speech recognition system. Even for a few people's speech clips, it is often impossible to find the speaker for speech training, such as for news videos. For speech recognition, it is impossible to find every speaker for speech training; in addition, even after speech training, it is still difficult to recognize non-standard speech, and the recognition rate is very low. However, if the speech recognition system is directly used for speech recognition of the news video without speech training, the recognition effect will be worse and the recognition rate will be lower, because the news programs of the video usually include the following various sounds: (1) with music Background news program preview; (2) advertisement; (3) weather forecast; (4) non-standard voice, such as the dialect of the person being interviewed; (5) standard voice. Among the above-mentioned speeches, the recognition rate of non-standard speeches is very low, and the recognition rates of (1)-(3) are even lower, and basically cannot be recognized. Therefore, if the speech recognition system is directly used to perform speech recognition on the entire news video indiscriminately, the result is: the speech recognition system recognizes all kinds of sounds contained in the news video, and finally leads to the result of speech recognition. Including correct recognition results (mainly the recognition of the standard speech in the above 5) and wrong recognition results (mainly the recognition of other speeches in the above 1 to 4), and the computer cannot know which are the correct results and which are is a wrong result. Therefore, when searching for videos based on this, if you search for videos corresponding to the text "Iraq War", many wrong results will appear.

Contents of the invention

For the deficiencies in the prior art, the purpose of this invention is to propose a kind of news video retrieval method based on voice classification and recognition, which can realize automatic recognition of standard voices such as standard mandarin in the news video without speaker training, thereby realizing News retrieval from text to video.

In order to achieve the above object, the technical solution adopted in the present invention is: a kind of news video retrieval method based on voice classification recognition, comprising the following steps:

(1) utilize sound classifier, segment out the speech segment of standard speech in the news video, described standard speech refers to the speech of pronunciation standard;

(2) Use the speech recognition system to recognize the speech fragments of the standard speech in the news video and convert them into text content;

(3) According to the text content obtained in step (2), carry out the retrieval of the corresponding video material, and realize the retrieval from the voice text to the news video.

Further, the standard voice described in the present invention is preferably Mandarin with standard pronunciation.

Further, in step (1), audio classification adopts a classification model based on a support vector machine, which is divided into two parts: classifier model training and classification prediction; audio features extracted during audio classification are logarithmic energy (log energy) A 13-dimensional feature vector composed of Mel-scale Frequency Cepstral Coefficients (MFCC for short).

Further, in order to make the present invention have better effect, in step (1), the process of classifier model training is: first select training sample, then extract the audio frequency that the logarithmic energy of each sample and Mel's cepstral coefficient form features, and write all these features into a feature file, and then use the support vector machine to generate a classifier model. The training samples include the following five categories: 1) standard speech; 2) music; 3) background noise; 4) silent; The category labeling of is also based on the frame unit, and the model training is performed using the labeled category.

Still further, in order to make the present invention have better effect, in the step (1), the process of classification prediction is: for the news video that will be classified, extract the audio feature that the logarithmic energy of news audio and Mel's cepstral coefficient form , and then use the classifier model trained by the support vector machine for automatic classification and labeling.

Still further, in order to make the present invention have better effect, in step (1), the speech clip of the standard speech in the news video that preliminarily divides out is carried out correction processing: When one or M frames of different categories appear independently, and M is a positive integer, these frames are judged as misrecognized frames, and these isolated frames are corrected as consecutive frames of the same category. This is because in consecutive frames of the same category, it is impossible to have very few frames of other categories sporadically, so these frames can be judged as misrecognized frames at this time, and these isolated frames can be corrected as consecutive frames of the same category.

Still further, in order to make the present invention have a better effect: when the speech fragments of standard Mandarin in the initially segmented news video are corrected, in practical applications, select M to be less than or equal to 10, that is, if a section of continuous same category If there are less than or equal to 10 frames of different categories in the middle of the audio, it is judged that these frames are misrecognized frames.

The effect of the present invention is that: compared with the existing method, the present invention can automatically recognize the standard speech in the news video without speaker training, thereby obtaining the most important text content reflecting the news video, and then realizing it through the text content News retrieval from text to video, so as to give full play to the great role of audio analysis and retrieval technology in information retrieval.

Why the present invention has above-mentioned invention effect, its reason is in:

The standard pronunciation of the announcer in the news video can reflect the main content of the news video; and the existing speech recognition system can better recognize the standard voice without speaker training. Therefore, the present invention is first used in the news video. , automatically segment all the speech segments of the standard speech; and then use the speech recognition system to recognize the segmented standard speech to obtain the most important text content reflecting the news video, thereby realizing news retrieval from text to video.

Description of drawings

Fig. 1 is a schematic flow chart of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, a news video retrieval method based on speech classification and recognition includes the following steps:

(1) Utilize sound classifier, segment out the speech segment of standard speech in the news video, the standard speech in the present embodiment is illustrated with standard mandarin as example;

Audio classification uses a classification model based on support vector machines, which is divided into two parts: classifier model training and classification prediction. The audio feature uses a 13-dimensional feature vector composed of log energy (log energy) and Mel cepstral coefficient (MFCC).

In this embodiment, the process of classifier model training is: first select training samples, then extract the audio features formed by the logarithmic energy and Mel cepstral coefficients of each sample, and write all these features in a feature file, A classifier model is then generated using a support vector machine. The training samples include the following five categories: 1) Standard Mandarin; 2) Music; 3) Background noise; 4) Silence; 5) Non-standard Mandarin. Classification is in units of frames, assigning a corresponding category to each audio frame, and the category labeling of training samples is also in units of frames, but because the time length of each audio frame is 23 milliseconds (sampling frequency 22050 Hz, sound frame contains 512 sampling points), it is impossible to perform accurate manual category labeling in this length of time, so the present invention selects a section of audio and judges it with the human ear. The content in this audio belongs to a category, and it is judged by the human ear. If other obvious categories appear, assign all frames in this audio to a category, and use the labeled category for model training.

In this embodiment, the process of classification prediction is: for the news video to be classified, extract the audio feature formed by the logarithmic energy of the news audio and the Mel cepstral coefficient, and then use the classifier model trained by the support vector machine to perform automatic Category labeling.

In this embodiment, the process of correcting the classification results is: correcting the speech segments of standard Mandarin in the initially segmented news video, because the classification label is based on frames, and the length of each frame is 23 milliseconds. There will be such a situation in the classification results: consecutive frames of the same category, there will be one independent or M (M is a positive integer) frames of different categories, due to continuous frames of the same category, it is impossible to sporadically appear a very small number of other categories frames, so in this embodiment, these frames are judged as misrecognized frames, and these isolated frames are corrected as consecutive frames of the same type. In practical applications, M can be selected to be less than or equal to 10, that is, if there are less than or equal to 10 frames of different categories in a continuous piece of audio of the same category, it can be judged that these frames are misrecognized frames.

(2) Use the speech recognition system to recognize the speech fragments of standard Mandarin in the news video and convert them into text content;

Utilize existing speech recognition software to carry out recognition, and the process of recognition is to input the standard mandarin speech that the present invention divides, and then speech recognition software carries out recognition.

(3) According to the text content obtained in step (2), carry out the retrieval of the corresponding video material, and realize the retrieval from the voice text to the news video.

The following experimental results show that, compared with the existing methods, the present invention can not only recognize the speech of the news video without speaker training, but also can obtain the most important text content reflecting the news video, thus providing a new way for text-to-video news Retrieval provides strong support, which proves that the present invention has achieved good results in news video retrieval based on speech classification recognition.

In the present embodiment, a news program of 1 hour, 4 minutes and 6 seconds has been adopted, which mainly includes the following sounds: 1) news program preview with music background; 2) advertisement; 3) weather forecast; 4) non-standard Mandarin, such as The dialect of the person being interviewed; 5) Standard Mandarin. Wherein, the recognition rate of 1)-4) is very low, take this news program as example, use existing speech recognition software, only have 6.2% to the correct recognition rate of 4) non-standard mandarin, and the recognition of 1)-3) The rate is lower and basically unrecognizable.

The object of the present invention is to segment the standard mandarin segment from the news video, and then recognize the standard mandarin part by voice recognition software to obtain the main text content of the news video.

1. Segmentation results of standard Mandarin

The present embodiment has used the following 2 standard evaluations that the present invention divides out the result of the standard mandarin in the news video:

Accuracy rate=the standard Mandarin result of the correct standard mandarin of the present invention division/the present invention division

Recall = all standard Mandarin included in the correct standard Mandarin/news video segmented by the present invention

Wherein, all the standard mandarin included in the news video are obtained by people manually listening to the news video, while the standard mandarin segmented by the present invention is automatically realized by a computer. Finally, in this news video, the precision rate=99.05% and the recall rate=96.69% of the standard mandarin segmented by the present invention have achieved very good results.

2. Recognition results of standard Mandarin

Using the existing speech recognition software to recognize the standard Mandarin segmented by the present invention, the correct recognition rate can reach 61.42%. If the present invention is not used to identify the whole news program, then the correct recognition rate is only 47.6%. The result that this brings is: because of the sound other than standard Mandarin, the recognition rate is extremely low. If these parts are not removed, the final The recognition results include correct recognition results and wrong recognition results, and the computer cannot know which are correct results and which are wrong results. When doing video retrieval in this way, such as querying "Iraq War" to display the corresponding video, many wrong results will appear. But, using the present invention, because the standard mandarin is basically segmented, and the standard mandarin includes the part broadcast by the announcer, this content can basically reflect the main content of the whole news video, the present invention only recognizes the standard mandarin. In this way, the present invention can obtain the main text content reflecting the news video, and the retrieval result can be greatly improved.

The method described in the present invention is not limited to the embodiments described in the specific implementation mode. For example, the standard pronunciation can be based on the same principle as long as there is a standard pronunciation language in any other country or region except my country's standard Mandarin. and methods to achieve news retrieval from text to video. Other implementations obtained by those skilled in the art according to the technical solution of the present invention also belong to the technical innovation scope of the present invention.

Claims (8)

1. A news video retrieval method based on speech classification and recognition, comprising the following steps: (1) utilize sound classifier, segment out the speech segment of standard speech in the news video, described standard speech refers to the speech of pronunciation standard; (2) Use the speech recognition system to recognize the speech fragments of the standard speech in the news video and convert them into text content; (3) According to the text content obtained in step (2), carry out the retrieval of the corresponding video material, and realize the retrieval from the voice text to the news video. 2. A news video retrieval method based on speech classification and recognition as claimed in claim 1, wherein the standard speech is Mandarin with standard pronunciation. 3. A kind of news video retrieval method based on speech classification and recognition as claimed in claim 1, characterized in that: in the step (1), when using a voice classifier to segment out the speech segment of the standard speech in the news video, the audio classification adopts The classification model based on support vector machine is divided into two parts: classifier model training and classification prediction; audio features extracted during audio classification use 13-dimensional feature vectors composed of logarithmic energy and Mel cepstral coefficients. 4. A kind of news video retrieval method based on speech classification recognition as claimed in claim 3, it is characterized in that: in step (1), the process of classifier model training is: at first select training samples, then extract each sample Audio features composed of logarithmic energy and Mel cepstral coefficients, and write all these features into a feature file, and then use the support vector machine to generate a classifier model. The training samples include the following five categories: 1) standard speech; 2) Music; 3) background noise; 4) silence; 5) non-standard speech; classification is in units of frames, assigning a corresponding category to each audio frame, and the category labeling of training samples is also in units of frames, using the marked category Do model training. 5. A kind of news video retrieval method based on speech classification and recognition as claimed in claim 3 or 4, characterized in that: in step (1), when classifying and predicting, for the news video to be classified, the pair of news audio is extracted The audio features composed of numerical energy and Mel cepstral coefficients are used, and then the classifier model trained by the support vector machine is used for automatic classification and labeling. 6. A kind of news video retrieval method based on speech classification and recognition as claimed in claim 1, 3 or 4, characterized in that: in step (1), the speech segment of the standard speech in the initially segmented news video is corrected Processing, that is, in the classification results, if one or M frames of different categories suddenly appear in consecutive frames of the same category, and M is a positive integer, these frames are judged as misrecognized frames, and these isolated frames are corrected for consecutive frames of the same category. 7. A kind of news video retrieval method based on speech classification and recognition as claimed in claim 5, characterized in that: in the step (1), the speech segment of the standard speech in the initially segmented news video is corrected, that is, in In the classification results, if one or M frames of different categories suddenly appear in consecutive frames of the same category, and M is a positive integer, these frames are judged as misrecognized frames, and these isolated frames are corrected as consecutive frames of the same category frame; the above-mentioned standard voice is Mandarin with standard pronunciation. 8. A news video retrieval method based on speech classification and recognition as claimed in claim 7, characterized in that: when the speech segment of the standard speech in the initially segmented news video is corrected, M is less than or equal to 10, that is If there are less than or equal to 10 frames of different categories in a continuous piece of audio of the same category, it is determined that these frames are incorrectly recognized frames.

Images