TWI731921B - Speech recognition method and device - Google Patents

Abstract

This application discloses a voice recognition method, which includes: using a preset voice knowledge source to generate a search space containing user-side preset information for decoding a voice signal; extracting a feature vector sequence of the voice signal to be recognized; and calculating; The feature vector corresponds to the probability of the basic unit of the search space; using the probability as an input, a decoding operation is performed in the search space to obtain a word sequence corresponding to the feature vector sequence. This application also provides a voice recognition device, and another voice recognition method and device. With the method provided in this application, since the user-side preset information is included when generating the search space for decoding, the information related to the client-side can be relatively accurately identified when recognizing the voice signal collected by the user-side. Thereby, the accuracy of speech recognition can be improved, and the user experience can be improved.

Description

Speech recognition method and device

This application relates to voice recognition technology, and specifically relates to a voice recognition method and device. This application also relates to another voice recognition method and device.

Voice is the acoustic expression of language, the most natural, effective, and convenient way for humans to communicate information, and it is also a support for human thinking. Automatic speech recognition (Automatic Speech Recognition-ASR) usually refers to the process of allowing computers and other devices to recognize and understand speech to convert human spoken language into corresponding output text or commands. The core framework is: on the basis of using statistical model modeling, according to the feature sequence O extracted from the voice signal to be recognized, the following Bayesian decision criterion is used to solve the optimal word sequence W corresponding to the voice signal to be recognized *: W*=argmaxP(O|W)P(W)

In specific implementation, the above process of finding the optimal word sequence is called the decoding process (the module that realizes the decoding function is usually called a decoder), that is: searching in a search space composed of multiple knowledge sources such as pronunciation dictionaries and language models Find the best word sequence shown in the above formula.

With the development of technology, the computing power and storage capacity of hardware have made great progress. Voice recognition systems have gradually been applied in the industry. Various applications that use voice as a human-computer interaction medium have also appeared on user-end devices. For example, for a call application on a smart phone, the user only needs to give a voice instruction (such as: "Call Zhang San") to automatically realize the phone call function.

Current voice recognition applications usually use two modes, one is based on the client and the server, that is: the client collects voice, uploads it to the server via the network, and the server recognizes the voice as text through decoding, and then sends it back To the user side. The reason for adopting this mode is that the computing power of the client is relatively weak, and its memory space is relatively limited, and the server has obvious advantages in both aspects; but in this mode, if there is no network access Environment, the user terminal cannot complete the voice recognition function. In response to the above problems, a second voice recognition application mode that only relies on the client has emerged. In this mode, the model and search space originally stored on the server are placed locally on the client device by reducing the scale. Complete the operation of collecting voice and decoding.

In practical applications, whether it is the first mode or the second mode, when the above-mentioned general framework is used for speech recognition, it is usually unable to effectively recognize the content related to the local information of the user equipment in the speech signal, such as: The name of the contact person, which leads to low recognition accuracy, brings inconvenience to the user's use, and affects the user's experience.

The embodiments of the present application provide a voice recognition method and device to solve the problem of low recognition accuracy of the existing voice recognition technology on the local related information of the user terminal. The embodiment of the present application also provides another voice recognition method and device.

This application provides a voice recognition method, including: using a preset voice knowledge source to generate a search space containing user-side preset information for decoding a voice signal; extracting a feature vector sequence of the voice signal to be recognized; and calculating features The vector corresponds to the probability of the basic unit of the search space; using the probability as an input, a decoding operation is performed in the search space to obtain a word sequence corresponding to the feature vector sequence.

Optionally, the search space includes: a weighted finite state converter.

Optionally, the basic unit of the search space includes: context-related triphones; the preset knowledge source includes: a pronunciation dictionary, a language model, and a triphone state binding field table.

Optionally, the generation of the search space for decoding the speech signal containing the user-side preset information by using the preset speech knowledge source includes: replacing tags to a pre-generated at least language model-based search space The weighted finite state converter adds user-side preset information corresponding to the preset theme category, and obtains a single weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model; wherein, the language The model is obtained by pre-training in the following manner: replacing the preset named entities in the text used for training the language model with tags corresponding to the preset subject category, and using the text to train the language model.

Optionally, the method of replacing tags is to add user-side preset information corresponding to the preset theme category to the pre-generated weighted finite state converter based at least on the language model, and obtain a binding column based on the triphone state A single weighted finite state converter for bit tables, pronunciation dictionaries, and language models, including: adding a user-side preset corresponding to the preset theme category to the pre-generated weighted finite state converter based on the language model by replacing tags Information; combining the weighted finite state converter with user-side preset information and the pre-generated weighted finite state converter based on the triphone state binding field table and pronunciation dictionary to obtain the single weighted finite State converter.

Optionally, the text used for training the language model refers to text for the preset topic category.

Optionally, the number of the preset topic categories is at least two; the number of the language models and the number of the weighted finite state machines based on at least the language models are respectively consistent with the number of the preset topic categories; By replacing tags, adding user-side preset information corresponding to the preset theme category to the pre-generated weighted finite state converter based at least on the language model, including: determining the preset theme category to which the voice signal to be recognized belongs; select The weighted finite state converter based at least on the language model that is generated in advance and corresponds to the preset topic category; by replacing the corresponding label with the user-side preset information corresponding to the preset topic category, Add client default information to the selected weighted finite state converter.

Optionally, the determination of the preset theme category to which the voice signal to be recognized belongs is implemented in the following manner: the preset theme category to which the voice signal is collected is determined according to the type of the user terminal or the application program that collects the voice signal.

Optionally, the preset theme category includes: making a call or sending a text message, playing music, or setting instructions; the corresponding client preset information includes: the name of the contact in the address book, the name of the music in the music library, Or, the instructions in the instruction set.

Optionally, the merging operation includes: merging using a prediction-based method.

Optionally, the vocabulary used for pre-training the language model is consistent with the words contained in the pronunciation dictionary.

Optionally, the calculating the probability that the feature vector corresponds to the basic unit of the search space includes: using a pre-trained DNN model to calculate the probability that the feature vector corresponds to each triphone state; according to the feature vector corresponding to each triphone state Probability, using the pre-trained HMM model to calculate the probability that the feature vector corresponds to each triphone.

Optionally, the execution speed of the step of using the pre-trained DNN model to calculate the probability of the feature vector corresponding to each triphone state is improved by the following method: using the data parallel processing capability provided by the hardware platform.

Optionally, the extracting the feature vector sequence of the voice signal to be recognized includes: performing framing processing on the voice signal to be recognized according to a preset frame length to obtain multiple audio frames; extracting the feature vector of each audio frame to obtain the Feature vector sequence.

Optionally, the extracting the feature vector of each audio frame includes: extracting the MFCC feature, the PLP feature, or the LPC feature.

Optionally, after the word sequence corresponding to the feature vector sequence is obtained, the following operation is performed: verify the accuracy of the word sequence through text matching with preset information on the client side, and according to the verification result Generate the corresponding speech recognition results.

Optionally, the verification of the accuracy of the word sequence through text matching with preset information on the client side, and obtaining the corresponding voice recognition result according to the verification result, includes: selecting from the word sequence corresponding to all The word to be verified in the preset information on the client side; search for the word to be verified in the preset information on the client side; if it is found, it is determined to pass the accuracy verification, and the word sequence is used as the voice recognition result; Otherwise, the word sequence is corrected through the fuzzy matching method based on pinyin, and the corrected word sequence is used as the voice recognition result.

Optionally, the modification of the word sequence through a fuzzy matching method based on pinyin includes: converting the word to be verified into a pinyin sequence to be verified; and converting each word in the preset information on the user side into a comparison. Pair the pinyin sequence; sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and select the top words sorted from high to low from the user-side preset information; Replace the word to be verified in the word sequence with the selected word to obtain the revised word sequence.

Optionally, the similarity includes: a similarity calculated based on the edit distance.

Optionally, the method is implemented on a user-end device; the user-end device includes: a smart mobile terminal, a smart speaker, or a robot.

Correspondingly, the present application also provides a voice recognition device, including: a search space generating unit, configured to use a preset voice knowledge source to generate a search space containing user-side preset information for decoding voice signals; features; The vector extraction unit is used to extract the feature vector sequence of the speech signal to be recognized; the probability calculation unit is used to calculate the probability that the feature vector corresponds to the basic unit of the search space; the decoding search unit is used to use the probability as the input and the A decoding operation is performed in the search space to obtain a word sequence corresponding to the feature vector sequence.

Optionally, the search space generating unit is specifically configured to add user-side preset information corresponding to the preset topic category to the weighted finite state converter generated in advance at least based on the language model by replacing tags, and Obtain a single weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model; the language model is pre-generated by the language model training unit, and the language model training unit is used to The preset named entities in the text for training the language model are replaced with tags corresponding to the preset topic categories, and the text is used to train the language model.

Optionally, the search space generating unit includes: a first user-side information adding subunit, which is used to add a language model-based weighted finite state converter corresponding to a preset topic category to a pre-generated language model-based weighted finite state converter by replacing tags The user-side preset information; the weighted finite state converter merging subunit is used to combine the weighted finite state converter with the user-side preset information and the pre-generated triphone-based state binding field table Combined with the weighted finite state converter of the pronunciation dictionary, the single weighted finite state converter is obtained.

Optionally, the decoding space generating unit includes: a second user-side information adding subunit, which is used to add and preset topic categories to the pre-generated weighted finite state converter at least based on the language model by replacing tags Corresponding user-side preset information; the unified weighted finite state converter acquisition subunit is used to obtain the binding field table, pronunciation dictionary, pronunciation dictionary based on triphone state after the second user-side information adding subunit completes the adding operation And a single weighted finite state converter of the language model; wherein the second user-side information adding subunit includes: a topic determination subunit for determining the preset topic category to which the speech signal to be recognized belongs; weighted finite state converter selection Sub-unit for selecting the weighted finite state converter based on the language model that is generated in advance and corresponding to the preset topic category; the label replacement sub-unit is used to correspond to the preset topic category by using The method of replacing the corresponding label with the default information on the client side to add the default information on the client side to the selected weighted finite state converter.

Optionally, the theme determining subunit is specifically configured to determine the preset theme category to which the voice signal belongs according to the type of the user terminal or the application program that collects the voice signal.

Optionally, the weighted finite state converter merging subunit is specifically configured to perform a merging operation using a prediction-based method, and obtain the single weighted finite state converter.

Optionally, the probability calculation unit includes: a triphone state probability calculation subunit for calculating the probability that a feature vector corresponds to each triphone state by using a pre-trained DNN model; a triphone probability calculation subunit for calculating the probability of each triphone state according to the feature The vector corresponds to the probability of each triphone state, and a pre-trained HMM model is used to calculate the probability that the feature vector corresponds to each triphone.

Optionally, the feature vector extraction unit includes: a framing subunit for performing framing processing on the speech signal to be recognized according to a preset frame length to obtain multiple audio frames; a feature extraction subunit for extracting each audio signal The feature vector of the frame to obtain the feature vector sequence.

Optionally, the device includes: an accuracy verification unit configured to verify the word sequence through text matching with the user-side preset information after the decoding search unit obtains the word sequence corresponding to the feature vector sequence According to the accuracy of the verification result, the corresponding voice recognition result is generated.

Optionally, the accuracy verification unit includes: a word-to-be-verified selection sub-unit for selecting a word to be verified from the word sequence corresponding to the user-side preset information; a search sub-unit for selecting a word to be verified in the word sequence; Search for the word to be verified in the preset information on the client side; a recognition result confirmation sub-unit for determining that the accuracy verification is passed after the search sub-unit finds the word to be verified, and to compare the word sequence As the result of speech recognition; the recognition result correction subunit is used to correct the word sequence through the fuzzy matching method based on pinyin after the search subunit does not find the word to be verified, and use the corrected word sequence as the voice Identification result.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, used to convert the word to be verified into a pinyin sequence to be verified; a comparison pinyin sequence conversion subunit, used to convert the Each word in the preset information on the client side is converted into a compared pinyin sequence; the similarity calculation and selection subunit is used to sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and from the The user-side preset information selects the top words sorted from high to low according to the similarity; the word to be verified replacement subunit is used to replace the word to be verified in the word sequence with the selected word to obtain the correction The sequence of words after.

In addition, this application also provides another voice recognition method, including: obtaining a word sequence corresponding to the voice signal to be recognized through decoding; verifying the accuracy of the word sequence by text matching with preset information on the client side, and according to the verification result Generate the corresponding speech recognition results.

Optionally, the verification of the accuracy of the word sequence through text matching with preset information on the client side, and generating a corresponding voice recognition result according to the verification result includes: selecting from the word sequence corresponding to the user The word to be verified in the preset information of the client; search for the word to be verified in the preset information of the client; if found, it is determined to pass the accuracy verification, and the word sequence is used as the voice recognition result; otherwise, it passes The word sequence is corrected based on the fuzzy matching method of pinyin, and the corrected word sequence is used as the voice recognition result.

Optionally, the modification of the word sequence through a fuzzy matching method based on pinyin includes: converting the word to be verified into a pinyin sequence to be verified; and converting each word in the preset information on the user side into a comparison. Pair the pinyin sequence; sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and select the top words sorted from high to low from the user-side preset information; Replace the word to be verified in the word sequence with the selected word to obtain the revised word sequence.

Correspondingly, the present application also provides another voice recognition device, including: a word sequence acquisition unit for acquiring a word sequence corresponding to the voice signal to be recognized through decoding; a word sequence verification unit for performing processing with user-end preset information Text matching verifies the accuracy of the word sequence, and generates a corresponding voice recognition result according to the verification result.

Optionally, the word sequence verification unit includes: a word-to-be-verified selection sub-unit for selecting a word to be verified from the word sequence corresponding to the user-side preset information; a search sub-unit for selecting the word to be verified in the word sequence; Search for the word to be verified in the preset information on the client side; a recognition result confirmation sub-unit for determining that the accuracy verification is passed after the search sub-unit finds the word to be verified, and to compare the word sequence As the result of speech recognition; the recognition result correction subunit is used to correct the word sequence through the fuzzy matching method based on pinyin after the search subunit does not find the word to be verified, and use the corrected word sequence as the voice Identification result.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, used to convert the word to be verified into a pinyin sequence to be verified; a comparison pinyin sequence conversion subunit, used to convert the Each word in the preset information on the client side is converted into a compared pinyin sequence; the similarity calculation and selection subunit is used to sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and from the The user-side preset information selects the top words sorted from high to low according to the similarity; the word to be verified replacement subunit is used to replace the word to be verified in the word sequence with the selected word to obtain the correction The sequence of words after.

Compared with the prior art, this application has the following advantages: the speech recognition method provided by this application uses a preset speech knowledge source to generate a search space containing user-side preset information for decoding speech signals. Calculate the probability that the feature vector extracted from the voice signal to be recognized corresponds to the basic unit of the search space, and perform a decoding operation in the search space according to the probability, so as to obtain the word sequence corresponding to the voice signal to be recognized. The above-mentioned method provided by the present application includes the user-side preset information when generating the search space for decoding, and therefore can relatively accurately identify the information related to the client-side when recognizing the voice signal collected by the user-side. Thereby, the accuracy of speech recognition can be improved, and the user experience can be improved.

901‧‧‧Search Space Generating Unit

902‧‧‧Feature Vector Extraction Unit

903‧‧‧Probability calculation unit

904‧‧‧Decoding Search Unit

1101‧‧‧Word Sequence Acquisition Unit

1102‧‧‧Word Sequence Verification Unit

FIG. 1 is a flowchart of an embodiment of a voice recognition method of the present application; FIG. 2 is a process flowchart of generating a search space containing user-side preset information for decoding a voice signal according to an embodiment of the present application; Fig. 3 is a schematic diagram of the G structure WFST before the replacement operation provided by an embodiment of the application; Fig. 4 is a schematic diagram of the G structure WFST after the replacement operation provided by an embodiment of the application; Fig. 5 is an extraction provided by the embodiment of the application The processing flowchart of the feature vector sequence of the voice signal to be recognized; FIG. 6 is a processing flowchart for calculating the probability of a feature vector corresponding to each triphone provided by an embodiment of the present application; FIG. 7 is a verification through text matching provided by an embodiment of the present application The accuracy of the word sequence and the processing flow chart of generating the corresponding voice recognition result according to the verification result; FIG. 8 is the overall framework diagram of the voice recognition provided by the embodiment of the application; FIG. 9 is the embodiment of the voice recognition device of the present application Schematic diagram; Figure 10 is a flowchart of another embodiment of the voice recognition method of the present application; Figure 11 is a schematic diagram of another embodiment of the voice recognition device of the present application.

In the following description, many specific details are set forth in order to fully understand this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar promotion without violating the connotation of this application. Therefore, this application is not limited by the specific implementation disclosed below.

In this application, a voice recognition method and corresponding device are respectively provided, and another voice recognition method and corresponding device are respectively provided, which are described in detail in the following embodiments. In order to facilitate understanding, before describing the embodiments, a brief description of the technical solutions of the present application, related technical terms, and the writing method of the embodiments are given.

The voice recognition method provided in this application is usually applied to applications that use voice as a human-computer interaction medium. Such applications recognize collected voice signals as text, and then perform corresponding operations based on the text. The voice signals usually involve users. The local default information (for example, the name of the contact in the address book). The existing speech recognition technology uses a general search space for decoding and recognition of the above-mentioned speech signal to be recognized, and the general search space does not consider the differences between such applications in different user terminals, so it is usually unable to effectively recognize the speech signal. The content related to the local information on the client side leads to low recognition accuracy. In response to this problem, the technical solution of the present application incorporates user-side preset information in the process of constructing the search space for decoding the voice signal, which is equivalent to customizing the specific voice recognition requirements of the user-side, so that it can effectively identify Local information related to the client can achieve the purpose of improving the accuracy of voice recognition.

In the speech recognition system, the process of obtaining the most matching word sequence based on the speech signal to be recognized is called decoding. The search space used to decode speech signals mentioned in this application refers to all possible speech recognitions covered by the speech knowledge sources (for example: acoustic models, pronunciation dictionaries, language models, etc.) involved in the speech recognition system The result is the space. Correspondingly, the decoding process is a process of searching and matching the speech signal to be recognized in the search space to obtain the best matching word sequence.

The form of the search space can be diversified, and search spaces in which various knowledge sources are at relatively independent levels can be used. The decoding process is a process of calculating and searching layer by layer; it can also be based on a weighted finite state converter (Weighted Finite State Translator). The search space of State Transducer (WFST for short) integrates various knowledge sources into a unified WFST network (also known as WFST search space). Considering that the latter facilitates the introduction of different knowledge sources and can improve search efficiency, it is a better way for the technical solution of this application to perform voice recognition. Therefore, the embodiments provided in this application focus on the implementation based on the WFST network.

The core of the WFST search space is to use a weighted finite state converter to analogize the grammatical structure of a language and related acoustic characteristics. The specific operation method is: express the knowledge sources at different levels in the form of WFST, and then use the characteristics of WFST and the merger algorithm to integrate the above-mentioned WFST at different levels into a single WFST network, which is used for voice Identification search space.

The basic unit of the WFST network (that is, the basic input unit that drives the WFST to perform state transitions) can be selected according to specific needs. Considering the influence of phoneme context on phoneme pronunciation, in order to obtain higher recognition accuracy, in the embodiment provided in this application, context dependent triphone (Context Dependent triphone, triphone or triphone for short) is used as the WFST network. The basic unit, the corresponding knowledge sources for constructing the WFST search space include: triphone state binding field table, pronunciation dictionary, and language model.

The triphone state binding field table usually contains the binding relationship between the triphones based on the pronunciation characteristics. Usually, when the acoustic model is trained with the triphone as the modeling unit, due to the large number of possible combinations of the triphones, In order to reduce the requirements for training data, usually based on pronunciation characteristics, decision tree clustering method can be used to cluster different triphones under the maximum likelihood criterion, and use bundling technology to bind triphones with the same pronunciation characteristics to Together for parameter sharing, the triphone state binding field table is obtained; the pronunciation dictionary usually contains the correspondence between phonemes and words, which is a bridge between the acoustic layer (physical layer) and the semantic layer , The content of the acoustic layer and the content of the semantic layer are coupled and associated together; the language model provides relevant knowledge of the language structure and is used to calculate the probability of the word sequence appearing in natural language. In specific implementations, n-grams are usually used. The (n-gram) grammar language model can be specifically modeled by counting the likelihood of words appearing after each other.

When using the WFST network constructed based on the above knowledge sources for speech recognition, in order to drive the WFST to perform the required search, you can first extract the feature vector sequence of the voice signal to be recognized, and then use the pre-trained model to calculate the feature vector corresponding to each The probability of triphones, and according to the probabilities of each triphone, a decoding operation is performed in the WFST search space to obtain a word sequence corresponding to the speech signal to be recognized.

It should be noted that in the embodiments provided in this application, context-sensitive triphones are used as the basic unit of the WFST network. In other implementations, other phonetic units can also be used as the basic unit of the WFST network, such as single Phoneme, or triphone state, etc. Using different basic units, there will be certain differences in the construction of the search space and the calculation rate according to the feature vector. For example, if the triphone state is used as the basic unit, then the HMM-based acoustic model can be integrated when constructing the WFST network. When performing speech recognition, the probability that the feature vector corresponds to each triphone state can be calculated. The above are all changes to the specific implementation. As long as the user-side preset information is included in the process of constructing the search space, the technical solution of this application can also be realized without departing from the technical core of this application. Within the scope of protection applied for.

Hereinafter, the embodiments of the present application will be described in detail. Please refer to FIG. 1, which is a flowchart of an embodiment of a voice recognition method of this application. The method includes steps 101 to 104. In the specific implementation, in order to improve the execution efficiency, the relevant preparation work can usually be completed before step 101 (this stage can also be called the preparation stage) to generate a class-based language model, pre-processing Set the structured WFST and the acoustic model for speech recognition, etc., so as to prepare for the execution of step 101. The preparation phase will be explained in detail below.

In the preparation phase, the language model can be trained in the following manner: replacing the preset named entities in the text used for training the language model with tags corresponding to the preset topic category, and using the text to train the language model. The named entity usually refers to an entity with a specific category in the text, for example: names of people, names of songs, names of organizations, names of places, and so on.

Let’s take a call application as an example to illustrate: the default subject category is call, the corresponding label is "$CONTACT", and the default named entity is a person’s name. Then when the language model is pre-trained, the person’s name in the training text can be replaced For the corresponding label, for example, replace "Xiaoming" in "I want to call Xiaoming" with "$CONTACT", and then get the training text "I want to call $CONTACT". The text training language model after the above entity replacement is used to obtain a class-based language model. On the basis of the above-mentioned language model obtained through training, a language model-based WFST can also be generated in advance, hereinafter referred to as a G-structure WFST.

Preferably, in order to reduce the scale of the language model and the corresponding G-structure WFST, when the language model is pre-trained, the text for the preset topic category (also called class-based training text) can be selected for training. For example, if the preset theme category is making a call, the text for the preset theme category may include: I want to call Xiaoming, make a call to Xiaoming, and so on.

Taking into account the diversity of user-end devices and applications that use voice as the human-computer interaction medium, two or more topic categories can be preset, and a class-based language model can be pre-trained and constructed for each topic category. G structure WFST based on the language model.

In the preparation stage, WFST based on pronunciation dictionary, referred to as L-structure WFST, and WFST based on triphone state binding field table, referred to as C-structure WFST can also be constructed in advance, and the above-mentioned WFST can be adjusted in a preset manner. Perform targeted and selective merging operations. For example, you can merge the WFST of the C structure and the L structure into the WFST of the CL structure, or you can merge the WFST of the L structure and the G structure into the WFST of the LG structure, and you can also combine The WFST of the C structure, the L structure and the G structure are merged into the WFST of the CLG structure. In this embodiment, the WFST of the CL structure and the WFST of the G structure are generated in the preparation stage (for the description of the merging operation, please refer to the relevant text in step 101).

In addition, an acoustic model for speech recognition can be pre-trained in the preparation phase. In this embodiment, each triphone is characterized by an HMM (Hidden Markov Model-Hidden Markov Model), and the hidden state of the HMM is one of the triphones (each triphone usually contains three states) , Using GMM (Gaussian mixture model-Gaussian mixture model) model to determine the transmitter rate of each hidden state output feature vector in HMM, using feature vectors extracted from a large amount of speech data as training samples, using Baum-Welch algorithm to learn With the parameters of the GMM model and the HMM model, the GMM model corresponding to each state and the HMM model corresponding to each triphone can be obtained. In the subsequent step 103, the pre-trained GMM and HMM models can be used to calculate the probability that the feature vector corresponds to each triphone.

In order to improve the accuracy of speech recognition, this embodiment uses DNN (Deep Neural Networks-Deep Neural Networks) model instead of GMM model when performing speech recognition. Correspondingly, in the preparation phase, it can be pre-trained for the feature vector according to the input. The DNN model corresponding to the state probability of each triphone is output. In specific implementation, based on the training of GMM and HMM models, the training samples can be forced to align, and labels corresponding to the states of each triphone can be added to the training samples, and the labeled training samples can be used to train to obtain the DNN model.

It should be noted that during the specific implementation, due to the relatively large amount of calculation in the preparation phase, the requirements for memory and calculation speed are relatively high, so the operations in the preparation phase are usually completed on the server side. In order to still be able to complete the voice recognition function in the absence of a network access environment, the method provided in this application is usually implemented on the client device. Therefore, each WFST generated in the preparation phase and each model used for acoustic probability calculation can be Pre-installed on the client device, for example: packaged with the application and installed on the client.

So far, the preparation phase involved in this embodiment has been described in detail, and specific steps 101 to 104 of this embodiment will be described in detail below.

Step 101: Use a preset voice knowledge source to generate a search space that contains preset information on the client side and is used to decode a voice signal.

In this step, the WFST search space is constructed to prepare for the subsequent speech recognition. In specific implementation, this step is usually performed during the startup phase (also known as the initialization phase) of the client application that uses voice as the human-computer interaction medium. By replacing tags, the weight is limited to the pre-generated at least based on the language model. The state converter adds user-side preset information corresponding to the preset theme category, and obtains a single weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model.

The processing process of this step may include the following steps 101-1 to 101-4, which will be further described with reference to FIG. 2 below.

Step 101-1: Determine the preset subject category to which the voice signal to be recognized belongs.

In a specific implementation, the preset theme category to which the voice signal belongs can be determined according to the type of the client terminal or the application program that collects the voice signal. The preset theme categories include: making calls, sending short messages, playing music, setting instructions, or other theme categories related to application scenarios. Among them, the client default information corresponding to making a call or sending a text message includes: the contact name in the address book; the client default information corresponding to the music being played includes: the name of the music in the music library; the user corresponding to the setting command The terminal default information includes: commands in the command set; for other application scenarios related topic categories, it can also correspond to the client default information involved in the application scenarios, which will not be repeated here.

For example: for a smart phone, the default theme category to which the voice signal to be recognized belongs can be determined according to the type of the client: making a call or sending a text message, and the corresponding client default information is the contact name in the address book; for a smart speaker, The theme category can be determined as: playing music, and the corresponding user-side preset information is the name of the song in the music library; for robots, the theme category can be determined as: setting commands, and the corresponding client-side default information is the commands in the command set.

Considering that the user-end device can have multiple applications that use voice as the human-computer interaction medium at the same time, different applications involve different user-end preset information. For example, a smart phone can also be equipped with a music player based on voice interaction. In this case, the preset theme category to which the voice signal to be recognized belongs can be determined according to the currently activated application.

Step 101-2: Select a pre-generated G structure WFST corresponding to the preset theme category.

In the case of multiple preset subject categories, multiple G-structure WFSTs are usually generated in the preparation stage, and each G-structure WFST corresponds to a different preset subject category. In this step, a G structure WFST corresponding to the preset theme category determined in step 101-1 is selected from a plurality of G structure WFSTs generated in advance.

Step 101-3: Add the client default information to the selected G structure WFST by replacing the corresponding label with the client default information corresponding to the preset theme category.

When training a class-based language model for each preset topic category in the preparation phase, replace the preset named entity in the training text with a label corresponding to the corresponding topic category. For example, if the topic category is making a call or sending a text message, the training text Replace the person’s name in the “$CONTACT” tag with the “$CONTACT” tag, and the theme category is the playing music. Replace the music name in the training text with the “$SONG” tag. Therefore, the generated G-structure WFST usually contains the corresponding preset theme category. Label information. This step replaces the corresponding tags in the G structure WFST selected in step 101-2 with the user terminal preset information corresponding to the preset theme category determined in step 101-1, so as to add the client terminal to the selected G structure WFST The purpose of the default information.

For example, if the subject category is making a call or sending a text message, you can replace the "$CONTACT" tag in the G structure WFST with the name of the contact person in the local address book of the client, such as "Zhang San", "Li Si", etc.; If the category is to play music, you can replace the "$SONG" tag in the G structure WFST with the name of the song in the local music library of the client, such as "March of Volunteers". The specific replacement can be achieved by replacing the state transition link corresponding to the label with a number of parallel state transition links. Please refer to Figure 3 and Figure 4 for examples of replacing contacts in the client address book, where Figure 3 is a schematic diagram of the G structure WFST before replacement, and Figure 4 is using "Zhang San" and "Zhang San" in the address book. The schematic diagram of the G structure WFST obtained after the replacement of "Li Si".

Step 101-4: Combine the WFST of the G structure with the user-side preset information and the WFST of the pre-generated CL structure to obtain a single WFST network.

In this embodiment, the knowledge source used for speech recognition involves content from the language layer (language model) to the entity layer (the triphone state binding field table). The task of this step is to merge the WFST of different levels (also It is called combination and combination) to get a single WFST network.

For two WFSTs, the basic condition for merging is that the output symbol of one WFST is a subset of the input symbol set of the other WFST. Under the premise of meeting the above requirements, if two WFSTs, such as A and B respectively, are integrated into a new WFST: C, then each state of C is composed of the state of A and the state of B, and each state of C Each success path consists of A’s success path P _a and B’s success path P _b . The input is i[P]=i[P _a ], and the output is o[P]=o[P _b ]. The weighted value is obtained by _{performing corresponding calculations on the weighted values of P a} and P _b , and the finally obtained C contains the characteristics of the finite state converter shared by A and B and the search space. In specific implementation, the merging algorithm provided by the OpenFst library can be used to complete the merging operation of the two WFSTs.

Specific to this embodiment, it can be understood that the WFST of the L structure can be regarded as the correspondence between monophones and words, and the WFST of the C structure establishes the correspondence between triphones and monophones, and its output is the same as that of the L structure. The input of WFST corresponds to each other and can be merged. In the preparation stage of this embodiment, the WFST of the CL structure has been obtained through merging. In this step, the WFST of the CL structure is added to the default information of the client in step 101-3. G-structure WFST is merged to obtain a WFST network whose input is triphone probability and output is word sequence, so that WFST at different levels and corresponding to different knowledge sources are integrated into a single WFST network, which constitutes a single WFST network. The search space for speech recognition.

Preferably, in order to speed up the merging speed of the CL structure WFST and the G structure WFST and reduce the time-consuming initialization, this embodiment does not use the conventional WFST merging method when performing the merging operation, but adopts a prediction-based merging method. (lookahead merge method). The lookahead merging method, that is, in the merging process of the two WFSTs, through the prediction of the future path, it is judged whether the merge operation currently performed will lead to an unreachable final state (non-coaccessible state), and if it is, the current state is blocked. Operation, do not perform subsequent merge operations. Through forecasting, unnecessary merging operations can be terminated early, which not only saves merging time, but also reduces the scale of the final WFST generated and reduces the storage space occupation. In specific implementation, a filter with lookahead function provided by the OpenFst library can be used to realize the above predictive screening function.

Preferably, in order to speed up the merging speed of the CL structure WFST and the G structure WFST, the vocabulary used in the pre-training of the language model in this embodiment is consistent with the words contained in the pronunciation dictionary. Generally speaking, the number of words in the vocabulary is usually greater than the number of words in the pronunciation dictionary, and the number of words in the vocabulary is directly related to the size of the WFST of the G structure. If the WFST of the G structure is relatively large, and CL The WFST of the structure is more time-consuming to merge, so when training the language model in the preparation phase in this embodiment, the scale of the vocabulary is reduced, so that the words in the vocabulary are consistent with the words in the pronunciation dictionary, thereby achieving a reduced CL structure The effect of the merging time of WFST and G structure WFST.

So far, through steps 101-1 to 101-4, the initialization process of the technical solution is completed, and the WFST search space containing the preset information on the user side is generated.

It should be noted that in this embodiment, in the preparation stage, the WFST of the CL structure is merged in advance, and the WFST of the G structure is generated. In this step 101, the user-side preset information is added to the G structure WFST, and the CL structure and The G structures are combined to obtain a single WFST. In other implementations, other merging strategies can also be adopted. For example, the WFST of LG structure is merged in advance in the preparation phase, and the user-side preset information is added to the WFST in this step 101, and then combined with the WFST generated in the preparation phase. The C structure WFST is merged; or, the WFST of the CLG structure is merged directly in the preparation stage, and the user-side preset information is added to the WFST in this step 101. Considering that the WFST generated in the preparation phase takes up storage space on the user side, in an application scenario where there are multiple class-based language models (corresponding to a WFST with multiple G structures), if the WFST of each G structure is combined with other The WFST merging will occupy a larger storage space. Therefore, the merging method adopted in this embodiment is a preferred implementation method, which can reduce the user-side storage space occupied by the WFST generated in the preparation phase.

Step 102: Extract the feature vector sequence of the voice signal to be recognized.

The voice signal to be recognized is usually a time-domain signal. In this step, a feature vector sequence that can characterize the voice signal is obtained through two processing processes of framing and feature vector extraction. This will be further described with reference to FIG. 5 below.

Step 102-1: Perform framing processing on the voice signal to be recognized according to the preset frame length to obtain multiple audio frames.

In specific implementation, the frame length can be preset according to requirements, for example, it can be set to 10ms or 15ms, and then the speech signal to be recognized is segmented frame by frame according to the frame length, thereby segmenting the speech signal into multiple audio frames. According to different splitting strategies adopted, adjacent audio frames may not overlap or overlap.

Step 102-2: Extract the feature vector of each audio frame to obtain the feature vector sequence.

After the voice signal to be recognized is divided into multiple audio frames, the feature vector that can characterize the voice signal can be extracted frame by frame. Due to the relatively weak description ability of the speech signal in the time domain, the Fourier transform can usually be performed for each audio frame, and then the frequency domain features can be extracted as the feature vector of the audio frame. For example, the MFCC (Mel Frequency Cepstrum Coefficient-Meer) can be extracted. Frequency cepstral coefficient) feature, PLP (Perceptual Linear Predictive-Perceptual Linear Predictive) feature, or LPC (Linear Predictive Coding-Linear Predictive Coding) feature, etc.

The following takes the extraction of the MFCC feature of a certain audio frame as an example to further describe the feature vector extraction process. First, the time-domain signal of the audio frame is obtained through FFT (Fast Fourier Transformation) to obtain the corresponding spectrum information, the spectrum information is passed through the Mel filter bank to obtain the Mel spectrum, and the cepstrum analysis is performed on the Mel spectrum. The core generally uses DCT (Discrete Cosine Transform) for inverse transformation, and then takes preset N coefficients (for example, N=12 or 38) to obtain the feature vector of the audio frame: MFCC feature. Each audio frame is processed in the above manner to obtain a series of feature vectors that characterize the speech signal, that is, the feature vector sequence described in this application.

Step 103: Calculate the probability that the feature vector corresponds to the basic unit of the search space.

In this embodiment, the basic unit of the WFST search space is triphones, so this step calculates the probability that the feature vector corresponds to each triphone. In order to improve the accuracy of speech recognition, this embodiment uses an HMM model and a DNN model with powerful feature extraction capabilities to calculate the probability. In other implementations, other methods can also be used, such as: traditional GMM and HMM model calculations The probability can also realize the technical solution of this application, and it is also within the protection scope of this application.

In specific implementation, on the basis of calculating the feature vector corresponding to the state of each triphone, the probability of the feature vector corresponding to each triphone can be further calculated. The processing process of this step will be further described with reference to Fig. 6 below.

Step 103-1: Use the pre-trained DNN model to calculate the probability that the feature vector corresponds to each triphone state.

The DNN model has been pre-trained in the preparation stage of this embodiment. In this step, the feature vector extracted in step 102 is used as the input of the DNN model, and the probability that the feature vector corresponds to each triphone state can be obtained. For example: the number of triphones is 1000, and each triphone contains 3 states, so there are a total of 3000 triphone states. The output of the DNN model in this step: The feature vector corresponds to the probability of each state in the 3000 triphone states.

Preferably, since the amount of calculation involved in the use of the DNN model is usually very large, this embodiment uses the data parallel processing capability provided by the hardware platform to increase the speed of the calculation using the DNN model. For example, currently embedded devices and mobile devices use ARM architecture platforms. On most current ARM platforms, there are SIMD (single instruction multiple data) NEON instruction sets. The set can process multiple data in one instruction, and has certain data parallel processing capabilities. In this embodiment, vectorized programming can form a programming generic of single instruction stream and multiple data flow, so that the hardware platform can be fully utilized The provided data parallel processing capability realizes the purpose of accelerating DNN calculation.

When the technical solution of this application is implemented on the client device, in order to be able to match the hardware capabilities of the client, the scale of the DNN model is usually reduced. This often leads to a decrease in the accuracy of the DNN model and the ability to recognize different voice content. Will follow the decline. In this embodiment, due to the hardware acceleration mechanism, the scale of the DNN model may not be reduced or minimized, so that the accuracy of the DNN model can be retained to the greatest extent and the recognition accuracy rate can be improved.

Step 103-2: According to the probability that the feature vector corresponds to each triphone state, the pre-trained HMM model is used to calculate the probability that the feature vector corresponds to each triphone.

In the preparation stage, the HMM model for each triphone has been trained. In this step, according to the probability of the consecutive input of several feature vectors corresponding to the state of each triphone, the HMM model is used to calculate the transition probability corresponding to each triphone, so as to obtain The feature vector corresponds to the probability of each triphone.

The calculation process is actually the process of calculating the corresponding transfer probability according to the propagation process of the continuous feature vector on each HMM. The calculation process is further explained by taking the calculation of the probability for a certain triphone (including 3 states) as an example. Among them, pe(i,j) represents the transmitter rate of the feature vector of the i-th frame in the j-th state, and pt(h,k) represents the transition probability from the h state to the k state: 1) The feature vector of the first frame Corresponding to the state 1 of the corresponding HMM, with the transmitter rate pe(1,1); 2) For the feature vector of the second frame, if the state 1 of the HMM transitions to the state 1, the corresponding probability is pe(1,1) )*pt(1,1)*pe(2,1), if you transition from state 1 to state 2, the corresponding probability is pe(1,1)*pt(1,2)*pe(2,2), Determine whether to transition to state 1 or state 2 based on the above probability; 3) The above-mentioned similar calculation method is also used for the feature vector of the third frame and the feature vector of the subsequent frame, until the transition from state 3, and the propagation on the HMM so far At the end, the probability of the feature vector of the consecutive multiple frames for the HMM is obtained, that is, the probability corresponding to the triphone represented by the HMM.

For the continuously input feature vector, the transfer probability of propagation on each HMM is calculated using the above method, so as to obtain the probability corresponding to each triphone.

Step 104: Using the probability as an input, perform a decoding operation in the search space to obtain a word sequence corresponding to the feature vector sequence.

According to the probability that the feature vector output in step 103 corresponds to each triphone, a decoding operation is performed in the WFST network to obtain a word sequence corresponding to the feature vector sequence. This process is usually a search process of performing a graph search and finding the path with the highest score. The commonly used search method is the Viterbi algorithm. Its advantage is that the dynamic programming method saves the amount of calculation and can also achieve time synchronization decoding.

Considering that in the actual decoding process, due to the huge search space, the calculation amount of the Viterbi algorithm is still very large. In order to reduce the calculation amount and increase the calculation speed, not all possible subsequent paths of the path are expanded during the decoding process. Instead, it only expands those paths near the best path, that is, in the process of searching using the Viterbi algorithm, appropriate pruning strategies can be used to improve the search efficiency, for example: the Viterbi column search algorithm or It adopts bar graph pruning strategy and so on.

So far, the word sequence corresponding to the feature vector sequence is obtained through decoding, that is, the recognition result corresponding to the speech signal to be recognized is obtained. Since the user-side preset information is added when constructing the search space for voice recognition in step 101, the above-mentioned voice recognition process can generally identify the voice content related to the local information of the user-side relatively accurately.

Considering that the local information on the client side may be modified or deleted by the user, in order to further ensure the accuracy of the word sequence obtained through the above decoding process, this embodiment also provides a preferred implementation method: Perform text matching to verify the accuracy of the word sequence, and generate a corresponding voice recognition result according to the verification result.

In specific implementation, the above-mentioned preferred embodiment may include steps 104-1 to 104-4 listed below, which will be further described with reference to FIG. 7 below.

Step 104-1: Select a word to be verified from the word sequence corresponding to the preset information on the client side.

For example: for a phone call application, the default information on the client side is the contact name in the address book, and the result of voice recognition is the word sequence "Call Xiaoming", then through matching with the template or the grammatical analysis process, you can It is determined that "Xiaoming" in the word sequence is the word to be verified corresponding to the preset information on the client side.

Step 104-2: Search for the word to be verified in the preset information on the client side. If it is found, it is determined that the accuracy verification is passed, and step 104-3 is executed; otherwise, step 104-4 is executed.

In this step, by performing precise matching at the text level, it is determined whether the word to be verified belongs to the corresponding user-side preset information, so as to verify the accuracy of the word sequence.

Still using the example in step 104-1, this step looks for the contact "Xiaoming" in the client address book, that is, whether the information related to the contact name in the address book contains the string "Xiaoming" If it is included, it is determined that the accuracy verification is passed, and the step 104-3 is continued; otherwise, the step 104-4 is performed.

Step 104-3: Use the word sequence as a voice recognition result.

The execution of this step shows that the word to be verified contained in the word sequence obtained by decoding is consistent with the preset information on the user side. The word sequence can be output as the voice recognition result, thereby triggering the use of the voice recognition result To perform the corresponding actions.

Step 104-4: Correct the word sequence through a fuzzy matching method based on pinyin, and use the corrected word sequence as a voice recognition result.

When this step is performed, it usually means that the word to be verified contained in the word sequence obtained through decoding does not match the preset information on the client side. If the word sequence is output as the voice recognition result, then the relevant application Normally, the correct operation cannot be performed. Therefore, in this case, the necessary corrections can be made to the word sequence through fuzzy matching at the pinyin level.

In the specific implementation, the above-mentioned correction function can be realized by the following ways: by looking up the pronunciation dictionary, the word to be verified is converted into a pinyin sequence to be verified, and each word in the preset information of the user terminal is also converted into a comparison. Pinyin sequence, and then sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and select the top words sorted according to the similarity from high to low from the user-side preset information, and finally use The selected word replaces the word to be verified in the word sequence to obtain the revised word sequence.

In specific implementations, different methods can be used to calculate the similarity between two pinyin sequences. This embodiment adopts the method of calculating the similarity based on the edit distance, for example, the edit distance between the two pinyin sequences is used to calculate the similarity between the two pinyin sequences. The reciprocal of the sum is added as the similarity. The Edit Distance refers to the minimum number of editing operations required to convert two strings from one to the other. The editing operations include replacing one character with another, and inserting one Character, delete a character. Generally speaking, the smaller the editing distance, the greater the similarity between the two strings.

The example in step 104-1 above is still used. The word sequence is "Call Xiaoming" and the word to be verified is "Xiaoming". If "Xiaoming" is not found in the contacts in the client address book, you can search for pronunciation Dictionary, convert Xiaoming into the pinyin sequence "xiaoming" to be verified, and convert each contact name in the address book into the corresponding pinyin sequence, that is: compare the pinyin sequence, and then calculate the "xiaoming" and each comparison in turn The edit distance between the pinyin sequences, and then select the contact name corresponding to the compared pinyin sequence with the smallest edit distance (highest similarity) (for example: "xiaomin" corresponding to "xiaomin"), and replace the word sequence in The word to be verified has completed the correction of the word sequence, and the corrected word sequence can be used as the final voice recognition result.

In specific implementation, it is also possible to first calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence and sort it from high to low, and select several (for example, three) compared pinyin in the top order The words corresponding to the sequence are then prompted to the user-end user through screen output and other methods, and the user selects the correct word from them, and replaces the word to be verified in the word sequence according to the word selected by the user.

So far, the specific implementation of the voice recognition method provided by the present application has been described in detail through the above steps 101 to 104. For ease of understanding, please refer to FIG. 8, which is an overall framework diagram of the speech recognition process provided in this embodiment. The dashed frame corresponds to the preparation stage described in this embodiment, and the solid frame corresponds to a specific speech recognition process.

It should be noted that step 101 described in this embodiment can be executed once every time a client application that uses voice as the interactive medium is started, that is, every time it is started, it will regenerate the default information of the client and be used for matching. The search space for decoding the speech signal can also be generated and stored only when the client application is first launched, and then periodically updated, which can reduce the time to generate the search space every time the application is launched Overhead (you can directly use the previously generated search space), improve the execution efficiency of speech recognition, and improve the user experience.

In addition, the method provided in this application is usually implemented on user-end equipment, and the user-end equipment includes: smart mobile terminals, smart speakers, robots, or other devices that can run the method. Implementation of the specific implementation of the method provided in this application. However, in other implementation manners, the method provided in this application can also be implemented in application scenarios based on client and server modes. In this case, each WFST generated in the preparation phase and the model used for acoustic probability calculation do not need Pre-installed on the client device, each time the client application is started, the corresponding client preset information can be uploaded to the server, and the subsequent collected voice signals to be recognized are also uploaded to the server, from the server side Implementing the method provided in this application and returning the decoded word sequence to the user terminal can also implement the technical solution of this application and obtain corresponding beneficial effects.

In summary, the voice recognition method provided by the present application includes user-side preset information when generating the search space for decoding the voice signal, so it can be relatively accurate when recognizing the voice signal collected by the user-side Recognize the local information related to the user terminal, which can improve the accuracy of voice recognition and improve the user experience.

In particular, the method provided in this application is used for voice recognition on the user-side equipment. Since the local information of the user-side is added, it can compensate to a certain extent for the problem of the decrease in the recognition accuracy caused by the probability calculation model and the shrinking of the search space. In this way, it can not only meet the needs of voice recognition in an environment without network access, but also achieve a certain recognition accuracy rate. Furthermore, after the word sequence is obtained by decoding, the accuracy of speech recognition can be further improved by adopting the matching verification scheme based on the text level and the pinyin level provided in this embodiment. The actual test results show that: the character error rate (CER) of the conventional speech recognition method is about 20%, while using the method provided in this application, the character error rate is below 3%. The above data fully explains the method The beneficial effect is significant.

In the above-mentioned embodiment, a voice recognition method is provided. Correspondingly, this application also provides a voice recognition device. Please refer to FIG. 9, which is a schematic diagram of an embodiment of a voice recognition device of this application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For related parts, please refer to the part of the description of the method embodiment. The device embodiments described below are merely illustrative.

A voice recognition device of this embodiment includes: a search space generating unit 901, configured to use a preset voice knowledge source to generate a search space containing user-side preset information for decoding a voice signal; feature vector extraction The unit 902 is used to extract the feature vector sequence of the speech signal to be recognized; the probability calculation unit 903 is used to calculate the probability that the feature vector corresponds to the basic unit of the search space; the decoding and search unit 904 is used to input the probability of the Perform a decoding operation in the search space to obtain a word sequence corresponding to the feature vector sequence.

Optionally, the search space generating unit is specifically configured to add user-side preset information corresponding to the preset topic category to the weighted finite state converter generated in advance at least based on the language model by replacing tags, and Obtain a single weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model; the language model is pre-generated by the language model training unit, and the language model training unit is used to The preset named entities in the text for training the language model are replaced with tags corresponding to the preset topic categories, and the text is used to train the language model.

Optionally, the search space generating unit includes: a first user-side information adding subunit, which is used to add a language model-based weighted finite state converter corresponding to a preset topic category to a pre-generated language model-based weighted finite state converter by replacing tags The user-side preset information; the weighted finite state converter merging subunit is used to combine the weighted finite state converter with the user-side preset information and the pre-generated triphone-based state binding field table Combined with the weighted finite state converter of the pronunciation dictionary, the single weighted finite state converter is obtained.

Optionally, the decoding space generating unit includes: a second user-side information adding subunit, which is used to add and preset topic categories to the pre-generated weighted finite state converter based at least on the language model by replacing tags Corresponding user-side preset information; the unified weighted finite state converter acquisition subunit is used to obtain the binding field table, pronunciation dictionary, pronunciation dictionary based on triphone state after the second user-side information adding subunit completes the adding operation And a single weighted finite state converter of the language model; wherein the second user-side information adding subunit includes: a topic determination subunit for determining the preset topic category to which the speech signal to be recognized belongs; weighted finite state converter selection Sub-unit for selecting the weighted finite state converter based on the language model that is generated in advance and corresponding to the preset topic category; the label replacement sub-unit is used to correspond to the preset topic category by using The method of replacing the corresponding label with the default information on the client side to add the default information on the client side to the selected weighted finite state converter.

Optionally, the theme determining subunit is specifically configured to determine the preset theme category to which the voice signal belongs according to the type of the user terminal or the application program that collects the voice signal.

Optionally, the weighted finite state converter merging subunit is specifically used to perform a merging operation using a prediction-based method, and obtain the single weighted finite state converter.

Optionally, the probability calculation unit includes: a triphone state probability calculation subunit for calculating the probability that a feature vector corresponds to each triphone state by using a pre-trained DNN model; a triphone probability calculation subunit for calculating the probability of each triphone state according to the feature The vector corresponds to the probability of each triphone state, and a pre-trained HMM model is used to calculate the probability that the feature vector corresponds to each triphone.

Optionally, the feature vector extraction unit includes: a framing subunit for performing framing processing on the speech signal to be recognized according to a preset frame length to obtain multiple audio frames; a feature extraction subunit for extracting each audio signal The feature vector of the frame to obtain the feature vector sequence.

Optionally, the device includes: an accuracy verification unit configured to verify the word sequence through text matching with the user-side preset information after the decoding search unit obtains the word sequence corresponding to the feature vector sequence According to the accuracy of the verification result, the corresponding voice recognition result is generated.

Optionally, the accuracy verification unit includes: a word-to-be-verified selection sub-unit for selecting a word to be verified from the word sequence corresponding to the preset information on the user side; and a search sub-unit for selecting the word to be verified in the word sequence; Search for the word to be verified in the preset information on the client side; a recognition result confirmation sub-unit for determining that the accuracy verification is passed after the search sub-unit finds the word to be verified, and to compare the word sequence As the result of speech recognition; the recognition result correction subunit is used to correct the word sequence through the fuzzy matching method based on pinyin after the search subunit does not find the word to be verified, and use the corrected word sequence as the voice Identification result.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, used to convert the word to be verified into a pinyin sequence to be verified; a comparison pinyin sequence conversion subunit, used to convert the Each word in the preset information on the client side is converted into a compared pinyin sequence; the similarity calculation and selection subunit is used to sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and from the The user-side preset information selects the top words sorted from high to low according to the similarity; the word to be verified replacement subunit is used to replace the word to be verified in the word sequence with the selected word to obtain the correction The sequence of words after.

In addition, this application also provides another voice recognition method. Please refer to FIG. 10, which is a flowchart of an embodiment of another voice recognition method provided by this application. To repeat, the following focuses on the differences. Another voice recognition method provided by this application includes:

Step 1001: Obtain a word sequence corresponding to the voice signal to be recognized through decoding.

For speech recognition, the decoding process is a process of searching in the search space used for speech recognition to obtain the best word sequence corresponding to the speech signal to be recognized. The search space may be a WFST network based on various knowledge sources, or it may be a search space of other forms; the search space may include or may not include preset information on the client side. This embodiment does not There is no specific restriction on this.

Step 1002: Verify the accuracy of the word sequence through text matching with preset information on the client side, and generate a corresponding voice recognition result according to the verification result.

This step includes the following operations: selecting the word to be verified from the word sequence corresponding to the user-side preset information; searching for the word to be verified in the user-side preset information; The accuracy of the statement is verified, and the word sequence is used as the voice recognition result; otherwise, the word sequence is corrected through the fuzzy matching method based on pinyin, and the corrected word sequence is used as the voice recognition result.

The modification of the word sequence through the fuzzy matching method based on pinyin includes: converting the word to be verified into a pinyin sequence to be verified; and converting each word in the user-side preset information into a matched pinyin sequence; Calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence in turn, and select the top words sorted from high to low according to the similarity from the preset information on the user side; use the selected word Replace the word to be verified in the word sequence to obtain the revised word sequence.

Wherein, the converted pinyin sequence can be realized by searching a pronunciation dictionary, and the similarity can be calculated according to the edit distance between the two pinyin sequences.

The method provided in this application is usually used in applications that use voice as an interactive medium. The voice to be recognized collected by such applications may involve user-side information. The method provided in this application uses the word sequence obtained by decoding The text matching with the preset information on the client side can verify the accuracy of the word sequence, thereby providing a basis for making necessary corrections to the word sequence. Further, by using fuzzy matching based on the pinyin level, the word sequence can be corrected, thereby improving the accuracy of speech recognition.

In the above-mentioned embodiment, another voice recognition method is provided. Correspondingly, this application also provides another voice recognition device. Please refer to FIG. 11, which is a schematic diagram of an embodiment of another voice recognition device of this application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. The device embodiments described below are merely illustrative.

A speech recognition device of this embodiment includes: a word sequence obtaining unit 1101, which is used to obtain a word sequence corresponding to a voice signal to be recognized through decoding; and a word sequence verification unit 1102, which is used to perform text matching with user-end preset information The accuracy of the word sequence is verified, and a corresponding voice recognition result is generated according to the verification result.

Optionally, the word sequence verification unit includes: a word-to-be-verified selection sub-unit for selecting a word to be verified from the word sequence corresponding to the user-side preset information; a search sub-unit for selecting the word to be verified in the word sequence; Search for the word to be verified in the preset information on the client side; a recognition result confirmation sub-unit for determining that the accuracy verification is passed after the search sub-unit finds the word to be verified, and to compare the word sequence As the result of speech recognition; the recognition result correction subunit is used to correct the word sequence through the fuzzy matching method based on pinyin after the search subunit does not find the word to be verified, and use the corrected word sequence as the voice Identification result.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, used to convert the word to be verified into a pinyin sequence to be verified; a comparison pinyin sequence conversion subunit, used to convert the Each word in the preset information on the client side is converted into a compared pinyin sequence; the similarity calculation and selection subunit is used to sequentially calculate the similarity between the pinyin sequence to be verified and each compared pinyin sequence, and from the The user-side preset information selects the top words sorted from high to low according to the similarity; the word to be verified replacement subunit is used to replace the word to be verified in the word sequence with the selected word to obtain the correction The sequence of words after.

Although this application is disclosed as above in preferred embodiments, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, this application The scope of protection shall be subject to the scope defined by the claims of this application.

In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

1. Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM) , Read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital multi-function Optical discs (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission media, can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

2. Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems or computer program products. Therefore, this application may adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Moreover, this application can be in the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program codes. .

Claims (31)

A method for speech recognition, which is characterized in that it comprises: using a preset speech knowledge source to generate a search space containing user-side preset information for decoding a speech signal; extracting a feature vector sequence of the speech signal to be recognized; and calculating; The feature vector corresponds to the probability of the basic unit of the search space; using the probability as the input, the decoding operation is performed in the search space to obtain the word sequence corresponding to the feature vector sequence, where the search space includes: a weighted finite state converter, Among them, the basic unit of the search space includes: context-related triphones; the preset knowledge sources include: pronunciation dictionary, language model, and triphone state binding field table, wherein, the preset phonetic knowledge source is used to generate The search space that contains the preset information on the client side and is used to decode the speech signal includes: by replacing tags, adding the pre-generated weighted finite state converter at least based on the language model corresponding to the preset topic category The user terminal presets information, and obtains a single-weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model; the language model is pre-trained in the following way: it will be used to train the language The preset named entities in the text of the model are replaced with tags corresponding to the preset topic categories, and the text is used to train the language model. The speech recognition method according to item 1 of the scope of the patent application, wherein the method of replacing tags is used to generate at least language-based The weighted finite state converter of the model adds the user-side preset information corresponding to the preset theme category, and obtains a single weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model, including: The way of replacing tags is to add the user-side preset information corresponding to the preset theme category to the pre-generated weighted finite state converter based on the language model; the weighted finite state converter with the client-side preset information added, and The pre-generated weighted finite state converter based on the triphone state binding field table and the pronunciation dictionary are combined to obtain the single weighted finite state converter. The speech recognition method according to item 1 of the scope of patent application, wherein the text used for training the language model refers to the text of the preset subject category. The speech recognition method according to item 1 of the scope of patent application, wherein the number of the preset topic categories is at least two; the number of the language models and the number of the weighted finite state machines based on at least the language model are respectively compared with the preset It is assumed that the number of subject categories is the same; the method of replacing tags, adding user-side preset information corresponding to the preset subject category to the weighted finite state converter based at least on the language model, includes: determining the voice signal to be recognized The default theme category to which it belongs; select the weighted finite state converter based at least on the language model that is generated in advance and corresponds to the default theme category; replace it with the user-side default information corresponding to the default theme category Add the default information of the client to the selected weighted finite state converter by means of the corresponding label. According to the voice recognition method described in item 4 of the scope of patent application, the determination of the preset subject category to which the voice signal to be recognized belongs is implemented in the following manner: the voice signal is determined according to the type of the user terminal or the application that collects the voice signal. The default theme category. According to the voice recognition method described in item 5 of the scope of patent application, the preset subject category includes: making a call or sending a text message, playing music, or setting instructions; the corresponding user-side preset information includes: The name of the contact, the name of the song in the music library, or the command in the command set. According to the speech recognition method described in item 2 of the scope of patent application, the merging operation includes: merging using a prediction-based method. The speech recognition method according to item 1 of the scope of patent application, wherein the vocabulary used for pre-training the language model is consistent with the words contained in the pronunciation dictionary. According to the speech recognition method described in item 1 of the scope of patent application, the calculation of the probability that the feature vector corresponds to the basic unit of the search space includes: using a pre-trained DNN model to calculate the probability that the feature vector corresponds to each triphone state; The feature vector corresponds to the probability of each triphone state, and the pre-trained HMM model is used to calculate the feature vector corresponding to each triphone. rate. According to the method for speech recognition according to item 9 of the scope of patent application, the execution speed of the step of using the pre-trained DNN model to calculate the probability of the feature vector corresponding to each triphone state is improved by the following method: provided by a hardware platform Parallel processing capacity of data. The speech recognition method according to any one of items 1 to 10 of the scope of patent application, wherein the extracting the feature vector sequence of the speech signal to be recognized includes: performing framing processing on the speech signal to be recognized according to a preset frame length, Obtain multiple audio frames; extract the feature vector of each audio frame to obtain the feature vector sequence. The speech recognition method according to item 11 of the scope of patent application, wherein the extracting the feature vector of each audio frame includes: extracting the MFCC feature, the PLP feature, or the LPC feature. The speech recognition method according to any one of items 1 to 10 of the scope of patent application, wherein, after the word sequence corresponding to the feature vector sequence is obtained, the following operation is performed: The matching verifies the accuracy of the word sequence, and generates the corresponding voice recognition result according to the verification result. The speech recognition method according to item 13 of the scope of patent application, wherein the verification of the accuracy of the word sequence is performed by text matching with the preset information of the client, and the corresponding voice recognition result is obtained according to the verification result, including: Select the word sequence to be verified corresponding to the default information of the client Word; find the word to be verified in the default information of the client; if it is found, it is determined to pass the accuracy verification, and the word sequence is used as the voice recognition result; otherwise, the word sequence is corrected by the fuzzy matching method based on pinyin, And use the corrected word sequence as the result of speech recognition. The speech recognition method according to item 14 of the scope of patent application, wherein the modification of the word sequence through a pinyin-based fuzzy matching method includes: converting the word to be verified into a pinyin sequence to be verified; and the user-side preset information Each word in is converted into a compared pinyin sequence; the similarity between the pinyin sequence to be verified and each compared pinyin sequence is calculated in turn, and the user-side preset information is selected according to the similarity from high to low The first word; replace the word to be verified in the word sequence with the selected word to get the revised word sequence. The speech recognition method according to item 15 of the scope of patent application, wherein the similarity includes a similarity calculated based on an edit distance. The voice recognition method according to any one of items 1 to 10 in the scope of patent application, wherein the method is implemented on a user-end device; the user-end device includes: a smart mobile terminal, a smart speaker, or a robot. A speech recognition device is characterized by comprising: a search space generating unit, which is used to generate a search containing preset information on the user side and used to decode a speech signal by using a preset speech knowledge source. Search space; feature vector extraction unit, used to extract the feature vector sequence of the speech signal to be recognized; probability calculation unit, used to calculate the probability that the feature vector corresponds to the basic unit of the search space; decoding search unit, used to use the probability as the input, Perform a decoding operation in the search space to obtain a word sequence corresponding to the feature vector sequence, wherein the search space generation unit is specifically configured to transform to a pre-generated weighted finite state based at least on a language model by replacing tags Add the user-side preset information corresponding to the preset theme category in the device, and obtain a single weighted finite state converter based on the triphone state binding field table, pronunciation dictionary, and language model; the language model is trained by the language model The language model training unit is pre-generated by the unit, and the language model training unit is used to replace the preset named entity in the text used for training the language model with a label corresponding to the preset topic category, and use the text to train the language model. The speech recognition device according to item 18 of the scope of patent application, wherein the search space generating unit includes: a first user-side information adding subunit, which is used to replace tags with a pre-generated language model based on limited weight The state converter adds the user-side preset information corresponding to the preset theme category; the weighted finite state converter merges the sub-unit for combining the weighted finite state converter with the client-side preset information and the pre-generated Weighted finite state transition based on triphone state binding field table and pronunciation dictionary Combine the converters to obtain the single weighted finite state converter. The speech recognition device according to item 18 of the scope of patent application, wherein the decoding space generating unit includes: a second user-side information adding subunit for adding tags to pre-generated weights based on language models at least The finite state converter adds the user-side preset information corresponding to the preset theme category; the unified weighted finite state converter obtains the subunit, which is used to obtain the triphone-based information after the second user-side information adding subunit completes the addition operation State binding field table, pronunciation dictionary, and single weighted finite state converter of language model; wherein, the second user terminal information adding subunit includes: a topic determination subunit for determining the preset to which the speech signal to be recognized belongs Subject category; weighted finite state converter selection subunit for selecting the weighted finite state converter based at least on the language model that is generated in advance and corresponding to the preset subject category; label replacement subunit for The method of replacing the corresponding label with the client default information corresponding to the default theme category adds the client default information to the selected weighted finite state converter. The voice recognition device according to item 20 of the scope of patent application, wherein the theme determining subunit is specifically configured to determine the preset theme category to which the voice signal belongs according to the type of the user terminal or the application program that collects the voice signal. The speech recognition device according to item 19 of the scope of patent application, wherein the weighted finite state converter merging subunit is specifically used for: The combination operation is performed based on the prediction method, and the single weighted finite state converter is obtained. The speech recognition device according to item 18 of the scope of patent application, wherein the probability calculation unit includes: a triphone state probability calculation subunit for calculating the probability that the feature vector corresponds to each triphone state by using a pre-trained DNN model; The triphone probability calculation subunit is used to calculate the probability that the feature vector corresponds to each triphone according to the probability that the feature vector corresponds to the state of each triphone using a pre-trained HMM model. The speech recognition device according to any one of items 18 to 23 in the scope of patent application, wherein the feature vector extraction unit includes: a framing subunit for performing framing processing on the speech signal to be recognized according to a preset frame length , To obtain multiple audio frames; feature extraction subunit, used to extract the feature vector of each audio frame, to obtain the feature vector sequence. The speech recognition device according to any one of items 18 to 23 in the scope of patent application, which includes: an accuracy verification unit for after the decoding and searching unit obtains the word sequence corresponding to the feature vector sequence, through the The user-side preset information performs text matching to verify the accuracy of the word sequence, and generates a corresponding voice recognition result according to the verification result. The speech recognition device according to item 25 of the scope of patent application, wherein the accuracy verification unit includes: a word selection subunit to be verified for selecting from the word sequence corresponding to The word to be verified in the user-side preset information; the search subunit is used to search for the word to be verified in the user-side preset information; the recognition result confirmation sub-unit is used when the search sub-unit finds the word to be verified , It is determined that the accuracy verification is adopted, and the word sequence is used as the voice recognition result; the recognition result correction subunit is used to correct the word sequence through the fuzzy matching method based on pinyin after the search subunit fails to find the word to be verified , And use the corrected word sequence as the voice recognition result. The speech recognition device according to item 26 of the scope of patent application, wherein the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, for converting the word to be verified into a pinyin sequence to be verified; The sequence conversion subunit is used to convert each word in the user-side preset information into a compared pinyin sequence; the similarity calculation selection subunit is used to sequentially calculate the difference between the pinyin sequence to be verified and each compared pinyin sequence The similarity of, and select the top words sorted according to the similarity from high to low from the preset information of the client; the word to be verified replacement subunit is used to replace the word to be verified in the word sequence with the selected word , Get the revised word sequence. A voice recognition method, which is characterized in that it comprises: obtaining a word sequence corresponding to a voice signal to be recognized through decoding; verifying the accuracy of the word sequence by text matching with user-side preset information, and generating a corresponding voice according to the verification result Recognition result, Wherein, the verification of the accuracy of the word sequence by text matching with the preset information of the client, and generating the corresponding voice recognition result according to the verification result, includes: selecting the pending information corresponding to the preset information of the client from the word sequence. Verification word; search for the word to be verified in the default information of the client; if found, it will be determined through the accuracy verification, and the word sequence will be used as the voice recognition result; otherwise, the word sequence will be corrected by the fuzzy matching method based on pinyin , And use the corrected word sequence as the voice recognition result. The speech recognition method according to item 28 of the scope of patent application, wherein the modification of the word sequence through a pinyin-based fuzzy matching method includes: converting the word to be verified into a pinyin sequence to be verified; and the user-side preset information Each word in is converted into a compared pinyin sequence; the similarity between the pinyin sequence to be verified and each compared pinyin sequence is calculated in turn, and the user-side preset information is selected according to the similarity from high to low The first word; replace the word to be verified in the word sequence with the selected word to get the revised word sequence. A voice recognition device, which is characterized by comprising: a word sequence acquisition unit for acquiring a word sequence corresponding to a voice signal to be recognized through decoding; a word sequence verification unit for performing a sentence with preset information on the client side The word matching verifies the accuracy of the word sequence, and generates the corresponding voice recognition result according to the verification result. The word sequence verification unit includes: a word selection subunit to be verified, which is used to select the user terminal corresponding to the word sequence from the word sequence. The word to be verified in the preset information; the search sub-unit is used to search for the word to be verified in the user-side preset information; the recognition result confirmation sub-unit is used to determine the pass after the search sub-unit finds the word to be verified The accuracy verification takes the word sequence as the voice recognition result; the recognition result correction sub-unit is used to correct the word sequence through the fuzzy matching method based on pinyin after the search sub-unit fails to find the word to be verified. The corrected word sequence is used as the result of speech recognition. The speech recognition device according to item 30 of the scope of patent application, wherein the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, for converting the word to be verified into a pinyin sequence to be verified; The sequence conversion subunit is used to convert each word in the user-side preset information into a compared pinyin sequence; the similarity calculation selection subunit is used to sequentially calculate the difference between the pinyin sequence to be verified and each compared pinyin sequence The similarity of, and select the top words sorted according to the similarity from high to low from the preset information of the client; the word to be verified replacement subunit is used to replace the word to be verified in the word sequence with the selected word , Get the revised word sequence.

Images