WO2021051544A1 - Voice recognition method and device - Google Patents

Abstract

Provided are a voice recognition method and device, relating to the technical field of artificial intelligence. The method comprises: obtaining a target voice to be recognized; extracting waveform characteristics and pitch characteristics corresponding to each frame of target speech; inputting, in order, the waveform characteristics and pitch characteristics corresponding to each frame of target speech into a trained speech recognition model, the speech recognition model comprising an encoding sub-model, a first decoding sub-model, and a second decoding sub-model; the encoding sub-model comprises a convolutional neural network and bidirectional long- and short-term memory networks; the first decoding sub-model comprises a speech–text matching unit, the second decoding sub-model comprises a text context matching unit, and the voice–text matching unit comprises a CTC loss function and attention model; according to the output of the voice recognition model, generating text corresponding to the target voice to be recognized. Thus by combining voice–text matching and text context matching, the accuracy of voice recognition is improved.

Description

Speech recognition method and device

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910869774.0, and the application name is "Voice Recognition Method and Device" on September 16, 2019, the entire content of which is incorporated into this application by reference.

【Technical Field】

This application relates to the field of artificial intelligence technology, and in particular to a voice recognition method and device.

【Background technique】

Speech recognition, also known as Automatic Speech Recognition (ASR), whose goal is to allow machines to recognize and understand speech signals into text, is an important branch of the development of modern artificial intelligence.

The traditional speech recognition technology is based on Hidden Markov Model (Hidden Markov Model; HMM), Gaussian Mixture Model (GMM), and Deep Neural Network-Hidden Markov Model (Deep Neural Network). -Hidden Markov Model; hereinafter referred to as: DNN-HMM) to establish an acoustic model. However, the accuracy of the above-mentioned speech recognition based on the above-mentioned acoustic model needs to be further improved.

【Content of Application】

The embodiments of the present application provide a voice recognition method and device, which can improve the accuracy of voice recognition.

In the first aspect, an embodiment of the present application provides a speech recognition method, which includes the following steps: acquiring the target speech to be recognized; extracting the waveform characteristics and pitch characteristics corresponding to the target speech in each frame; The waveform features and the tone features corresponding to the speech are sequentially input into the trained speech recognition model; wherein, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding sub-model, the encoding sub-model The model includes a convolutional neural network and a bidirectional long and short-term memory network, the first decoding sub-model includes a speech-text matching unit, the second decoding sub-model includes a text context matching unit, and the speech-text matching unit includes a CTC loss Function and attention model; according to the output of the voice recognition model, the text corresponding to the target voice to be recognized is generated.

In a second aspect, an embodiment of the present application also provides a voice recognition device, including: a first acquisition module, configured to acquire a target voice to be recognized; a first extraction module, configured to extract the target voice corresponding to each frame Waveform feature and pitch feature; a first input module for sequentially inputting the waveform feature and the pitch feature corresponding to each frame of the target speech into the trained speech recognition model; wherein, the speech recognition model includes An encoding sub-model, a first decoding sub-model and a second decoding sub-model, the encoding sub-model includes a convolutional neural network and a two-way long short-term memory network, the first decoding sub-model includes a voice-text matching unit, the first The second decoding sub-model includes a text context matching unit, and the voice-text matching unit includes a CTC loss function and an attention model; a generating module is used to generate the corresponding target voice to be recognized according to the output of the voice recognition model Text.

In the third aspect, the embodiments of the present application also provide a computer non-volatile readable storage medium. The computer non-volatile readable storage medium includes a stored program, wherein the non-volatile computer is controlled when the program is running. The device where the volatile readable storage medium is located executes the above voice recognition method.

In a fourth aspect, an embodiment of the present application also provides a computer device, including a memory and a processor, the memory is used to store information including program instructions, the processor is used to control the execution of the program instructions, and the program instructions are The above voice recognition method is realized when the processor is loaded and executed.

In the above technical solution, in the process of voice-text matching, the CTC loss function and attention model are combined, so that the voice features and text can be aligned. In the process of speech recognition, voice-text matching and text context matching are combined. Improve the accuracy of speech recognition.

【Explanation of the drawings】

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG. 1 is a schematic flowchart of a voice recognition method provided by an embodiment of this application;

FIG. 2 is a flowchart of an example of a voice recognition method provided by an embodiment of the application;

FIG. 3 is a schematic structural diagram of a speech recognition device provided by an embodiment of this application;

Fig. 4 is a schematic structural diagram of an embodiment of a computer device of this application.

【detailed description】

In order to better understand the technical solutions of the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

FIG. 1 is a schematic flowchart of a voice recognition method provided by an embodiment of the application. As shown in FIG. 1, the above method may include:

Step S101: Obtain a target voice to be recognized.

Specifically, the target voice in this embodiment is usually audio data input by the user obtained through an interactive application.

Step S102: Extract the waveform feature and pitch feature corresponding to each frame of target speech.

Specifically, the waveform feature and the pitch feature are one of the voice features in speech recognition. Among them, before extracting waveform features and pitch features, it is generally necessary to preprocess the target speech. Specifically, the input target voice is pre-emphasized first, and the high-frequency part of the voice signal is improved by using a high-pass filter to make the frequency spectrum smoother, and then the pre-emphasized target voice is framed and windowed, thereby The non-stationary speech signal is transformed into a short-term stable signal, and then the end point is detected to distinguish the speech and noise, and extract the effective speech part.

Among them, the waveform feature is extracted through the following steps: the pre-processed target speech is subjected to fast Fourier change, thereby converting the speech signal in the time domain into an energy spectrum in the frequency domain for analysis, and then passing the energy spectrum through a set of mel The triangular filter bank of the scale highlights the formant characteristics of the speech, and then the logarithmic energy output by each filter bank is calculated to obtain the waveform characteristics corresponding to the target speech.

The tonal feature is extracted by the following steps: the preprocessed target speech, the voice with a frequency between 50-450 Hz is extracted to obtain the tonal feature corresponding to the target speech. It should be noted that the human tone is between 50-450Hz, which is quite different from the tone of the environmental background sound. By extracting the voice between 50-450Hz, it is possible to filter out a large part of the silent voice segment and improve Subsequent speech recognition efficiency.

In step S103, the waveform characteristics and pitch characteristics corresponding to each frame of target speech are sequentially input into the trained speech recognition model.

Among them, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding sub-model. The encoding sub-model includes a convolutional neural network and a two-way long and short-term memory network. The first decoding sub-model includes a speech-text matching unit. The second decoding sub-model includes a text context matching unit, and the speech-text matching unit includes a CTC loss function and an attention model.

It should be particularly noted that the speech recognition model provided by the embodiments of this application includes an encoding sub-model, a first decoding sub-model and a second decoding sub-model. The encoding sub-model is used to encode waveform features and tonal features to generate abstractions. Feature vector to facilitate matching with the text in the text library.

The first decoding sub-model is used to determine the text corresponding to the target speech according to the feature vector. It can be understood that speech has the characteristic of continuity, so it is necessary to continuously input target speech frames into the speech recognition model, and the speech recognition model continuously performs speech recognition on each frame of target speech. That is, the coding sub-model continuously performs feature coding on the waveform characteristics and tonal characteristics corresponding to each frame of target speech, and the first decoding sub-model continuously recognizes the feature codes to generate text corresponding to each frame of target speech.

Specifically, the waveform feature and pitch feature corresponding to each frame of target speech is an 80-dimensional feature vector. The coding sub-model includes a convolutional neural network and a bidirectional long-short-term memory network. The convolutional neural network includes 4 layers of convolutional layers and 2 layers of maximum pooling layers, which are used to convolve waveform features and pitch features to generate the target speech. Convolution feature, the convolution feature corresponding to each frame of target speech is sequentially input into the bidirectional long and short-term memory network to obtain the time sequence feature of the target speech. The first decoding sub-model includes a CTC loss function and an attention model. The CTC loss function uses a frame-level letter sequence and a blank label, which can separate the timing features of the target speech to facilitate matching with the text. The attention model can group the temporal features corresponding to the same text and the same phrase in order to align the temporal features. Among them, the CTC loss function separates the timing characteristics of the target voice by determining the position of the blank label in the target voice, without considering the related voice before and after, and the attention model is based on determining the key information of the target voice to determine the effective voice in the target voice Partly, it depends on the relationship between voices.

It should be noted that in the embodiment of the present application, the timing features are respectively input to the CTC loss function and the attention model, and the outputs of the two are combined to achieve separation and alignment of the timing features of the target speech.

It can be understood that after separating and aligning the timing features of the target speech, the timing characteristics of the target speech are split and used to match multiple words or phrases to obtain the text output corresponding to the first decoding sub-model.

In addition, the embodiment of the present application also includes a second decoding sub-model, which is used to predict the text corresponding to the next frame of target speech based on the generated text. It should be noted that, since the target voice to be recognized is human voice, the recognized text has front-to-back logic, and the recognition of the target voice in this application is implemented sequentially, so the contextual connection between texts can be used. Predict the characters that have not been recognized based on the characters that have been recognized.

Specifically, the second decoding sub-model includes a text context matching unit, and the text context matching unit includes a cyclic neural network language model. The cyclic neural network language model is trained through a large amount of text content, and the trained cyclic neural network language model can be based on the input Above the text content, predict the following text content.

Step S104: Generate text corresponding to the target voice to be recognized according to the output of the voice recognition model.

In summary, the voice recognition method provided by the embodiments of the present application obtains the target voice to be recognized, and extracts the waveform characteristics and pitch characteristics corresponding to each frame of the target voice. The waveform characteristics and pitch characteristics corresponding to each frame of target speech are sequentially input into the trained speech recognition model. Among them, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding sub-model. The encoding sub-model includes a convolutional neural network and a two-way long and short-term memory network. The first decoding sub-model includes a speech-text matching unit. The second decoding sub-model includes a text context matching unit, and the speech-text matching unit includes a CTC loss function and an attention model. According to the output of the voice recognition model, the text corresponding to the target voice to be recognized is generated. Thus, the combination of voice-text matching and text context matching improves the accuracy of speech recognition.

It should be understood that the speech recognition model needs to be pre-trained before it can be used for speech recognition. The speech recognition model provided in the embodiment of the present application is trained through the following steps:

Step S11: Obtain the reference voice and the reference text corresponding to the reference voice.

Among them, the reference voice and the reference text corresponding to the reference voice are standard information used to train the voice recognition model, and the reference text corresponding to the reference voice is the correct recognition result of the reference voice.

Step S12: Extract the waveform feature and pitch feature corresponding to each frame of reference speech.

In the training process, the reference speech is first processed into frames, and then the waveform characteristics and pitch characteristics corresponding to each frame of the reference speech are extracted.

Step S13: Input the waveform feature and pitch feature corresponding to a frame of reference speech into the convolutional neural network to obtain the convolutional feature of the reference speech.

Among them, the convolutional neural network includes 4 convolutional layers and 2 maximum pooling layers.

Specifically, the waveform feature and the pitch feature are one of the voice features in speech recognition. Among them, before extracting waveform features and pitch features, it is generally necessary to preprocess the target speech. Specifically, the input target voice is pre-emphasized first, and the high-frequency part of the voice signal is improved by using a high-pass filter to make the frequency spectrum smoother, and then the pre-emphasized target voice is framed and windowed, thereby The non-stationary speech signal is transformed into a short-term stable signal, and then the end point is detected to distinguish the speech and noise, and extract the effective speech part.

Among them, the waveform feature is extracted through the following steps: the pre-processed target speech is subjected to fast Fourier change, thereby converting the speech signal in the time domain into an energy spectrum in the frequency domain for analysis, and then passing the energy spectrum through a set of mel The triangular filter bank of the scale highlights the formant characteristics of the speech, and then the logarithmic energy output by each filter bank is calculated to obtain the waveform characteristics corresponding to the target speech.

The tonal feature is extracted by the following steps: the preprocessed target speech, the voice with a frequency between 50-450 Hz is extracted to obtain the tonal feature corresponding to the target speech. It should be noted that the human tone is between 50-450Hz, which is quite different from the tone of the environmental background sound. By extracting the voice between 50-450Hz, it is possible to filter out a large part of the silent voice segment and improve Subsequent speech recognition efficiency.

In step S14, the convolutional features corresponding to each frame of the reference voice are sequentially input into the bidirectional long and short-term memory network to obtain the time sequence features of the reference voice.

In step S15, the time sequence characteristics of the reference voice are input into the CTC loss function and the attention model respectively, and the text corresponding to each frame of the reference voice is sequentially output.

Step S16, input the text corresponding to the output reference voice into the text context matching unit after training.

Step S17, according to the output of the voice-text matching unit, the parameters of the text context matching unit and the reference text are trained on the parameters of the convolutional neural network, the bidirectional long-short-term memory network, the voice-text matching unit, and the text context matching unit.

Based on the foregoing description, it can be known that the waveform characteristics and pitch characteristics corresponding to the reference voice can be used to distinguish different reference voices. Through the convolutional neural network and the two-way long and short-term memory network, the features can be abstracted and the features corresponding to different reference voices can be highlighted. Use the difference between the features corresponding to different reference voices. Through the CTC loss function and the attention model, the text corresponding to the reference voice is determined.

Compare the text corresponding to the reference voice with the reference text, adjust the text context matching unit, and the reference text, and compare the parameters of the convolutional neural network, two-way long and short-term memory network, voice-text matching unit, and text context matching unit to complete the alignment. Training of speech recognition model.

In addition, the text context matching unit includes a recurrent neural network language model, and the text context matching unit is trained through the following steps:

Step S21: Obtain the reference text.

In step S22, the reference text is sequentially input into the cyclic neural network language model.

Step S23, training the parameters of the recurrent neural network language model according to the context of the reference text.

It should be noted that the cyclic neural network language model can predict the following text of the reference text based on the upper part of the reference text, and train the parameters of the cyclic neural network language model according to the predicted result and the real result.

Since the trained text context matching unit needs to be used when training the speech recognition model, the recurrent neural network language model needs to be trained separately. The reference text used can be the reference text corresponding to the reference speech used to train the speech recognition model. It may also be other reference texts, which are not limited in the embodiments of the present application.

It can be understood that the speech recognition model provided by the embodiment of the present application includes a first decoding sub-model and a second decoding sub-model, which are used to decode the output of the two-way long and short-term memory network to determine the text corresponding to the target speech.

It should be noted that the first decoding sub-model provided by the embodiments of the present application can determine the text corresponding to the target speech according to the time sequence characteristics of the target speech, and the second decoding sub-model can determine the following text of the target speech according to the recognized text. Predicting and combining the output of the two can improve the accuracy of speech recognition. However, the influence of the first decoding sub-model and the second decoding sub-model on the final recognition result is different, which can be specifically implemented by weighting. In other words, the parameters of the speech-text matching unit include the first weight corresponding to the speech-text matching unit, and the parameters of the text context matching unit include the second weight corresponding to the text context matching unit.

The greater the first weight, the greater the influence of the recognition result of the first decoding sub-model on the recognition result of the speech recognition model, the greater the second weight, the greater the influence of the recognition result of the second decoding sub-model on the recognition result of the speech recognition model The greater the influence.

Furthermore, the speech-text matching unit includes a CTC loss function and an attention model. The CTC loss function and the attention model recognize the timing characteristics of the target speech from different focuses. Combining the output of the two can improve the timing characteristics. Accuracy of recognition. However, the CTC loss function and the attention model have different influences on the recognition results of time series features, and they can also be implemented by weights. Specifically, the parameters of the speech-text matching unit also include the third weight of the CTC loss function and the fourth weight corresponding to the attention model.

The greater the third weight, the greater the influence of the recognition result of the CTC loss function on the recognition result of the time series feature, and the greater the fourth weight, the greater the influence of the recognition result of the attention model on the recognition result of the time series feature.

In order to more clearly explain the voice recognition method provided by the embodiment of the present application, an example is given below. FIG. 2 is a flowchart of an example of the voice recognition method provided by the embodiment of the present application. As shown in Figure 2, the target voice to be recognized is divided into frames to obtain N frames of target voice. Input N frames of target speech into the convolutional neural network respectively to obtain the convolutional features corresponding to N frames of target speech, and input the convolutional features corresponding to N frames of target speech into the bidirectional long and short-term memory network to obtain the timing characteristics corresponding to N frames of target speech .

The timing features corresponding to the N frames of target speech are input into the CTC loss function and the attention model respectively, and the cyclic neural network speech model is used to predict the text.

For example, for the second text, the CTC loss function determines the most likely text A1 based on the time series features, the attention model determines the most likely text A2 based on the time series features, and the cyclic neural network speech model based on the first text, The most likely text is A3. Combining the third weight corresponding to the CTC loss function, the fourth weight corresponding to the attention model, the first weight corresponding to the speech-text matching unit, and the second weight corresponding to the text context matching unit, determine the second text.

In order to implement the above-mentioned embodiment, the embodiment of the present application also proposes a voice recognition device. FIG. 3 is a schematic structural diagram of a voice recognition device provided by an embodiment of this application. As shown in FIG. 3, the device includes: The acquisition module 210, the first extraction module 220, the first input module 230, and the generation module 240.

The first acquiring module 210 is used to acquire the target voice to be recognized.

The first extraction module 220 is used for extracting waveform characteristics and pitch characteristics corresponding to each frame of target speech.

The first input module 230 is configured to sequentially input the waveform characteristics and pitch characteristics corresponding to each frame of target speech into the trained speech recognition model.

Among them, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding sub-model. The encoding sub-model includes a convolutional neural network and a two-way long and short-term memory network. The first decoding sub-model includes a speech-text matching unit. The second decoding sub-model includes a text context matching unit, and the speech-text matching unit includes a CTC loss function and an attention model.

The generating module 240 is configured to generate text corresponding to the target voice to be recognized according to the output of the voice recognition model.

Further, in order to train the voice recognition model, the device further includes: a second obtaining module 310, configured to obtain the reference voice and the reference text corresponding to the reference voice. The second extraction module 320 is used to extract the waveform characteristics and pitch characteristics corresponding to each frame of reference speech. The second input module 330 is configured to input the waveform characteristics and pitch characteristics corresponding to a frame of reference speech into the convolutional neural network to obtain the convolutional characteristics of the reference speech. Among them, the convolutional neural network includes 4 convolutional layers and 2 maximum pooling layers. The third input module 340 is configured to sequentially input the convolutional features corresponding to each frame of the reference voice into the bidirectional long and short-term memory network to obtain the time sequence features of the reference voice. The fourth input module 350 is configured to input the timing characteristics of the reference voice into the CTC loss function and the attention model respectively, and sequentially output the text corresponding to each frame of the reference voice. The fifth input module 360 is used for inputting the text corresponding to the reference voice that has been output into the trained text context matching unit. The first training module 370 is used for matching the text context unit and the reference text according to the output of the speech-text matching unit, and the convolutional neural network, the bidirectional long-short-term memory network, the speech-text matching unit, and the text context matching unit. Parameters for training.

Further, in order to train the text context matching unit, the text context matching unit includes a cyclic neural network language model, and the device further includes: a third obtaining module 410 for obtaining reference text. The sixth input module 420 is used for sequentially inputting the reference text into the cyclic neural network language model. The second training module 430 is used to train the parameters of the recurrent neural network language model according to the context of the reference text.

It should be noted that the foregoing explanation of the embodiment of the voice recognition method is also applicable to the voice recognition device of this embodiment, and will not be repeated here.

In summary, the voice recognition device provided by the embodiment of the present application obtains the target voice to be recognized, and extracts the waveform characteristics and pitch characteristics corresponding to each frame of the target voice. The waveform characteristics and pitch characteristics corresponding to each frame of target speech are sequentially input into the trained speech recognition model. Among them, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding sub-model. The encoding sub-model includes a convolutional neural network and a two-way long and short-term memory network. The first decoding sub-model includes a speech-text matching unit. The second decoding sub-model includes a text context matching unit, and the speech-text matching unit includes a CTC loss function and an attention model. According to the output of the voice recognition model, the text corresponding to the target voice to be recognized is generated. Thus, the combination of voice-text matching and text context matching improves the accuracy of speech recognition.

Fig. 4 is a schematic structural diagram of an embodiment of a computer device of this application. The computer device may include a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, The voice recognition method provided in the embodiments of the present application can be implemented.

Among them, the above-mentioned computer equipment may be a server, such as a cloud server, or the above-mentioned computer equipment may also be an electronic device, such as a smart phone, a smart watch, a personal computer (Personal Computer; hereinafter referred to as: PC), a notebook computer or a tablet computer, etc. Smart device, this embodiment does not limit the specific form of the above-mentioned computer device.

Fig. 4 shows a block diagram of an exemplary computer device 52 suitable for implementing the embodiments of the present application. The computer device 52 shown in FIG. 4 is only an example, and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

As shown in FIG. 4, the computer device 52 is in the form of a general-purpose computing device. The components of the computer device 52 may include, but are not limited to: one or more processors or processing units 56, a system memory 78, and a bus 58 connecting different system components (including the system memory 78 and the processing unit 56).

The bus 58 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure among multiple bus structures. For example, these architectures include but are not limited to industry standard architecture (Industry Standard Architecture; hereinafter referred to as ISA) bus, Micro Channel Architecture (hereinafter referred to as MAC) bus, enhanced ISA bus, video electronics Standards Association (Video Electronics Standards Association; hereinafter referred to as VESA) local bus and Peripheral Component Interconnection (hereinafter referred to as PCI) bus.

The computer device 52 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 52, including volatile and non-volatile media, removable and non-removable media.

The system memory 78 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (Random Access Memory; hereinafter referred to as RAM) 70 and/or a cache memory 72. The computer device 52 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, the storage system 74 may be used to read and write non-removable, non-volatile magnetic media (not shown in FIG. 4, usually referred to as a "hard drive"). Although not shown in FIG. 4, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and a removable non-volatile optical disk (such as a compact disk read-only memory) can be provided. Disc Read Only Memory; hereinafter referred to as CD-ROM), Digital Video Disc Read Only Memory (hereinafter referred to as DVD-ROM) or other optical media) read and write optical disc drives. In these cases, each drive may be connected to the bus 58 through one or more data media interfaces. The memory 78 may include at least one program product, and the program product has a set of (for example, at least one) program modules, and these program modules are configured to perform the functions of the embodiments of the present application.

A program/utility tool 80 having a set of (at least one) program module 82 may be stored in, for example, the memory 78. Such program module 82 includes, but is not limited to, an operating system, one or more application programs, and other programs Modules and program data, each of these examples or some combination may include the realization of a network environment. The program module 82 usually executes the functions and/or methods in the embodiments described in this application.

The computer device 52 may also communicate with one or more external devices 54 (such as a keyboard, pointing device, display 64, etc.), and may also communicate with one or more devices that enable a user to interact with the computer device 52, and/or communicate with Any device (such as a network card, modem, etc.) that enables the computer device 52 to communicate with one or more other computing devices. This communication can be performed through an input/output (I/O) interface 62. In addition, the computer device 52 can also communicate with one or more networks (such as a local area network (Local Area Network; hereinafter referred to as: LAN), a wide area network (hereinafter referred to as WAN)) and/or a public network, such as the Internet, through the network adapter 60. ) Communication. As shown in FIG. 4, the network adapter 60 communicates with other modules of the computer device 52 through the bus 58. It should be understood that although not shown in FIG. 4, other hardware and/or software modules can be used in conjunction with the computer device 52, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tapes Drives and data backup storage systems, etc.

The processing unit 56 executes various functional applications and data processing by running programs stored in the system memory 78, such as implementing the voice recognition method provided in the embodiments of the present application.

The embodiment of the present application also provides a computer non-volatile readable storage medium on which a computer program is stored, and the above-mentioned computer program can implement the voice recognition method provided by the embodiment of the present application when the computer program is executed by a processor.

The above-mentioned non-volatile computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer non-volatile readable medium may be a computer readable signal medium or a computer readable storage medium. The computer non-volatile readable storage medium may be, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above, for example. More specific examples (non-exhaustive list) of computer non-volatile readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (Read Only Memory; hereinafter referred to as ROM), Erasable Programmable Read Only Memory (Erasable Programmable Read Only Memory; hereinafter referred to as EPROM) or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage Components, magnetic storage devices, or any suitable combination of the above. In this document, a computer non-volatile readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

The computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and computer-readable program code is carried therein. This propagated data signal can take many forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium. The computer-readable medium can be sent, propagated, or transmitted for use by or with the instruction execution system, apparatus, or device. Combined program.

The program code contained on the computer-readable medium can be transmitted by any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

The computer program code used to perform the operations of this application can be written in one or more programming languages or a combination thereof. The programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional procedural programming languages. Programming language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including local area network (Local Area Network; hereinafter referred to as: LAN) or Wide Area Network (hereinafter referred to as: WAN), or Connect to an external computer (for example, use an Internet service provider to connect via the Internet).

In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean specific features described in conjunction with the embodiment or example , The structure, materials, or characteristics are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples and the features of the different embodiments or examples described in this specification without contradicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless specifically defined otherwise.

Any process or method description described in the flowchart or described in other ways herein can be understood as a module, segment or part of code that includes one or more executable instructions for implementing custom logic functions or steps of the process , And the scope of the preferred embodiments of the present application includes additional implementations, which may not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. This should It is understood by those skilled in the art to which the embodiments of the present application belong.

Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determination" or "in response to detection". Similarly, depending on the context, the phrase "if determined" or "if detected (statement or event)" can be interpreted as "when determined" or "in response to determination" or "when detected (statement or event) )" or "in response to detection (statement or event)".

It should be noted that the terminals involved in the embodiments of the present application may include, but are not limited to, personal computers (Personal Computer; hereinafter referred to as PC), Personal Digital Assistants (Personal Digital Assistant; hereinafter referred to as PDA), wireless handheld devices, and tablets Computer (Tablet Computer), mobile phone, MP3 player, MP4 player, etc.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional units.

The above-mentioned integrated unit implemented in the form of a software functional unit may be stored in a computer non-volatile readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (Processor) execute part of the steps of the methods in the various embodiments of the present application . The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as ROM), random access memory (Random Access Memory; hereinafter referred to as RAM), magnetic disks or optical disks, etc. A medium that can store program codes.

The above are only preferred embodiments of this application, and are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection of this application. Within range.

Claims (20)

A speech recognition method, characterized in that the method includes: Obtain the target voice to be recognized; Extracting waveform characteristics and pitch characteristics corresponding to the target speech in each frame; The waveform feature and the pitch feature corresponding to each frame of the target speech are sequentially input into the trained speech recognition model; wherein, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding A sub-model, the encoding sub-model includes a convolutional neural network and a bidirectional long short-term memory network, the first decoding sub-model includes a speech-text matching unit, the second decoding sub-model includes a text context matching unit, the speech -The text matching unit includes CTC loss function and attention model; According to the output of the voice recognition model, the text corresponding to the target voice to be recognized is generated. The method of claim 1, wherein the speech recognition model is trained through the following steps: Acquiring a reference voice and a reference text corresponding to the reference voice; Extracting the waveform feature and the pitch feature corresponding to the reference voice in each frame; The waveform feature and the pitch feature corresponding to one frame of the reference voice are input to the convolutional neural network to obtain the convolutional feature of the reference voice; wherein, the convolutional neural network includes 4 layers of convolution Layer and 2 maximum pooling layer; Sequentially input the convolution features corresponding to the reference speech in each frame to the two-way long and short-term memory network, so as to obtain the time sequence characteristics of the reference speech; Inputting the timing characteristics of the reference voice into the CTC loss function and the attention model respectively, and sequentially outputting the text corresponding to each frame of the reference voice; Input the text corresponding to the reference voice that has been output into the trained text context matching unit; According to the output of the speech-text matching unit, for the text context matching unit and the reference text, the convolutional neural network, the two-way long-short-term memory network, the speech-text matching unit, and the reference text The parameters of the text context matching unit are trained. 3. The method of claim 2, wherein the text context matching unit comprises a cyclic neural network language model, and the text context matching unit is trained through the following steps: Get reference text; Sequentially inputting the reference text into the recurrent neural network language model; Training the parameters of the recurrent neural network language model according to the context of the reference text. The method according to claim 3, wherein the parameters of the speech-text matching unit include a first weight corresponding to the speech-text matching unit, and the parameters of the text context matching unit include all The second weight corresponding to the text context matching unit. The method of claim 4, wherein the parameters of the speech-text matching unit further include a third weight corresponding to the CTC loss function and a fourth weight corresponding to the attention model. A speech recognition device, characterized in that the device includes: The first acquiring module is used to acquire the target voice to be recognized; The first extraction module is used to extract the waveform characteristics and pitch characteristics corresponding to the target speech in each frame; The first input module is used to input the waveform feature and the pitch feature corresponding to each frame of the target speech into the trained speech recognition model sequentially; wherein, the speech recognition model includes an encoding sub-model, and the first A decoding sub-model and a second decoding sub-model, the coding sub-model includes a convolutional neural network and a bidirectional long-short-term memory network, the first decoding sub-model includes a speech-text matching unit, and the second decoding sub-model includes a text A context matching unit, where the speech-text matching unit includes a CTC loss function and an attention model; The generating module is used to generate the text corresponding to the target voice to be recognized according to the output of the voice recognition model. The device according to claim 6, wherein the device further comprises: The second acquiring module is configured to acquire a reference voice and a reference text corresponding to the reference voice; The second extraction module is configured to extract the waveform feature and the pitch feature corresponding to the reference voice in each frame; The second input module is configured to input the waveform feature and the pitch feature corresponding to one frame of the reference voice into the convolutional neural network to obtain the convolution feature of the reference voice; wherein, the convolution The neural network includes 4 convolutional layers and 2 maximum pooling layers; A third input module, configured to sequentially input the convolutional features corresponding to the reference voice in each frame into the bidirectional long-term and short-term memory network to obtain the timing features of the reference voice; A fourth input module, configured to input the timing characteristics of the reference voice into the CTC loss function and the attention model, respectively, to sequentially output the text corresponding to each frame of the reference voice; A fifth input module, configured to input the text corresponding to the reference voice that has been output into the trained text context matching unit; The first training module is configured to perform, according to the output of the speech-text matching unit, to the text context matching unit, and the reference text, to the convolutional neural network, the bidirectional long short-term memory network, and the The parameters of the speech-text matching unit and the text context matching unit are trained. 8. The device according to claim 7, wherein the text context matching unit comprises a cyclic neural network language model, and the device further comprises: The third obtaining module is used to obtain the reference text; A sixth input module, configured to sequentially input the reference text into the cyclic neural network language model; The second training module is used to train the parameters of the recurrent neural network language model according to the context of the reference text. The device of claim 8, wherein the parameter of the speech-text matching unit includes a first weight corresponding to the speech-text matching unit, and the parameter of the text context matching unit includes all The second weight corresponding to the text context matching unit. 9. The device of claim 9, wherein the parameters of the speech-text matching unit further comprise a third weight corresponding to the CTC loss function, and a fourth weight corresponding to the attention model. A computer non-volatile readable storage medium, wherein the computer non-volatile readable storage medium includes a stored program, wherein the computer non-volatile readable storage medium is controlled when the program is running Perform the following steps on the device: Obtain the target voice to be recognized; Extracting waveform characteristics and pitch characteristics corresponding to the target speech in each frame; The waveform feature and the pitch feature corresponding to each frame of the target speech are sequentially input into the trained speech recognition model; wherein, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding A sub-model, the encoding sub-model includes a convolutional neural network and a bidirectional long short-term memory network, the first decoding sub-model includes a speech-text matching unit, the second decoding sub-model includes a text context matching unit, the speech -The text matching unit includes CTC loss function and attention model; According to the output of the voice recognition model, the text corresponding to the target voice to be recognized is generated. 11. The computer non-volatile readable storage medium according to claim 11, wherein when the program is running, controlling the device where the computer non-volatile readable storage medium is located further executes the following steps: Acquiring a reference voice and a reference text corresponding to the reference voice; Extracting the waveform feature and the pitch feature corresponding to the reference voice in each frame; The waveform feature and the pitch feature corresponding to one frame of the reference voice are input to the convolutional neural network to obtain the convolutional feature of the reference voice; wherein, the convolutional neural network includes 4 layers of convolution Layer and 2 maximum pooling layer; Sequentially input the convolution features corresponding to the reference speech in each frame to the two-way long and short-term memory network, so as to obtain the time sequence characteristics of the reference speech; Inputting the timing characteristics of the reference voice into the CTC loss function and the attention model respectively, and sequentially outputting the text corresponding to each frame of the reference voice; Input the text corresponding to the reference voice that has been output into the trained text context matching unit; According to the output of the speech-text matching unit, for the text context matching unit and the reference text, the convolutional neural network, the two-way long-short-term memory network, the speech-text matching unit, and the reference text The parameters of the text context matching unit are trained. The computer non-volatile readable storage medium according to claim 12, wherein, when the program is running, controlling the device where the computer non-volatile readable storage medium is located further executes the following steps: Get reference text; Sequentially inputting the reference text into the recurrent neural network language model; Training the parameters of the recurrent neural network language model according to the context of the reference text. The computer non-volatile readable storage medium according to claim 13, wherein the parameter of the speech-text matching unit includes a first weight corresponding to the speech-text matching unit, and the text context The parameter of the matching unit includes a second weight corresponding to the text context matching unit. The computer non-volatile readable storage medium according to claim 14, wherein the parameters of the speech-text matching unit include a first weight corresponding to the speech-text matching unit, and the text context The parameter of the matching unit includes a second weight corresponding to the text context matching unit. A computer device includes a memory and a processor, the memory is used to store information including program instructions, the processor is used to control the execution of the program instructions, and the feature is that the program instructions are implemented when the program instructions are loaded and executed by the processor The following steps: Obtain the target voice to be recognized; Extracting waveform characteristics and pitch characteristics corresponding to the target speech in each frame; The waveform feature and the pitch feature corresponding to each frame of the target speech are sequentially input into the trained speech recognition model; wherein, the speech recognition model includes an encoding sub-model, a first decoding sub-model and a second decoding A sub-model, the encoding sub-model includes a convolutional neural network and a bidirectional long short-term memory network, the first decoding sub-model includes a speech-text matching unit, the second decoding sub-model includes a text context matching unit, the speech -The text matching unit includes CTC loss function and attention model; According to the output of the voice recognition model, the text corresponding to the target voice to be recognized is generated. The computer device according to claim 16, wherein the following steps are further implemented when the program instructions are loaded and executed by the processor: Acquiring a reference voice and a reference text corresponding to the reference voice; Extracting the waveform feature and the pitch feature corresponding to the reference voice in each frame; The waveform feature and the pitch feature corresponding to one frame of the reference voice are input to the convolutional neural network to obtain the convolutional feature of the reference voice; wherein, the convolutional neural network includes 4 layers of convolution Layer and 2 maximum pooling layer; Sequentially input the convolution features corresponding to the reference speech in each frame to the two-way long and short-term memory network, so as to obtain the time sequence characteristics of the reference speech; Inputting the timing characteristics of the reference voice into the CTC loss function and the attention model respectively, and sequentially outputting the text corresponding to each frame of the reference voice; Input the text corresponding to the reference voice that has been output into the trained text context matching unit; According to the output of the speech-text matching unit, for the text context matching unit and the reference text, the convolutional neural network, the two-way long-short-term memory network, the speech-text matching unit, and the reference text The parameters of the text context matching unit are trained. The computer device according to claim 17, wherein the program instructions further implement the following steps when being loaded and executed by the processor: Get reference text; Sequentially inputting the reference text into the recurrent neural network language model; Training the parameters of the recurrent neural network language model according to the context of the reference text. The computer device according to claim 18, wherein the parameter of the speech-text matching unit comprises a first weight corresponding to the speech-text matching unit, and the parameter of the text context matching unit comprises The second weight corresponding to the text context matching unit. 19. The computer device of claim 19, wherein the parameters of the speech-text matching unit further comprise a third weight corresponding to the CTC loss function, and a fourth weight corresponding to the attention model.

Images