CN117636843A - Speech recognition method, device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the application provides a voice recognition method, a voice recognition device, an electronic device and a storage medium. The embodiment of the application provides a multi-task voice recognition model comprising a plurality of voice recognition models, and simultaneously integrates a plurality of loss functions to jointly enhance the voice recognition precision of the multi-task voice recognition model.

Description

Speech recognition method, device, electronic equipment and storage medium

Technical field

This application relates to the field of speech processing technology, and specifically to a speech recognition method, device, electronic equipment and storage medium.

Background technique

Speech translation is an automatic translation technology that converts spoken input in one language into translated text in another language. Voice translation technology can help people communicate in real time across language barriers, and is suitable for meetings, business negotiations, travel communication and other scenarios. In addition, voice translation is also used in multilingual service industries to provide a better customer experience. It is also used for language learning, assisting the hearing-impaired in communicating, and is widely adopted by smart devices and voice assistants. The development of speech translation technology will further expand its application scenarios, promote international exchanges and cooperation, and provide people with more convenient cross-language communication methods.

Traditional speech translation usually adopts a cascaded approach, which is achieved by sequentially connecting an automatic speech recognition (ASR) model and a machine translation (MT) model. However, this cascaded approach may suffer from high latency, error propagation, and a large number of parameters. and other problems, and the accuracy of speech recognition is not high.

Contents of the invention

Embodiments of the present application provide a speech recognition method, device, electronic device and storage medium, which can improve the accuracy of speech recognition.

In a first aspect, the speech recognition method provided by this application includes: obtaining an initial multi-task speech recognition model, where the initial multi-task speech recognition model includes a first initial speech recognition model and a second initial speech recognition model;

Utilize the initial multi-task speech recognition model to identify a preset speech sample sequence to obtain a first speech recognition result recognized by the first initial speech recognition model and a second speech recognition result recognized by the second initial speech recognition model;

A first model loss of the first speech recognition result is calculated using a plurality of loss functions in a first loss function set, and a second model loss of the second speech recognition result is calculated using a plurality of loss functions in a second loss function set. model loss;

Update the initial multi-task speech recognition model using the first model loss and the second model loss to obtain an updated multi-task speech recognition model;

Speech recognition using an updated multi-task speech recognition model.

In an optional embodiment, the first initial speech recognition model includes a first speech recognition module and a second speech recognition module, and the second initial speech recognition model includes a third speech recognition module and a fourth speech recognition module. ;

and using the initial multi-task speech recognition model to identify a preset speech sample sequence to obtain a first speech recognition result corresponding to the first initial speech recognition model and a second speech recognition result corresponding to the second initial speech recognition model. ,include:

Use the first speech recognition module to identify the preset speech sample sequence and obtain the speech transcription text;

Use the second speech recognition module to identify the preset speech sample sequence to obtain the first translated text;

Use the third speech recognition module to identify the preset speech sample sequence to obtain a second translated text;

Using the fourth speech recognition module to recognize the speech transcription text, obtain a third translated text;

Wherein, the first speech recognition result includes the speech transcription text and the first translated text, and the second speech recognition result includes the second translated text and the third translated text.

In an optional embodiment, the first loss function set includes a first loss function, a second loss function, a third loss function and a fourth loss function;

The method of using the first set of loss functions to calculate the first model loss of the first speech recognition result, and using the second set of loss functions to calculate the second model loss of the second speech recognition result, includes:

Using the first loss function to calculate a third model loss of the speech transcription text;

Calculating a fourth model loss of the first translated text using the second loss function;

Using the first initial speech recognition model as a teacher model and the second initial speech recognition model as a student model, the third loss function is used to calculate the first initial speech recognition model and the second initial speech recognition model. fifth model loss between;

using the fourth loss function to calculate a sixth model loss between the speech transcription text and the first translated text;

Wherein, the first model loss includes the third model loss, the fourth model loss, the fifth model loss and the sixth model loss.

In an optional embodiment, the second loss function set includes a fifth loss function, a sixth loss function, a seventh loss function and an eighth loss function;

The method of using a first set of loss functions to calculate a first model loss of the first speech recognition result, and using a second set of loss functions to calculate a second model loss of the second speech recognition result, further includes:

Calculating a seventh model loss of the second translated text using the fifth loss function;

Calculating an eighth model loss of the third translated text using the sixth loss function;

Taking the first initial speech recognition model as the student model and the second initial speech recognition model as the teacher model, the seventh loss function is used to calculate the first initial speech recognition model and the second initial speech recognition model. ninth model loss between;

Calculating a tenth model loss between the second translated text and the third translated text using the eighth loss function;

Wherein, the second model loss includes the seventh model loss, the eighth model loss, the ninth model loss and the tenth model loss.

In an optional embodiment, updating the initial multi-task speech recognition model using the first model loss and the second model loss to obtain an updated multi-task speech recognition model includes:

Calculate a first gradient parameter in the first initial speech recognition model based on the first model loss;

Calculate a second gradient parameter in the second initial speech recognition model based on the second model loss;

Simultaneously update the model parameters of the initial multi-task speech recognition model according to the first gradient parameter and the second gradient parameter to obtain an updated multi-task speech recognition model.

In an optional embodiment, using the updated multi-task speech recognition model for speech recognition includes:

Determine whether the model loss of the updated multi-task speech recognition model meets the target loss standard;

If the model loss of the updated multi-task speech recognition model meets the target loss standard, the updated multi-task speech recognition model is used for speech recognition.

In an optional embodiment, the updated multi-task speech recognition model includes a first speech recognition model and a second speech recognition model;

If the model loss of the updated multi-task speech recognition model meets the target model loss standard, using the updated multi-task speech recognition model for speech recognition includes:

Obtain the target speech sample to be recognized;

Using the second speech recognition model to identify the target speech sample, obtain a target speech representation sequence corresponding to the target speech sample;

using the second speech recognition model to determine the predicted distribution of the target speech text corresponding to the target speech representation sequence;

The predicted distribution of the target speech text is mapped to obtain the target translation text of the target speech sample.

In the second aspect, the speech recognition device provided by this application includes:

A model acquisition module, configured to acquire an initial multi-task speech recognition model, where the initial multi-task speech recognition model includes a first initial speech recognition model and a second initial speech recognition model;

A first speech recognition module, configured to use the initial multi-task speech recognition model to recognize a preset speech sample sequence, and obtain the first speech recognition result recognized by the first initial speech recognition model and the second initial speech recognition model recognition The second speech recognition result;

A model loss calculation module, configured to calculate the first model loss of the first speech recognition result using a plurality of loss functions in a first loss function set, and to calculate the first model loss of the first speech recognition result using a plurality of loss functions in a second loss function set. Second model loss for second speech recognition results;

A model update module, configured to update the initial multi-task speech recognition model using the first model loss and the second model loss to obtain an updated multi-task speech recognition model;

The second speech recognition module is used for speech recognition using the updated multi-task speech recognition model.

In a third aspect, the electronic device provided by this application includes a memory and a processor. The memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the speech recognition method provided by this application.

In the fourth aspect, the computer-readable storage medium provided by this application stores a plurality of instructions, which are suitable for loading by the processor to implement the steps of the speech recognition method provided by this application.

Embodiments of the present application provide a speech recognition method, device, electronic device and storage medium. The speech recognition method provided by the present application uses an initial multi-task speech recognition model including a first initial speech recognition model and a second initial speech recognition model. Model training, while using multiple different loss functions to calculate model losses during the training process, using multiple model losses to simultaneously update the model parameters in the initial multi-task speech recognition model, and obtain the post-training multi-task speech recognition model. Embodiments of the present application provide a multi-task speech recognition model including multiple speech recognition models, and simultaneously integrate multiple loss functions to jointly enhance the speech recognition accuracy of the multi-task speech recognition model.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of a speech recognition system provided by an embodiment of the present application;

Figure 2 is a schematic flow chart of one embodiment of the speech recognition method provided by the embodiment of the present application;

Figure 3 is a schematic flowchart of an embodiment of obtaining the first speech recognition result and the second speech recognition result provided by the embodiment of the present application;

Figure 4 is a schematic architectural diagram of an embodiment of the initial speech recognition model provided by the embodiment of this application;

Figure 5 is a schematic flow chart of an embodiment of calculating the first model loss and the second model loss provided by the embodiment of the present application;

Figure 6 is a schematic flowchart of an embodiment of obtaining an updated multi-task speech recognition model provided by an embodiment of the present application;

Figure 7 is a schematic flowchart of an embodiment of speech recognition provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of a speech recognition model training device provided by an embodiment of the present application;

Figure 9 shows a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

It should be noted that the principles of this application are implemented in an appropriate computing environment to illustrate. The following description is based on the illustrated specific embodiments of the present application, and should not be regarded as limiting other specific embodiments of the present application that are not described in detail here.

In the following description of this application, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

In the following description of this application, the terms "first\second\third" involved are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understandable that "first\second\third" Where permitted, the specific order or sequence may be interchanged so that the embodiments of the application described herein can be practiced in an order other than that illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

In order to improve the effect of speech recognition, embodiments of the present application provide a speech recognition method, a speech recognition device, an electronic device, a computer-readable storage medium, and a computer program product. The speech recognition method may be executed by a training device of a speech recognition model, or by an electronic device integrating the training device of a speech recognition model.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of this application.

Please refer to Figure 1. The present application also provides a speech recognition system. As shown in Figure 1, the speech recognition system includes an electronic device 100. The electronic device 100 integrates a training device and/or speech recognition of the speech recognition model provided by the present application. device.

The electronic device 100 may be any device equipped with a processor and capable of processing, such as a mobile electronic device equipped with a processor such as a smartphone, tablet computer, handheld computer, notebook computer, smart speaker, or a desktop computer, television, Servers, industrial equipment and other fixed electronic equipment equipped with processors.

In addition, as shown in Figure 1, the speech recognition system may also include a memory 200 for storing speech samples to be recognized.

In this embodiment of the present application, the memory 200 may be a cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a system that uses a cluster Functions such as applications, grid technology, and distributed storage file systems bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to jointly provide external data storage. and a storage system for business access functions.

Currently, the storage method of the storage system is to create logical volumes. When creating logical volumes, physical storage space is allocated to each logical volume. The physical storage space may be composed of disks of a certain storage device or several storage devices. The client stores data on a certain logical volume, that is, the data is stored on the file system. The file system divides the data into many parts. Each part is an object. The object not only contains data but also contains data identification (ID, ID entity). and other additional information, the file system writes each object to the physical storage space of the logical volume separately, and the file system records the storage location information of each object, so that when the client requests to access data, the file system can according to each The storage location information of the object allows the client to access the data.

The process of the storage system allocating physical storage space to a logical volume, specifically based on the capacity estimation of the objects stored in the logical volume (this estimation often has a large margin relative to the actual capacity of the objects to be stored) and independent redundant disks The group of RAID (Redundant Array of Independent Disk) divides the physical storage space into stripes in advance. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

It should be noted that the scene diagram of the speech recognition system shown in Figure 1 is only an example. The speech recognition system and scenes described in the embodiments of the present application are for the purpose of more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a Limitations of the technical solutions provided by the embodiments of the present application. Persons of ordinary skill in the art will know that with the evolution of speech recognition systems and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

Each is explained in detail below. It should be noted that the serial numbers of the following embodiments are not used to limit the preferred order of the embodiments.

Please refer to Figure 2. Figure 2 is a schematic flow chart of an embodiment of the speech recognition method provided by the embodiment of the present application. As shown in Figure 2, the flow of the speech recognition method provided by the present application is as follows:

201. Obtain the initial multi-task speech recognition model.

As the name suggests, the multi-task model means that there are multiple different task models that perform model training and subsequent recognition processes at the same time. The multi-task model uses multiple different models at the same time, so it can effectively improve the efficiency of model training and recognition, and can also improve the efficiency of model training and recognition to a certain extent. to alleviate the overfitting of the model and improve the generalization ability of the model. This application provides a multi-task speech recognition model that includes multiple different speech recognition models. By training the multi-task speech recognition model, a speech recognition model with better speech recognition effect can be obtained.

Specifically, an initial multi-task speech recognition model needs to be obtained, and the initial multi-task speech recognition model includes a first initial speech recognition model and a second initial speech recognition model. In this embodiment of the present application, it is necessary to train the first initial speech recognition model and the second initial speech recognition model simultaneously to adjust the model parameters of the initial multi-task speech recognition model and realize the training of the initial multi-task speech recognition model. The specific training process will be described in detail in subsequent embodiments and is not limited here.

202. Use the initial multi-task speech recognition model to recognize the preset speech sample sequence, and obtain the first speech recognition result recognized by the first initial speech recognition model and the second speech recognition result recognized by the second initial speech recognition model.

203. Calculate the first model loss of the first speech recognition result using multiple loss functions in the first loss function set, and calculate the second model loss of the second speech recognition result using multiple loss functions in the second loss function set.

In the process of training the initial multi-task speech recognition model, you first need to use the initial multi-task speech recognition model to identify speech samples, mainly to identify a preset speech sample sequence and obtain a speech recognition result. Specifically, it is necessary to use the first initial speech recognition model to recognize the preset sample speech sequence to obtain the first speech recognition result; and to use the second initial sample speech recognition model to recognize the preset sample speech sequence to obtain the second recognition result. . Then calculate the loss of the speech recognition result, that is, calculate the first speech loss corresponding to the first speech recognition result, and the second speech loss corresponding to the second speech recognition result.

In the embodiment of the present application, the first speech loss and the second speech loss need to be calculated separately, and when calculating the first speech loss and the second speech loss, multiple loss functions are used for calculation. Specifically, using multiple loss functions in the first loss function set to calculate the first model loss of the first speech recognition result; and using multiple loss functions in the second loss function set to calculate the second model of the second speech recognition result. loss. Different from the existing technology that only uses a single loss function to calculate the model loss, this application uses multiple loss functions to calculate the model loss of different speech recognition models respectively, which can obtain a more accurate model loss, thereby better optimizing the initial multi-task speech recognition. Model.

204. Use the first model loss and the second model loss to update the initial multi-task speech recognition model to obtain an updated multi-task speech recognition model.

205. Use the updated multi-task speech recognition model for speech recognition.

After the first model loss and the second model loss are respectively calculated, the initial multi-task speech recognition model can be updated using the first model loss and the second model loss at the same time to obtain an updated multi-task speech recognition model. Re-judge whether the updated multi-task speech recognition model meets the model requirements, mainly to judge whether the model loss of the updated multi-task speech recognition model meets the target loss standard, or to judge whether the updated multi-task speech recognition model has converged . If the target loss standard is met, it means that the updated multi-task speech recognition model has met the requirements, and the training can be stopped; and the updated multi-task speech recognition model can be used for speech recognition. If the target loss standard is not met, the model parameters of the multi-task speech recognition model need to be readjusted based on the new model loss to obtain a new multi-task speech recognition model.

Embodiments of the present application provide a speech recognition method that requires the use of an initial multi-task speech recognition model including a first initial speech recognition model and a second initial speech recognition model for model training, and at the same time utilizes multiple different loss functions during the training process. Calculate the model loss, use multiple model losses to simultaneously update the model parameters in the initial multi-task speech recognition model, and obtain the trained multi-task speech recognition model. Embodiments of the present application provide a multi-task speech recognition model including multiple speech recognition models, and simultaneously integrate multiple loss functions to jointly enhance the speech recognition accuracy of the multi-task speech recognition model.

As shown in Figure 3, a schematic flow chart of an embodiment of obtaining the first speech recognition result and the second speech recognition result provided by the embodiment of the present application mainly includes the following steps:

301. Use the first speech recognition module to recognize the preset speech sample sequence and obtain the speech transcription text.

302. Use the second speech recognition module to recognize the preset speech sample sequence to obtain the first translation text.

303. Use the third speech recognition module to recognize the preset speech sample sequence and obtain the second translation text.

304. Use the fourth speech recognition module to recognize the speech transcription text and obtain the third translated text.

The first speech recognition result includes a speech transcription text and a first translated text; the second speech recognition result includes a second translated text and a third translated text.

As shown in Figure 4, it is a schematic architectural diagram of an embodiment of the initial speech recognition model provided by the embodiment of the present application; the speech recognition process will be described below in conjunction with Figure 3 and Figure 4. In the embodiment shown in Figure 4, the initial multi-task speech recognition model includes a first initial speech recognition model on the left side of the figure, and a second initial speech recognition model on the right side; the first initial speech recognition model and the second initial speech recognition model. The initial speech recognition models each have multiple different functional modules. For the first initial speech recognition model on the left, it mainly includes a speech encoder, a first shared codec, a first speech recognition module and a second speech recognition module; for the second initial speech recognition model, it mainly includes a speech codec, a second shared codec, a third speech recognition module and a fourth speech recognition module. Among them, the first initial speech recognition module and the second initial speech recognition module share the same speech coder; that is, the model architecture shown in Figure 4 actually includes only one speech coder. The first initial speech recognition model and the second speech recognition model each have a shared codec; the structures of the first shared codec and the second shared codec are the same, but their parameters are different so that they are respectively applicable to the first Initial speech codec and second initial speech codec.

In the above embodiment, the speech encoder mainly includes a speech pre-training model to recognize the speech sequence. The speech encoder can extract features in the speech sequence and compress the length of the speech sequence. Generally speaking, the length of the speech sequence is much longer than the length of the text sequence, which consumes a lot of computing resources; therefore, by inputting the obtained speech sequence into the speech encoder, the speech encoder is used to extract speech features and compress the length of the speech sequence. to reduce the consumption of computing resources. In the embodiment of the present application, the speech encoder not only includes a speech pre-training model, but also can further connect multiple convolutional neural network (Convolutional Neural Networks, CNN) layers in series based on the speech pre-training model, such as two layers CNN layer. Multiple CNN layers connected in series can enhance the nonlinear representation ability of speech features recognized by the speech pre-training model and facilitate subsequent recognition of speech features.

In the foregoing embodiments, the first initial speech recognition model and the second speech recognition model share the same speech coder; on this basis, in other embodiments, the two can also share the same shared codec, that is, multi-tasking Only a speech encoder and a shared codec are included in the speech recognition model. Thereby, the connection between the first speech recognition model and the second speech recognition model is further improved, the accuracy of the multi-task speech recognition model is improved, and the consumption of computing resources is reduced.

The input of the speech encoder provided by the embodiment of the present application is the original speech sequence to be recognized, and the output is a speech representation sequence with acoustic characteristics. Since the first initial speech recognition model and the second initial speech recognition model share the same speech encoder, the speech representation sequence output by the speech encoder will be input into the first initial speech recognition model and the second initial speech recognition model respectively, and the subsequent speech recognition process.

For the first initial speech recognition model, the speech representation sequence output by the speech encoder will continue to be input into the first shared codec. At the same time, combined with the first speech recognition module and the second speech recognition module, the speech transcript text and the speech representation sequence are obtained respectively. First translated text. Taking the speech representation sequence as a language A representation sequence as an example, the first speech recognition module will perform speech transcription of the language A representation sequence to obtain a speech transcription text of language A. The second speech recognition module not only transcribes speech into text, but ultimately obtains the first translated text in language B that is different from language A; that is, the second speech recognition module not only performs language transcription, but also performs language translation. Although the first speech recognition module and the second speech recognition module finally output text, the language of the speech transcription text is the same as the input language, which is language A, and the language of the first translated text is different from the input language, which is language B. language. In this embodiment of the present application, the inputs of the first speech recognition module and the second speech recognition module are the same, but the outputs are different.

For the second initial speech recognition model, it includes not only the speech codec, the second shared codec, but also a text embedding module. Among them, the speech transcription text recognized by the first initial speech recognition model needs to be used as the input of the second initial speech recognition model. Specifically, in the second initial speech recognition model, the speech encoder normally recognizes the preset speech sample sequence and obtains the speech representation sequence; and the speech transcription text will be input into the text embedding module, and the text embedding module will embed the speech transcription text. , into the corresponding mathematical representation; that is, converted into data that can be recognized by the second initial speech recognition model. The specific conversion process may refer to the existing technology and is not limited here.

The second initial speech recognition model also includes a third speech recognition module and a fourth speech recognition module; wherein the shared codec and the third speech recognition module jointly recognize the speech representation sequence to obtain the second translated text; the shared codec and The fourth speech recognition module jointly recognizes the speech transcribed text after text embedding to obtain a third translated text. Taking the speech representation sequence as the representation sequence of language A as an example, in the foregoing embodiment, it is determined that the speech transcription text is the speech transcription text of language A; and the second translation text is the second translation text of language B, that is, the third speech recognition The module not only converts speech into text, but also translates the speech; while the fourth speech recognition module recognizes the transcribed text in language A and obtains the third translated text in language B. That is to say, in the embodiment of the present application, the inputs of the third speech recognition module and the fourth speech recognition module are different, but the final output is the same, which is the translated text of language B.

It should be noted that the outputs of the third speech recognition module and the fourth speech recognition module described in the above embodiment are the same, which means that both output the translation text in language B; it does not mean that the specific content of the translation text actually output is also same. At the same time, the first initial speech recognition model and the second initial speech recognition model share the same speech encoder, which can reduce computing consumption; and the speech transcription text recognized by the first initial speech recognition model is used as the input of the second initial speech recognition model, which can Promote knowledge transfer between different speech recognition models and further reduce model training costs.

In the above embodiment, the initial multi-task speech recognition module is used to recognize the preset speech sample sequence and obtain the recognition results. The recognition results also need to be used to update the initial multi-task speech recognition model to implement training of the initial multi-task speech recognition model. Different from the existing technology that only uses one or two loss functions to train the initial multi-task speech recognition model, this application uses more than two loss functions to train the initial multi-task speech recognition model. Specifically, this application uses a first loss function set including multiple loss functions to train a first initial speech recognition model, and uses a second loss function set including multiple loss functions to train a second initial speech recognition model.

In a specific embodiment, the first loss function set includes a first loss function, a second loss function, a third loss function and a fourth loss function; the second loss function set includes a fifth loss function, a sixth loss function, The seventh loss function and the eighth loss function. As shown in Figure 5, the steps for calculating the first model loss and the second model loss at this time may include:

501. Use the first loss function to calculate the third model loss of the speech transcription text.

502. Use the second loss function to calculate the fourth model loss of the first translated text.

503. Using the first initial speech recognition model as the teacher model and the second initial speech recognition model as the student model, use the third loss function to calculate the fifth model loss between the first initial speech recognition model and the second initial speech recognition model.

504. Use the fourth loss function to calculate the sixth model loss between the speech transcription text and the first translation text.

505. Use the fifth loss function to calculate the seventh model loss of the second translated text.

506. Use the sixth loss function to calculate the eighth model loss of the third translation text.

507. Using the first initial speech recognition model as the student model and the second initial speech recognition model as the teacher model, use the seventh loss function to calculate the ninth model loss between the first initial speech recognition model and the second initial speech recognition model.

508. Use the eighth loss function to calculate the tenth model loss between the second translation text and the third translation text.

Specifically, the initial multi-task speech recognition model provided by this application obtained four outputs, which are the speech transcription text, the first translated text, the second translated text and the third translated text; therefore, it is necessary to calculate the corresponding four output results respectively. Loss. Specifically, the method includes using the first loss function to calculate the third model loss of the speech transcription text, using the second loss function to calculate the fourth model loss of the first translated text, using the fifth loss function to calculate the seventh model loss of the second translated text, and The eighth model loss of the third translated text is calculated using the sixth loss function.

At the same time, because the initial multi-task speech recognition model includes a first initial speech recognition model and a second initial speech recognition model, and the output result of the first initial speech recognition model is used as the input of the second initial speech recognition model; that is, the first initial speech recognition model The model and the second initial speech recognition model influence each other. Therefore, in the embodiment of the present application, it is also necessary to calculate the first initial speech recognition model and the second initial speech recognition model that interact with each other, and the model loss caused during speech recognition. Specifically, the first initial speech recognition model can be the teacher model, the second initial speech recognition model can be the student model, and the third loss function can be used to calculate the third loss between the first initial speech recognition model and the second initial speech recognition model. Five model losses. and using the first initial speech recognition model as the student model, using the second initial speech recognition model as the teacher model, and using the seventh loss function to calculate the ninth model loss between the first initial speech recognition model and the second initial speech recognition model. This application additionally introduces a third loss function and a seventh loss function to calculate: the model loss between the first initial speech recognition model and the second initial speech recognition model. This can take into account the interaction between different models and better train multi-task speech recognition models.

The foregoing embodiments respectively describe the third model loss, the fourth model loss, the fifth model loss, the seventh model loss, the eighth model loss and the ninth model loss; and the model loss calculated in the embodiment of the present application also includes the sixth model loss. model loss and tenth model loss. Specifically, due to the first speech recognition module and the second speech recognition module included in the first initial speech recognition model, the speech transcription text and the first translation text are respectively output, specifically the output texts in different languages. Therefore, the speech transcription text and the first translation text can be in the same speech representation space; and in order to maintain the consistency of the speech transcription text and the first translation text in the semantic representation space, it is also necessary to calculate the difference between the speech transcription text and the first translation text. The sixth model loss between. Generally speaking, the smaller the loss of the sixth model, the better the speech recognition effect of the first speech recognition module and the second speech recognition module, that is, the better the performance of the multi-task speech recognition model. Therefore, in this application, a fourth loss function can be introduced and the first speech recognition module and the second speech recognition module can be regularized in the semantic representation space. The specific calculation process can refer to the existing technology and is not limited here.

Similarly, for the second initial speech recognition model, the outputs of the third speech recognition module and the fourth speech recognition module are both translated texts in the same language, so the outputs of the two share the output space of the same language. In order to maintain the consistency between the second translated text and the third translated text, it is necessary to calculate the tenth model loss between the third speech recognition module and the fourth speech recognition module within the output space of the same language. Specifically, an eighth loss function can be introduced to regularize the third speech recognition module and the fourth speech recognition module within the output space of the same language. The specific calculation process can refer to the existing technology and is not limited here.

It should be noted that in the above embodiments, the specific functions of the third loss function and the seventh loss function may be the same; but the first initial speech recognition model and the second initial speech recognition model are respectively used as teacher models, and the same loss function is used. The calculated model loss is different. In a specific embodiment, the fourth loss function may be a mean square error (MSE) loss function; the eighth loss function may be a divergence (Kullback-Leibler, KL) loss function. Other loss functions can be selected according to actual needs and are not limited here. At the same time, the regularization in the foregoing embodiment is a hard constraint. In other embodiments, soft constraints can also be applied to different speech recognition modules to ensure consistency between translated texts. The specific calculation process can refer to the existing technology and is not limited here.

As shown in Figure 6, a schematic flow chart of an embodiment of obtaining an updated multi-task speech recognition model provided by an embodiment of the present application may include the following steps:

601. Calculate the first gradient parameter in the first initial speech recognition model according to the first model loss.

602. Calculate the second gradient parameter in the second initial speech recognition model according to the second model loss.

603. Simultaneously update the model parameters of the initial multi-task speech recognition model according to the first gradient parameter and the second gradient parameter to obtain an updated multi-task speech recognition model.

In the embodiment of the present application, after the model loss is calculated using the loss function, the model loss can be used to update the initial multi-task speech recognition model. Specifically, this application can respectively use the first model loss to reversely calculate the first gradient parameter in the first initial speech recognition model, and use the second model loss to calculate the second gradient parameter in the second initial speech recognition model; but because The first initial speech recognition model and the second initial speech recognition model share a speech encoder, and the output result of the first initial speech recognition model is used as the input of the second initial speech recognition model. Therefore, in this application, it is necessary to use the first gradient parameter and the second gradient parameter to update the model parameters in the initial multi-task speech recognition model at the same time. The first gradient parameter cannot be used alone to update the first initial speech recognition model, or it cannot be used alone. The second gradient parameters update the second initial speech recognition model. The specific process of calculating the model gradient and updating the model parameters can refer to the existing technology, and is not limited here.

After obtaining the updated multi-task speech recognition model, it is also necessary to determine whether the updated multi-task speech recognition model has met the target speech recognition model standard, for example, to determine whether the updated multi-task speech recognition model has converged. If the updated multi-task speech recognition model has converged, it can be confirmed that the speech recognition model at this time has met the requirements, and the training can be stopped; if the updated multi-task speech recognition model has not converged, you need to continue training the multi-task speech model. , until the final multi-task speech recognition model converges or reaches other speech recognition model standards.

After obtaining the final target multi-task speech recognition model that meets the target speech recognition model standards, the target multi-task speech recognition can be used for speech recognition. As shown in Figure 7, a schematic flow diagram of an embodiment of speech recognition provided by the embodiment of the present application includes:

701. Obtain the target speech sample to be recognized.

702. Use the second speech recognition model to recognize the target speech sample, and obtain the target speech representation sequence corresponding to the target speech sample.

703. Use the second speech recognition model to determine the predicted distribution of the target speech text corresponding to the target speech representation sequence.

704. Map the predicted distribution of the target speech text to obtain the target translation text of the target speech sample.

In the actual speech recognition process, only the trained second speech recognition model is used to identify the target speech sample to be recognized, without using the first speech recognition model to assist the recognition. Specifically, the speech encoder in the second speech sample recognition model can be used to identify the target speech sample and obtain the target speech representation sequence corresponding to the target speech sample; and then the target speech representation sequence is input into the second shared encoder. The second shared codec and the fourth speech recognition module jointly recognize the target speech representation sequence, obtain the predicted distribution of the target speech text corresponding to the target speech representation sequence, and further map the predicted distribution of the target speech text to obtain the target speech sample. Target translation text. Wherein, the target translation text and the target speech sample are in different languages, and the second speech recognition model translates the target speech sample into translation texts in different languages.

In the above embodiments, this application mainly improves the existing multi-task speech recognition model. Therefore, compared with the existing speech recognition model, the multi-task speech recognition model provided by this application has the parameters of the second speech recognition module and The decoding speed is consistent with the parameters and decoding speed of the fourth speech recognition module in the existing speech recognition model; therefore, no additional reasoning overhead will be added, and while the speech recognition accuracy is improved, a lot of computing resources will not be added.

As shown in the table below, the multi-task speech recognition model provided by this application has improved recognition accuracy compared with existing speech recognition models. Among them, MTL-Baseline is the multi-task speech recognition model in the prior art described in the aforementioned embodiments, and ST-MKD-ASR and ST-MKD-MT are respectively the first speech recognition model and the second speech recognition model provided by this application. . Wav2vec 2.0, HuBERT and WavLM are different speech pre-training models for recognizing speech sequences; ConST (SOTA) is the actual speech pre-training module after the modification of Wav2vec 2.0, and CRESS is the actual speech pre-training model after the modification of HuBERT. It can be seen from the table that compared with the MTL-Baseline model in the prior art, the speech recognition accuracy of ST-MKD-ASR and ST-MKD-MT provided by this application has been improved. At the same time, if WavLM is used as a speech pre-training model, the speech recognition accuracy can be further improved.

At the same time, in the embodiment of the present application, the first speech recognition model and the second speech recognition model are respectively used as the teacher model to calculate the model loss. Compared with the existing technology that only calculates the model loss once, it also effectively improves the final multi-task speech quality. Recognition model accuracy. As shown in the table below, MKD is the speech recognition accuracy determined by two-way knowledge distillation, that is, the speech recognition accuracy obtained after calculating the model loss by using the first speech recognition model and the second speech recognition model as the teacher model respectively; while KD is the one-way knowledge distillation The determined speech recognition accuracy is the speech recognition accuracy obtained after calculating the model loss using only the first speech recognition model or the second speech recognition model as the teacher model.

In other embodiments of the present application, additional translated text can also be introduced to assist the speech pre-training model in speech recognition, that is, a higher-accuracy speech pre-training model is obtained to improve the recognition accuracy of speech sequences, thereby improving multi-task speech. Recognition accuracy of the recognition model.

In order to facilitate better implementation of the speech recognition method provided by the embodiment of the present application, the embodiment of the present application also provides a training device for a speech recognition model based on the above speech recognition method. The meaning of the nouns is the same as in the above speech recognition method. For specific implementation details, please refer to the description in the above method embodiment.

Please refer to Figure 8. Figure 8 is a schematic structural diagram of a speech recognition model training device provided by an embodiment of the present application. The speech recognition model training device may include a model acquisition module 801, a first speech recognition module 802, and a model loss calculation module 803. , model update module 804 and second speech recognition module 805, wherein:

The model acquisition module 801 is used to acquire an initial multi-task speech recognition model. The initial multi-task speech recognition model includes a first initial speech recognition model and a second initial speech recognition model.

The first speech recognition module 802 is used to use the initial multi-task speech recognition model to recognize the preset speech sample sequence, and obtain the first speech recognition result recognized by the first initial speech recognition model and the second speech recognition recognized by the second initial speech recognition model. result.

The model loss calculation module 803 is used to calculate the first model loss of the first speech recognition result using multiple loss functions in the first loss function set, and calculate the second speech recognition using multiple loss functions in the second loss function set. The resulting second model loss.

The model update module 804 is used to update the initial multi-task speech recognition model using the first model loss and the second model loss to obtain an updated multi-task speech recognition model.

The second speech recognition module 805 is used to perform speech recognition using the updated multi-task speech recognition model.

Embodiments of the present application provide a speech recognition device that uses an initial multi-task speech recognition model including a first initial speech recognition model and a second initial speech recognition model to perform model training. At the same time, multiple different loss functions are used to calculate the model during the training process. Loss, use multiple model losses to simultaneously update the model parameters in the initial multi-task speech recognition model, and obtain the trained multi-task speech recognition model. Embodiments of the present application provide a multi-task speech recognition model including multiple speech recognition models, and simultaneously integrate multiple loss functions to jointly enhance the speech recognition accuracy of the multi-task speech recognition model.

In some embodiments, the first initial speech recognition model includes a first speech recognition module and a second speech recognition module, and the second initial speech recognition model includes a third speech recognition module and a fourth speech recognition module. The first speech recognition module 802 is mainly used for:

Use the first speech recognition module to identify the preset speech sample sequence and obtain the speech transcription text;

Use the second speech recognition module to recognize the preset speech sample sequence to obtain the first translated text;

Use the third speech recognition module to identify the preset speech sample sequence and obtain the second translated text;

The fourth speech recognition module is used to recognize the speech transcription text and obtain the third translated text.

The first speech recognition result includes a speech transcription text and a first translated text, and the second speech recognition result includes a second translated text and a third translated text.

In some embodiments, the first loss function set includes a first loss function, a second loss function, a third loss function and a fourth loss function. The model loss calculation module 803 can be specifically used for:

Using the first loss function to calculate the third model loss of the speech transcription text; using the second loss function to calculate the fourth model loss of the first translated text;

Using the first initial speech recognition model as the teacher model and the second initial speech recognition model as the student model, use the third loss function to calculate the fifth model loss between the first initial speech recognition model and the second initial speech recognition model;

Using the fourth loss function to calculate the sixth model loss between the speech transcription text and the first translated text;

Among them, the first model loss includes the third model loss, the fourth model loss, the fifth model loss and the sixth model loss.

In some embodiments, the second loss function set includes a fifth loss function, a sixth loss function, a seventh loss function, and an eighth loss function. The model loss calculation module 803 can be specifically used for:

The fifth loss function is used to calculate the seventh model loss of the second translated text; the sixth loss function is used to calculate the eighth model loss of the third translated text;

Using the first initial speech recognition model as the student model and the second initial speech recognition model as the teacher model, use the seventh loss function to calculate the ninth model loss between the first initial speech recognition model and the second initial speech recognition model;

Using the eighth loss function to calculate the tenth model loss between the second translated text and the third translated text;

Among them, the second model loss includes the seventh model loss, the eighth model loss, the ninth model loss and the tenth model loss.

In some embodiments, the model update module 804 may be used to:

Calculate the first gradient parameter in the first initial speech recognition model based on the first model loss;

Calculate second gradient parameters in the second initial speech recognition model based on the second model loss;

The model parameters of the initial multi-task speech recognition model are simultaneously updated according to the first gradient parameter and the second gradient parameter to obtain an updated multi-task speech recognition model.

In some embodiments, the second speech recognition module 805 may be used to:

Determine whether the model loss of the updated multi-task speech recognition model meets the target loss standard;

If the model loss of the updated multi-task speech recognition model meets the target loss standard, the updated multi-task speech recognition model is used for speech recognition.

In some embodiments, the updated multi-task speech recognition model includes a first speech recognition model and a second speech recognition model. The second speech recognition module 805 can be specifically used for:

Obtain the target speech sample to be recognized;

Use the second speech recognition model to identify the target speech sample and obtain the target speech representation sequence corresponding to the target speech sample;

The second speech recognition model is used to determine the predicted distribution of the target speech text corresponding to the target speech representation sequence; the predicted distribution of the target speech text is mapped to obtain the target translation text of the target speech sample.

An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the speech recognition method provided by this embodiment or the steps in the speech recognition method by calling a computer program stored in the memory. .

Please refer to FIG. 9 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include components such as a processor 901 of one or more processing cores, a memory 902 of one or more computer-readable storage media, a power supply 903 and an input unit 904. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 9 does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange different components. in:

The processor 901 is the control center of the electronic device, using various interfaces and lines to connect various parts of the entire electronic device, by running or executing software programs and/or modules stored in the memory 902, and calling software programs stored in the memory 902. Data, perform various functions of electronic devices and process data. Optionally, the processor 901 may include one or more processing cores; optionally, the processor 901 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface and application programs. etc., the modem processor mainly handles wireless communications. It can be understood that the above modem processor may not be integrated into the processor 901.

The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data processing by running the software programs and modules stored in the memory 902 . The memory 902 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the storage data area may store data based on Data created by the use of electronic devices, etc. In addition, memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 with access to the memory 902 .

The electronic device also includes a power supply 903 that supplies power to various components. Optionally, the power supply 903 can be logically connected to the processor 901 through a power management system, thereby realizing functions such as charging, discharging, and power consumption management through the power management system. The power supply 903 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

The electronic device may also include an input unit 904 that may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the electronic device may also include a display unit, an image capture component, etc., which will not be described again here. Specifically, in this embodiment, the processor 901 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 902 according to the following instructions, and the processor 901 will execute the instructions provided by this application. Steps in speech recognition methods, such as:

Obtain an initial multi-task speech recognition model, which includes a first initial speech recognition model and a second initial speech recognition model;

Using the initial multi-task speech recognition model to identify the preset speech sample sequence, obtain the first speech recognition result recognized by the first initial speech recognition model and the second speech recognition result recognized by the second initial speech recognition model;

Calculating a first model loss for the first speech recognition result using a plurality of loss functions in the first loss function set, and calculating a second model loss for the second speech recognition result using a plurality of loss functions in the second loss function set;

Update the initial multi-task speech recognition model using the first model loss and the second model loss to obtain the updated multi-task speech recognition model;

Speech recognition using an updated multi-task speech recognition model.

It should be noted that the electronic device provided by the embodiments of the present application and the speech recognition method in the above embodiments belong to the same concept. The specific implementation process can be found in the above related embodiments and will not be described again here.

The present application also provides a computer-readable storage medium on which a computer program is stored. When the stored computer program is executed on the processor of the electronic device provided by the embodiment of the present application, the processor of the electronic device is caused to execute the present invention. Apply the steps in the speech recognition method provided. The storage medium may be a magnetic disk, an optical disk, a read only memory (Read Only Memory, ROM) or a random access memory (Random Access Memory, RAM), etc.

The present application also provides a computer program product or computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the above speech recognition method.

The speech recognition method, device, electronic equipment and storage medium provided by this application have been introduced in detail above. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for To help understand the methods and core ideas of this application; at the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of this application. In summary, the contents of this specification should not be understood as a limitation on this application.

It should be noted that when the above embodiments of this application are applied to specific products or technologies, the relevant data of the user needs to be obtained from the user's permission or consent, and the collection, use and processing of the relevant data need to comply with the laws and regulations of the relevant countries and regions. Relevant laws, regulations and standards.

Claims (10)

1. A method of speech recognition, the method comprising:

acquiring an initial multitasking voice recognition model, wherein the initial multitasking voice recognition model comprises a first initial voice recognition model and a second initial voice recognition model;

utilizing the initial multitasking voice recognition model to recognize a preset voice sample sequence to obtain a first voice recognition result recognized by the first initial voice recognition model and a second voice recognition result recognized by the second initial voice recognition model;

calculating a first model loss of the first speech recognition result using a plurality of loss functions in a first set of loss functions, and calculating a second model loss of the second speech recognition result using a plurality of loss functions in a second set of loss functions;

updating the initial multi-task speech recognition model by using the first model loss and the second model loss to obtain an updated multi-task speech recognition model;

and performing voice recognition by using the updated multitasking voice recognition model.

2. The method of claim 1, wherein the first initial speech recognition model comprises a first speech recognition module and a second speech recognition module, and wherein the second initial speech recognition model comprises a third speech recognition module and a fourth speech recognition module;

The step of utilizing the initial multitasking speech recognition model to recognize a preset speech sample sequence to obtain a first speech recognition result corresponding to the first initial speech recognition model and a second speech recognition result corresponding to the second initial speech recognition model comprises the following steps:

recognizing the preset voice sample sequence by using the first voice recognition module to obtain a voice transcription text;

identifying the preset voice sample sequence by using the second voice identification module to obtain a first translation text;

identifying the preset voice sample sequence by using the third voice identification module to obtain a second translation text;

recognizing the voice transcribed text by using the fourth voice recognition module to obtain a third translated text;

wherein the first speech recognition result comprises the speech transcribed text and the first translated text, and the second speech recognition result comprises the second translated text and the third translated text.

3. The method according to claim 2, wherein the first set of loss functions includes a first loss function, a second loss function, a third loss function, and a fourth loss function;

The calculating a first model loss of the first speech recognition result using a first set of loss functions and calculating a second model loss of the second speech recognition result using a second set of loss functions, comprising:

calculating a third model penalty for the speech transcribed text using the first penalty function;

calculating a fourth model penalty for the first translated text using the second penalty function;

taking the first initial voice recognition model as a teacher model, the second initial voice recognition model as a student model, and calculating a fifth model loss between the first initial voice recognition model and the second initial voice recognition model by using the third loss function;

calculating a sixth model penalty between the speech transcribed text and the first translated text using the fourth penalty function;

wherein the first model loss comprises the third model loss, a fourth model loss, a fifth model loss, and a sixth model loss.

4. The method according to claim 2, wherein the second set of loss functions includes a fifth loss function, a sixth loss function, a seventh loss function, and an eighth loss function;

The calculating the first model loss of the first speech recognition result using the first set of loss functions and the calculating the second model loss of the second speech recognition result using the second set of loss functions further comprises:

calculating a seventh model penalty for the second translated text using the fifth penalty function;

calculating an eighth model penalty for the third translated text using the sixth penalty function;

taking the first initial voice recognition model as a student model, taking the second initial voice recognition model as a teacher model, and calculating a ninth model loss between the first initial voice recognition model and the second initial voice recognition model by utilizing the seventh loss function;

calculating a tenth model penalty between the second translated text and the third translated text using the eighth penalty function;

wherein the second model loss comprises the seventh model loss, eighth model loss, ninth model loss, and tenth model loss.

5. The method of claim 1, wherein updating the initial multi-tasking speech recognition model using the first model loss and the second model loss results in an updated multi-tasking speech recognition model, comprising:

Calculating a first gradient parameter in the first initial speech recognition model according to the first model loss;

calculating a second gradient parameter in the second initial speech recognition model from the second model loss;

and simultaneously updating the model parameters of the initial multi-task voice recognition model according to the first gradient parameters and the second gradient parameters to obtain an updated multi-task voice recognition model.

6. The method of claim 1, wherein the performing speech recognition using the updated multi-tasking speech recognition model comprises:

judging whether the model loss of the updated multitasking voice recognition model meets the target loss standard or not;

and if the model loss of the updated multi-task voice recognition model meets the target loss standard, performing voice recognition by using the updated multi-task voice recognition model.

7. The method of claim 1, wherein the updated multitasking speech recognition model comprises a first speech recognition model and a second speech recognition model;

if the model loss of the updated multi-task voice recognition model meets the target model loss standard, performing voice recognition by using the updated multi-task voice recognition model, including:

Acquiring a target voice sample to be identified;

identifying the target voice sample by using the second voice identification model to obtain a target voice representation sequence corresponding to the target voice sample;

determining a prediction distribution of target voice texts corresponding to the target voice representation sequences by using the second voice recognition model;

mapping the prediction distribution of the target voice text to obtain the target translation text of the target voice sample.

8. A speech recognition apparatus, comprising:

the model acquisition module is used for acquiring an initial multi-task voice recognition model, wherein the initial multi-task voice recognition model comprises a first initial voice recognition model and a second initial voice recognition model;

the first voice recognition module is used for recognizing a preset voice sample sequence by utilizing the initial multitasking voice recognition model to obtain a first voice recognition result recognized by the first initial voice recognition model and a second voice recognition result recognized by the second initial voice recognition model;

a model loss calculation module for calculating a first model loss of the first speech recognition result using a plurality of loss functions in a first set of loss functions, and calculating a second model loss of the second speech recognition result using a plurality of loss functions in a second set of loss functions;

The model updating module is used for updating the initial multi-task voice recognition model by utilizing the first model loss and the second model loss to obtain an updated multi-task voice recognition model;

and the second voice recognition module is used for carrying out voice recognition by utilizing the updated multitasking voice recognition model.

9. An electronic device comprising a memory storing a computer program and a processor for running the computer program in the memory to perform the steps of the speech recognition method of any one of claims 1 to 7.

10. A computer readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the speech recognition method of any one of claims 1 to 7.