WO2018153200A1 - Hlstm model-based acoustic modeling method and device, and storage medium - Google Patents

Abstract

A highway long short time memory (HLSTM) model-based acoustic modeling method and device, and a storage medium, the method comprising: training a randomly initialized HLSTM model on the basis of a preset function and optimizing a training result (101); carrying out forward calculation on training data by means of the optimized HLSTM model (102); and training a randomly initialized long short time memory (LSTM) model on the basis of the result of the forward calculation and the preset function so as to obtain a model that is an acoustic model of a speech recognition system (103), the HLSTM model and the LSTM model having the same network parameters.

Description

Acoustic modeling method, device and storage medium based on HLSTM model

Cross-reference to related applications

The present application is filed on the basis of the Chinese Patent Application No. PCT Application No.

Technical field

The present disclosure relates to the field of speech recognition technologies, and in particular, to an acoustic modeling method, apparatus, and storage medium based on a Highway Long Short Time Memory (HLSTM) model.

Background technique

In recent years, major vocabulary continuous speech recognition systems have made significant progress. The traditional speech recognition system uses the Hidden Markov Model (HMM) to express the time-varying characteristics of speech signals, and uses the Gaussian Mixture Model (GMM) to model the pronunciation diversity of speech signals. Later, deep learning techniques were introduced into the field of speech recognition research, which significantly improved the performance of speech recognition systems and truly pushed speech recognition to a commercially available level. Due to the huge practical value of speech recognition technology, this field has become a research hotspot for technology giants, Internet companies and well-known universities. After the deep neural network (DNN) was introduced into speech recognition, the application of neural network sequence discrimination training and Convolutional Neural Network (CNN) in speech recognition was further studied.

Subsequently, the Long Short Time Memory (LSTM) model was introduced into acoustic modeling, and the LSTM model has stronger acoustic modeling capabilities than a simple feedforward network. Due to the increasing amount of data, it is necessary to deepen the number of layers of the acoustic model neural network to improve the modeling ability. However, as the number of network layers in the LSTM model deepens, the training difficulty of the network increases, and the gradient disappears. In order to avoid the disappearance of the gradient, an HLSTM model based on the LSTM model was proposed, which introduced direct connections between memory cells in adjacent layers of the LSTM model.

The proposed HLSTM model enables the deeper network structure to be practically applied in the recognition system and greatly improves the recognition accuracy. Although the deep HLSTM model has stronger modeling capabilities, the deepening of the number of layers and the introduction of new connections (the above-mentioned direct connection) also make the acoustic model have a more complex network structure, so the forward calculation takes longer. , eventually leading to slower decoding. Therefore, how to improve the performance without increasing the complexity of the acoustic model becomes a problem to be solved.

Summary of the invention

Embodiments of the present disclosure provide an acoustic modeling method, apparatus, and storage medium based on an HLSTM model.

The technical solution of the embodiment of the present disclosure is implemented as follows:

Embodiments of the present disclosure provide an acoustic modeling method based on an HLSTM model, including:

The randomly initialized HLSTM model is trained based on a preset function, and the training result is optimized;

Performing forward calculation by using the HLSTM model obtained by the optimization;

And training the randomly initialized LSTM model based on the result of the forward calculation and the preset function, and the obtained model is an acoustic model of the speech recognition system;

Wherein, the HLSTM model is the same as the network parameter of the LSTM model.

In the above solution, the training is performed on the randomly initialized HLSTM model based on a preset function, and the training result is optimized, including:

Training the randomly initialized HLSTM model with the cross entropy objective function;

The HLSTM model obtained by the training is optimized according to the state-level minimum Bayesian risk criterion.

In the above solution, the cross entropy objective function is:

为t时刻的语音特征在y状态输出点的标注值；所述p(y|X _t)为神经网络t时刻的语音特征，对应y状态点的输出；所述X表示训练数据；所述S为输出状态点的数目，所述N为语音特征总时长。 Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature.

In the above solution, the objective function corresponding to the state-level minimum Bayesian risk criterion is:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores.

In the above solution, the number of network layers of the HLSTM model is greater than or equal to the number of network layers of the LSTM model.

In the above solution, the training the randomly initialized LSTM model based on the result of the forward calculation and the preset function, including:

Obtaining an output result of each frame obtained by the forward calculation;

为所述前向计算得到的每帧输出结果。 Training a randomly initialized LSTM model based on the output result of each frame and a cross entropy objective function; wherein, in the cross entropy objective function

The result is output for each frame obtained by the forward calculation.

An embodiment of the present disclosure further provides an acoustic modeling device based on an HLSTM model, including:

The HLSTM model processing module is configured to train the randomly initialized HLSTM model based on a preset function and optimize the training result;

a calculation module configured to perform training forward calculation through the optimized HLSTM model;

The LSTM model processing module is configured to train the randomly initialized long and short time memory LSTM model based on the forward calculation result and the preset function, and the obtained model is an acoustic model of the speech recognition system;

Wherein, the HLSTM model is the same as the network parameter of the LSTM model.

In the above solution, the HLSTM model processing module includes:

a first training unit configured to train the randomly initialized HLSTM model using a cross entropy objective function;

An optimization unit configured to optimize the HLSTM model obtained by the training according to a state-level minimum Bayesian risk criterion.

In the above solution, the cross entropy objective function is:

为t时刻的语音特征在y状态输出点的标注值；所述p(y|X _t)为神经网络t时刻的语音特征，对应y状态点的输出；所述X表示训练数据；所述S为输出状态点的数目，所述N为语音特征总时长。 Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature.

In the above solution, the objective function corresponding to the state-level minimum Bayesian risk criterion is:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores.

In the above solution, the LSTM model processing module includes:

An obtaining unit configured to obtain an output result of each frame obtained by the forward calculation;

为所述前向计算得到的每帧输出结果。 a second training unit configured to train the randomly initialized LSTM model based on the output result of each frame and the cross entropy objective function; wherein, in the cross entropy objective function

The result is output for each frame obtained by the forward calculation.

Embodiments of the present disclosure further provide a storage medium having stored thereon a computer program that, when executed by a processor, implements the steps of any of the above methods.

The HLSTM model-based acoustic modeling method, apparatus and storage medium provided by the embodiments of the present disclosure train a randomly initialized HLSTM model based on a preset function, and optimize the training result; and the training data is obtained through the optimization. The HLSTM model performs forward calculation; based on the result of the forward calculation and the preset function, training the randomly initialized LSTM model, and the obtained model is an acoustic model of the speech recognition system; wherein the HLSTM model and the The network parameters of the LSTM model are the same. The embodiment of the present disclosure transmits the network information of the optimized HLSTM model to the LSTM network through the posterior probability, thereby improving the performance of the LSTM baseline model without increasing the complexity of the model.

DRAWINGS

1 is a schematic flow chart of an acoustic modeling method based on an HLSTM model according to an embodiment of the present disclosure;

2 is a network structure diagram of a bidirectional HLSTM model according to an embodiment of the present disclosure;

3 is a schematic structural diagram of an acoustic modeling device based on an HLSTM model according to an embodiment of the present disclosure;

4 is a schematic structural diagram of an HLSTM model processing module according to an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of an LSTM model processing module according to an embodiment of the present disclosure.

detailed description

The present disclosure is described in detail below in conjunction with the specific embodiments.

FIG. 1 is a schematic flowchart of an acoustic modeling method based on an HLSTM model according to an embodiment of the present disclosure. As shown in FIG. 1 , the method includes:

Step 101: Train the randomly initialized HLSTM model based on a preset function, and optimize the training result.

Step 102: Perform training calculation by using the HLSTM model obtained by the optimization;

Step 103: Train the randomly initialized LSTM model based on the result of the forward calculation and the preset function, and obtain the model as an acoustic model of the speech recognition system;

Wherein, the HLSTM model is the same as the network parameter of the LSTM model.

Here, the HLSTM model and the LSTM model may both be bidirectional or both unidirectional. The network parameters may include: an input layer node number, an output layer node number, an input observation vector, a hidden layer node number, a recursive delay, and a mapping layer connected after each hidden layer.

The embodiment of the present disclosure transmits the network information of the optimized HLSTM model to the LSTM network through the posterior probability, thereby improving the performance of the LSTM baseline model without increasing the complexity of the model.

As an example, the randomly initialized HLSTM model is shown in FIG. 2, and the dotted line box is an inter-layer memory unit connection (direct connection) set on the basis of the LSTM model, as shown in FIG. 2. Since the direct connection between adjacent layer memory cells is introduced in the HLSTM model, the problem of gradient disappearance can be avoided, and the difficulty of network training is reduced, so that a deeper structure can be used in practical applications. On the other hand, subject to the limitation of the parameter quantity, the number of network layers cannot be infinitely deepened because the larger parameter quantity model causes over-fitting compared to the amount of training data. In actual use, the number of network layers of the HLSTM model can be adjusted based on the amount of training data available.

In the embodiment of the present disclosure, the training is performed on the randomly initialized HLSTM model based on a preset function, and the training result is optimized, including:

Training the randomly initialized HLSTM model with the cross entropy objective function;

The HLSTM model obtained by the training is optimized according to the state-level minimum Bayesian risk criterion.

Wherein the cross entropy objective function is:

为t时刻的语音特征在y状态输出点的标注值；所述p(y|X _t)为神经网络t时刻的语音特征，对应y状态点的输出；所述X表示训练数据；所述S为输出状态点的数目，所述N为语音特征总时长。 Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature.

Wherein, the objective function corresponding to the state-level minimum Bayesian risk criterion is:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores.

In the embodiment of the present disclosure, the number of network layers of the HLSTM model is greater than or equal to the number of network layers of the LSTM model.

In the embodiment of the present disclosure, the training the randomly initialized LSTM model based on the result of the forward calculation and the preset function, including:

Obtaining an output result of each frame obtained by the forward calculation;

为所述前向计算得到的每帧输出结果。 Training a randomly initialized LSTM model based on the output result of each frame and a cross entropy objective function; wherein, in the cross entropy objective function

The result is output for each frame obtained by the forward calculation.

After comparing the HLSTM and LSTM, it is found that the performance improvement of the LSTM model after the introduction of the direct connection, that is, the HLSTM model is significantly higher than that obtained by the LSTM model. Therefore, the performance of the discrimination training on the HLSTM model is Promotion is very meaningful.

The embodiment of the present disclosure further provides an acoustic modeling device based on the HLSTM model, which is used to implement the above embodiments and specific implementations, and has not been described again. As used below, the term "module" "unit" may implement a combination of software and/or hardware of a predetermined function. As shown in Figure 3, the device comprises:

The HLSTM model processing module 301 is configured to train the randomly initialized HLSTM model based on a preset function, and optimize the training result;

The calculating module 302 is configured to perform training calculation by using the optimized HLSTM model;

The LSTM model processing module 303 is configured to train the randomly initialized LSTM model based on the result of the forward calculation and the preset function, and the obtained model is an acoustic model of the speech recognition system;

Wherein, the HLSTM model is the same as the network parameter of the LSTM model.

The embodiment of the present disclosure transmits the network information of the optimized HLSTM model to the LSTM network through the posterior probability, thereby improving the performance of the LSTM baseline model without increasing the complexity of the model.

As an example, the randomly initialized HLSTM model is shown in FIG. 2, and the dotted line box is an inter-layer memory unit connection (direct connection) set on the basis of the LSTM model, and the connection formula is as shown in FIG. 2. Since the direct connection between adjacent layer memory cells is introduced in the HLSTM model, the problem of gradient disappearance can be avoided, and the difficulty of network training is reduced, so that a deeper structure can be used in practical applications. On the other hand, subject to the limitation of the parameter quantity, the number of network layers cannot be infinitely deepened because the larger parameter quantity model causes over-fitting compared to the amount of training data. In actual use, the number of network layers of the HLSTM model can be adjusted based on the amount of training data available.

In the embodiment of the present disclosure, as shown in FIG. 4, the HLSTM model processing module 301 includes:

The first training unit 3011 is configured to train the randomly initialized HLSTM model by using a cross entropy objective function;

The optimization unit 3012 is configured to optimize the HLSTM model obtained by the training according to the state-level minimum Bayesian risk criterion.

Wherein the cross entropy objective function is:

为t时刻的语音特征在y状态输出点的标注值；所述p(y|X _t)为神经网络t时刻的语音特征，对应y状态点的输出；所述X表示训练数据；所述S为输出状态点的数目，所述N为语音特征总时长。 Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature.

Wherein, the objective function corresponding to the state-level minimum Bayesian risk criterion is:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores.

In the embodiment of the present disclosure, as shown in FIG. 5, the LSTM model processing module 303 includes:

The obtaining unit 3031 is configured to acquire an output result of each frame obtained by the forward calculation;

为所述前向计算得到的每帧输出结果。 a second training unit 3032, configured to train the randomly initialized LSTM model based on the output result of each frame and the cross entropy objective function; wherein, in the cross entropy objective function

The result is output for each frame obtained by the forward calculation.

In the embodiment of the present disclosure, the number of network layers of the HLSTM model is greater than or equal to the number of network layers of the LSTM model.

After comparing the HLSTM and LSTM, it is found that the performance improvement of the LSTM model after the introduction of the direct connection, that is, the HLSTM model is significantly higher than that obtained by the LSTM model. Therefore, the performance of the discrimination training on the HLSTM model is Promotion is very meaningful.

In practical applications, the HLSTM model processing module 301, the calculation module 302, the LSTM model processing module 303, the first training unit 3011, the optimization unit 3012, the acquisition unit 3031, and the second training unit 3032 may be in an HLSTM model-based acoustic modeling device. Processor implementation.

The present disclosure is described below in conjunction with a specific scenario embodiment.

In this embodiment, a deep two-way HLSTM model with stronger modeling capability is trained as a "teacher" model, a randomly initialized two-way LSTM model is used as a "student" model, and a "teacher" model is used to train a student with a relatively small parameter amount. "model. The specific method is described as follows:

First, training the "teacher" model

First, the HLSTM model is randomly initialized. The network structure of the HLSTM model is shown in Figure 2. Since HLSTM introduces direct connection between adjacent layer memory cells, the problem of gradient disappearance is avoided, and the difficulty of network training is reduced. Therefore, a deeper structure can be used in practical applications. On the other hand, subject to the limitation of the parameter quantity, the number of network layers cannot be infinitely deepened because the excessive parameter quantity model causes over-fitting compared to the amount of training data. In actual use, the number of HLSTM network layers can be adjusted according to the amount of training data available. In this embodiment, the training data can be 300h (hours), and the HLSTM model used is 6 layers, namely: an input layer, an output layer, and four hidden layers between them.

The HLSTM model is iteratively updated using the CrossEntropy (CE) objective function. The CE objective function formula is as follows:

为t时刻的语音特征在y状态输出点的标注值；所述p(y|X _t)为神经网络t时刻的语音特征，对应y状态点的输出；所述X表示训练数据；所述S为输出状态点的数目，所述N为语音特征总时长。 Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature.

The HLSTM model generated based on CE objective function training has better recognition performance. On this basis, the model is further optimized by the discriminative sequence-level training criterion, namely: State-level Minimum Bayes Risk (SMBR) criterion. The difference from the acoustic model training of the CE criterion is that the discriminative sequence-level training criterion tries to learn more classes from the positive and negative training samples on a limited training set by optimizing the function related to the system recognition rate. Distinguish information. Its objective function is as follows:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores. Through the comparison experiment between HLSTM and LSTM, it is found that the performance improvement obtained by introducing the newly connected model (HLSTM model) for discriminative training is significantly higher than that obtained by the LSTM model. Therefore, the performance improvement of the HLSTM model by discriminative training is Very meaningful. At this point, the trained model is the "teacher" model.

Second, training the "student" model

Randomly initialize an LSTM model with three hidden layers. The other parameters of the model are consistent with the "teacher" model. Next, you need to pass the information learned by the HLSTM model to the LSTM model. The information transmission method of the embodiment of the present disclosure is to perform the forward calculation by using the "teacher" model, obtain the output corresponding to each frame input, mark the obtained output, and use the CE criterion mentioned above as the objective function to train. The "student" model, the trained LSTM model is used as an acoustic model for speech recognition systems.

An advantage of embodiments of the present disclosure is to improve LSTM baseline model performance without increasing model complexity. Although the HLSTM model has stronger modeling capabilities and higher recognition performance, the decoding real-time rate is also one of the indicators for evaluating the performance of the recognition system. The HLSTM model has a higher parameter size and model complexity than the LSTM model, which inevitably slows down the decoding speed. The HLSTM model network information is transmitted to the LSTM network through the posterior probability, thereby improving the performance of the LSTM baseline model. Although there is an inevitable performance loss in the information transfer process, the “student” model performance is lower than the “teacher” model, but Still higher than the performance of the directly trained LSTM model.

The method embodiments are described below in conjunction with specific model parameters.

Step 1: Extract the speech features of the training data. The EM algorithm is used to iteratively update the mean variance of the GMM-HMM system, and the GMM-HMM system is used to force the alignment of the feature data to obtain the three-factor cluster state annotation.

Step 2: Train the two-way HLSTM model based on the cross entropy criterion.

In this embodiment, a six-layer bidirectional HLSTM model is used, and the parameter quantity of the model is 190M. The specific configuration is as follows: the input layer has 260 nodes, the input observation vector is 2 frames for the context, and the number of nodes of the four hidden layers are both For 1024, the recursive delay is 1, 2, 3, 4; each layer of hidden layer is connected with a 512-dimensional mapping layer to reduce the dimension reduction parameter. The number of nodes in the output layer is 2821, which corresponds to 2821 triphone clustering states.

Step 3: The model generated in step 2 is used as a seed model, and the bidirectional HLSTM model is iteratively updated based on the state-level minimum Bayesian risk criterion.

Step 4: Perform the forward calculation by using the two-way HLSTM model generated in step three to obtain the output vector.

Step 5: The output vector obtained in step 4 is labeled corresponding to the input feature, and the bidirectional LSTM model with three hidden layers is trained, and the parameter quantity is 120M. The network parameters of the model are consistent with the HLSTM model in step 2.

Those skilled in the art will appreciate that embodiments of the present disclosure can be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or a combination of software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) including computer usable program code.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

Based on this, an embodiment of the present disclosure further provides a storage medium, in particular a computer readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in the embodiment of the present disclosure are implemented.

The above description is only for the preferred embodiments of the present disclosure, and is not intended to limit the scope of the disclosure.

Industrial applicability

The solution provided by the embodiment of the present disclosure trains the randomly initialized HLSTM model based on a preset function, and optimizes the training result; and performs training data through the optimized HLSTM model for forward calculation; To the calculated result and the preset function, the randomly initialized LSTM model is trained, and the obtained model is an acoustic model of the speech recognition system; wherein the HLSTM model has the same network parameters as the LSTM model. The embodiment of the present disclosure transmits the network information of the optimized HLSTM model to the LSTM network through the posterior probability, thereby improving the performance of the LSTM baseline model without increasing the complexity of the model.

Claims (12)

An acoustic modeling method based on the direct connected long and short time memory HLSTM model, including: The randomly initialized HLSTM model is trained based on a preset function, and the training result is optimized; Performing forward calculation by using the HLSTM model obtained by the optimization; And training the randomly initialized long and short time memory LSTM model based on the result of the forward calculation and the preset function, and the obtained model is an acoustic model of the speech recognition system; Wherein, the HLSTM model is the same as the network parameter of the LSTM model. The method of claim 1, wherein the training of the randomly initialized HLSTM model based on a preset function and optimizing the training result comprises: Training the randomly initialized HLSTM model with the cross entropy objective function; The HLSTM model obtained by the training is optimized according to the state-level minimum Bayesian risk criterion. The method of claim 2 wherein said cross entropy objective function is:

Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature. The method of claim 2 wherein the objective function corresponding to the state-level minimum Bayesian risk criterion is:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores. The method of any of claims 1-4, wherein the number of network layers of the HLSTM model is greater than or equal to the number of network layers of the LSTM model. The method of claim 3, wherein the training the randomly initialized LSTM model based on the result of the forward calculation and the preset function comprises: Obtaining an output result of each frame obtained by the forward calculation; Training a randomly initialized LSTM model based on the output result of each frame and a cross entropy objective function; wherein, in the cross entropy objective function

The result is output for each frame obtained by the forward calculation. An acoustic modeling device based on a direct connected long and short time memory HLSTM model, comprising: The HLSTM model processing module is configured to train the randomly initialized HLSTM model based on a preset function and optimize the training result; a calculation module configured to perform training forward calculation through the optimized HLSTM model; The LSTM model processing module is configured to train the randomly initialized long and short time memory LSTM model based on the forward calculation result and the preset function, and the obtained model is an acoustic model of the speech recognition system; Wherein, the HLSTM model is the same as the network parameter of the LSTM model. The apparatus of claim 7 wherein said HLSTM model processing module comprises: a first training unit configured to train the randomly initialized HLSTM model using a cross entropy objective function; An optimization unit configured to optimize the HLSTM model obtained by the training according to a state-level minimum Bayesian risk criterion. The apparatus of claim 8 wherein said cross entropy objective function is:

Wherein the F _CE represents a cross entropy objective function;

The speech feature at time t is the annotation value of the output point in the y state; the p(y|X _t ) is the speech feature of the neural network t, corresponding to the output of the y state point; the X represents the training data; To output the number of state points, the N is the total duration of the speech feature. The apparatus of claim 8 wherein the objective function corresponding to the state-level minimum Bayesian risk criterion is:

Wherein, the _Wu is an annotated text of a voice; the W and W′ are labels corresponding to a decoding path of the seed model; the p(O _u |S) is an acoustic likelihood probability; and the A(W, W _u ) represents the number of correct state annotations in the sequence of decoding states; the seed model is: the HLSTM model obtained after the optimization; the u represents an index of the statement number in the training data, and the k is an acoustic score coefficient, The O _u is the speech feature of the corpus of the u sentence, the S represents the state sequence of the decoding path, and the P(W) and P(W') are both language model probability scores. The apparatus of claim 9 wherein said LSTM model processing module comprises: An obtaining unit configured to obtain an output result of each frame obtained by the forward calculation; a second training unit configured to train the randomly initialized LSTM model based on the output result of each frame and the cross entropy objective function; wherein, in the cross entropy objective function

The result is output for each frame obtained by the forward calculation. A storage medium having stored thereon a computer program, the computer program being executed by a processor to perform the steps of the method of any one of claims 1 to 6.

Images