CN116052649A - Loss weight self-adaptive element learning method in low-resource speech recognition - Google Patents

Abstract

A loss weight adaptive meta-learning method in low-resource speech recognition, related to the field of speech recognition. Aiming at the problems existing in the prior art that the MAML algorithm is unstable and the training loss weight is difficult to adjust accurately, a loss weight adaptive meta-learning method in low-resource speech recognition is provided. The method of fine-tuning weights through homoscedastic uncertainty solves the instability of MAML algorithm to a certain extent, and applies it to speech recognition. On the basis of it, VGG‑CNN network and Adapter module are introduced to improve the overall recognition performance. Compared with other meta-learning methods, it has smaller fluctuations and is more stable. It does not need to manually adjust the loss weight or spend a lot of money to accurately adjust the weight. It can automatically and efficiently adjust the training loss weight to an appropriate value. It can be used on any model, with high flexibility, and the added VGG-CNN module and Adapter module increase the calculation cost at low cost.

Description

A loss weight adaptive meta-learning approach for low-resource speech recognition

Technical Field

The present invention relates to the field of speech recognition, and in particular to a loss weight adaptive meta-learning method in low-resource speech recognition.

Background Art

With the vigorous development of machine learning, end-to-end models have achieved excellent results in the field of speech recognition, but these good performances often rely on a large amount of labeled speech data. However, in most scenarios, many languages do not have abundant and correctly labeled data. The high cost of annotation and the scarce amount of data further limit the effect and generalization performance of the model. Although the application of data augmentation (such as speed perturbation, noise addition, and speech synthesis) and some special training methods (such as transfer learning, multilingual joint learning, and adversarial learning) can alleviate the above problems to a certain extent, these methods still have problems such as overfitting and weak results.

In order to solve the above problems, meta-learning was introduced into speech recognition under low-resource conditions. Meta-learning is also called learning to learn, which can effectively absorb the prior knowledge of other tasks and efficiently use this knowledge to learn new tasks. The model-independent meta-learning algorithm MAML (Model Agnostic Meta-Learning) is the most flexible and effective meta-learning algorithm based on gradient descent, which does not introduce redundant parameters (Chelsea Finn, Pieter Abbeel, and Sergey Levine, Model agnostic meta-learning for fast adaptation of deepnetworks, in International conference on machine learning. PMLR, 2017, pp. 1126–1135). It obtains the best initialization parameters of the model by training different tasks. In the fine-tuning stage, only a small amount of sample data is needed for the model to learn new tasks.

Although the performance of the MAML algorithm is better than other low-resource methods, due to the double-layer loss back-propagation update method of the MAML algorithm during the training process, the corresponding losses of different tasks in the training stage are unstable, which further leads to large fluctuations in the overall recognition performance of the model. In addition, different training tasks have different effects on the target task to be recognized, and are controlled by the corresponding loss weights. Appropriate loss weights can further improve the performance of the target model, otherwise, the performance of the model will be lost. However, the cost of finding the best weights by setting hyperparameters is very high and relies on complex manual adjustments.

Based on this, the method of using homoscedastic uncertainty to adjust the weight of loss is used in the MAML algorithm (A.Boiarov, K.Khabarlak, and I.Yastrebov, Multi-task meta learning modification with stochastic approximation, 2021). Homoscedastic uncertainty, which can capture the relative confidence between different tasks, can be used as the basis for measuring the loss weight in multi-task learning problems. Borrowing this point, in the MAML algorithm, different languages are treated as different tasks, and the weights between different tasks are adjusted based on homoscedastic uncertainty to stabilize the loss, thereby improving the overall performance of the model. The improved MAML algorithm based on this has been applied to image recognition (WenfengShen Lin Ding, Peng Liu and Shengbo Chen, Gradient-based meta-learning using uncertainty to weigh loss for few-shot learning, arXiv:2208.0813, 2022), which evaluates the corresponding weights in the form of a single hyperparameter. Although the hyperparameters are updated as the model is trained, the individual parameters cannot learn the effective information in the task well and cannot be further utilized.

How to more effectively use homoscedastic uncertainty in speech recognition to adjust the loss weights between different tasks in the MAML algorithm, and how to enable it to learn more effective feature generalization knowledge to assist model training and improve the overall model's recognition performance for the target language are issues that are worth studying.

Chinese Patent 113178190A uses meta-learning such as matching networks and twin networks for speech recognition, but these methods are highly correlated with the model and are not as flexible as the MAML algorithm used in the present invention. On this basis, although Patent 112560904A applies the MAML algorithm to machine learning, it does not adjust the loss weight, and the instability of the MAML algorithm still exists.

Summary of the invention

The purpose of the present invention is to provide a loss weight adaptive meta-learning method in low-resource speech recognition in view of the technical problems of the instability of the MAML algorithm and the difficulty in accurately adjusting the training loss weight in the prior art. The method of fine-tuning the weights by homoscedastic uncertainty solves the problems of the instability of the MAML algorithm to a certain extent, and applies it to speech recognition. At the same time, the VGG-CNN network and the Adapter module are further introduced on this basis to improve the overall recognition performance.

A loss weight adaptive meta-learning method for low-resource speech recognition according to the present invention comprises the following steps:

1) Classify the multiple languages used in training according to their languages and divide them into training sets and validation sets;

2) Divide the speech data of the training set and validation set of each language category into two parts: the support set and the query set;

3) Collect statistics on the text of all speech data and create a training dictionary;

4) Write code to build a speech recognition model consisting of a Transformer model, a VGG-CNN network, and an Adapter module;

5) Carry out model pre-training stage;

5.1) In each training iteration, for each language’s training set speech data, first extract k data from the training set’s support set, extract features through the VGG-CNN network, and then feed the features into the Transformer model for training, and update the model parameters of the current iteration round;

5.2) Then extract k data from the query set of the corresponding language, extract features through the VGG-CNN network, and send the features to the Transformer model to test the recognition performance of the model to obtain the corresponding query set loss. At the same time, send the features to the Adapter module, and after taking the mean and logarithm operations, obtain the variance required to adjust the loss weight;

5.3) Use the obtained variance to adjust the query set loss of the corresponding language in the current round through the formula;

5.4) After completing the above training operation for a single language, initialize the model parameters to the values at the beginning of the current training round;

6) Repeat steps 5.1) to 5.4) for the training and validation sets of other languages to obtain the adjusted query set losses of all languages used for training in the current iteration, and sum and average them;

7) Using the mean loss of the query set after the operation in step 6), perform gradient descent update on all model parameters at the beginning of the current iteration of the model;

8) After updating the model parameters, the current parameters of the model will be used as the model initialization parameters for the next iteration, and steps 5) to 7) will be repeated;

9) After each training epoch, the currently trained model is tested using the speech data of the validation set, and the model parameters with the best performance after the test are saved. When the model parameters saved at a certain time are the same optimal values after multiple tests, the above training process is terminated;

10) Prepare training sets, validation sets, and test sets for the languages to be recognized;

11) Using the training set of the language to be recognized, train the model saved in step 9);

11.1) During training, in each iteration, a batch of speech data is extracted from the training set. After extracting features through the VGG-CNN network, the features are sent to the Transformer model to test the recognition performance of the model and obtain the corresponding loss. At the same time, the features are sent to the Adapter module. After taking the mean and logarithm operations, the variance required to adjust the loss weight is obtained.

11.2) Use the obtained variance to adjust the training loss of the required recognition language in the current round through the formula;

11.3) Update the model parameters using the adjusted loss obtained in step 11.2);

12) After each epoch of training, use the speech data of the target language verification set to test the currently trained model and save the model with the best performance;

13) After repeatedly training multiple epochs, the above training process for the target language is terminated, and a speech recognition model with excellent recognition performance can be obtained.

Compared with the prior art, the present invention has the following technical effects:

The present invention can effectively improve the speech recognition performance of the target language under low-resource conditions. At the same time, it has been verified that the algorithm performance of the present invention has smaller fluctuations and is more stable than other meta-learning methods. There is no need to manually adjust the loss weight or accurately adjust the weight at a high cost. The present invention can automatically and efficiently adjust the training loss weight to an appropriate value. At the same time, the present invention does not make any changes to the model and can be used on any model. It is highly flexible, and the added VGG-CNN module and Adapter module increase the computational cost to be almost negligible.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the structure of the Adapter module.

Figure 2 is a schematic diagram of the structure of the VGG-CNN module.

FIG. 3 is a schematic diagram of the training process of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

和用于评估此次训练性能的验证集

The model-independent meta-learning algorithm MAML can be applied to all model learning that relies on gradient descent because it does not introduce unnecessary parameters. The overall process of the MAML algorithm is similar to the pre-training-fine-tuning process. It is based on the learning of pre-training data, so that the target data can be learned more effectively in the fine-tuning stage. However, unlike ordinary model training tasks, in addition to the source speech data D _source used for training and the target speech data D _{tar get} used for subsequent fine-tuning, the MAML algorithm divides different speech data into different tasks according to their corresponding languages, and each task i is further divided into a support set used to simulate model training during the training process.

and a validation set for evaluating the performance of this training

中随机选取k个数据对当前模型进行梯度更新，用θ表示模型的参数，则元训练过程如公式1所示：In the training phase, the MAML algorithm is divided into two phases: meta-training and meta-testing. In the meta-training phase, for task i, the algorithm will select

Randomly select k data to update the gradient of the current model, and use θ to represent the parameters of the model. The meta-training process is shown in Formula 1:

是相应的交叉熵损失；α则是对应的学习率。在相同的模型参数下，对所有的任务同步进行上述的训练。随后在元测试阶段，分别抽取对应

上k个数据评估训练后参数为θ′_i的模型性能，之后将模型的参数充值为当前轮次的初始值θ，再利用所有任务验证集的损失之和更新模型当前轮次的初始化参数，如公式2所示：Among them, θ′ _i is the parameter of the updated model for training corresponding task i;

is the corresponding cross entropy loss; α is the corresponding learning rate. Under the same model parameters, all tasks are trained synchronously. Then, in the meta-test phase, the corresponding

The performance of the model with parameters θ′ _i after training is evaluated on the last k data, and then the parameters of the model are recharged to the initial value θ of the current round, and then the sum of the losses of all task validation sets is used to update the initialization parameters of the model in the current round, as shown in Formula 2:

Among them, β is the corresponding learning rate. The model parameters obtained after updating through formula 2 will be used as the model initialization parameters for the next round to repeat the above update process. The overall process can be summarized as the following formula 3:

It can be found from the above formula that in the updating process, the quadratic derivative of the model parameter θ is involved, which greatly increases the computational complexity of the algorithm. The subsequent first-order derivative approximation method is proposed, as shown in Formula 4, which replaces the outer layer's derivative of θ′ _i with the derivative of θ. On the basis of greatly reducing the computational cost, its effect is almost not affected. The MAML algorithm used in the present invention is further extended on this basis.

After repeated iterations of the above process, very good initialization parameters can be obtained. On this basis, even if the target data D _target is low-resource, the model can achieve very good performance after a small amount of gradient updates.

In the meta-test phase of the present invention, the MAML algorithm adds the evaluation losses of all tasks to update the initialization parameters of the model. However, different tasks have different sensitivities to the same loss function, and simply adding the losses will further amplify the negative impact of this sensitivity. At the same time, the unique double-layer gradient update method of the MAML algorithm and the small amount of data used in a single update have been proven to be less stable after a large number of experiments. Due to the above two reasons, the algorithm's update of the overall model parameters occasionally produces large fluctuations, and may even damage the performance of the model during the fine-tuning phase.

The present invention aims to achieve loss weight adaptation in the meta-test phase by utilizing variance uncertainty, so that the corresponding weights can be automatically adjusted to achieve the best effect. At the same time, through this adaptive adjustment process, the corresponding adaptive module can learn more useful generalized deep representations, and apply them to the fine-tuning phase to assist in the training of target data, so that the model can obtain better performance.

Homoscedastic uncertainty, also known as task-related uncertainty, is a quantity that remains constant for the same task but varies between different tasks. Homoscedastic uncertainty can obtain the relative confidence between different tasks in multi-task learning, reflecting the uncertainty of different tasks. It was first used in multi-task learning as a weight to measure the loss of different tasks to enhance the stability of the loss. Therefore, similar to multi-task learning, meta-learning can use it as a basis for evaluating the loss weights of different tasks.

Speech recognition can generally be considered as a regression problem based on Gaussian distribution:

是基于模型参数θ以及模型输入

的模型输出，L表示输出向量的长度，T表示输入音频数据的时域维度，F则是音频的频域维度；

是对应输入的真实标签，σ²∈

是与同方差不确定性相关的方差标量，本发明基于该标量可以间接反映不同任务之间相对分布的特点，使用其来作为衡量不同任务之间损失权重的基础。基于极大似然估计以及交叉熵损失可以得到：in,

is based on the model parameters θ and the model input

The model output is L, which represents the length of the output vector, T represents the time domain dimension of the input audio data, and F represents the frequency domain dimension of the audio.

is the true label of the corresponding input, σ ² ∈

It is a variance scalar related to homoscedastic uncertainty. Based on the fact that this scalar can indirectly reflect the relative distribution between different tasks, this paper uses it as the basis for measuring the loss weights between different tasks. Based on maximum likelihood estimation and cross entropy loss, we can get:

The fluctuation of MAML algorithm performance is mainly concentrated on the validation set loss in the meta-test phase. The present invention uses homoscedastic uncertainty to adjust the loss weight in the meta-test phase. Therefore, based on the above formula and the formula of the meta-test phase of the MAML algorithm, the validation set loss corresponding to N tasks in the meta-test phase can be further obtained:

以及

是对应任务i在元测试阶段的验证集损失以及用于微调损失权重方差标量。模型输入

是由B个输入音频数据x组成的，

则是其对应的真实标签。Among them, θ′ _i is the model parameter obtained after the meta-training phase for the corresponding task i,

as well as

is the validation set loss of task i in the meta-test phase and the variance scalar used to fine-tune the loss weight. Model input

is composed of B input audio data x,

is the corresponding true label.

在原先简单加和的基础上通过基于同方差不确定性的σ_i进行微调，强化不同任务间的差异性，强化损失的稳定性，从而进一步地使模型在训练过程中学习到更加泛化的语音特征，提高模型的整体识别性能。Compared with the common MAML algorithm, the present invention

On the basis of the original simple addition, fine-tuning is performed based on _σi based on homoscedastic uncertainty to enhance the differences between different tasks and the stability of the loss, so that the model can further learn more generalized speech features during the training process and improve the overall recognition performance of the model.

I.，Gurevych，I.，&Ruder，S.(2020).MAD-X：An Adapter-Based Framework for Multi-Task Cross-LingualTransfer(arXiv：2005.00052).arXiv)基于验证集计算得到，Adapter模块结构如图1所示：模块首先对输入数据进行均值化操作，随后对其进行下采样，在经过ReLU激活函数后通过升采样操作使其恢复原有维度，最后将操作后的数据与最开始的输入数据进行残差链接，得到最后的输出结果。On this basis, the present invention further replaces the hyperparameter σ _i, which is updated with training and serves as a fine-tuning parameter and a regularization term, with a shared Adapter module (Pfeiffer, J.,

I., Gurevych, I., & Ruder, S. (2020). MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (arXiv: 2005.00052). arXiv) is calculated based on the validation set. The Adapter module structure is shown in Figure 1: the module first averages the input data, then downsamples it, and restores its original dimension through upsampling after the ReLU activation function. Finally, the operated data is residually linked with the initial input data to obtain the final output result.

The schematic diagram of the structure of the VGG-CNN module is shown in Figure 2; the obtained value is logarithmized to prevent it from being negative, and g(X) is used to represent the process of the model in VGG-CNN, Adpater, and taking the mean and logarithm. The calculation formula of _σi and the weight adjustment formula are further obtained as follows:

Compared with the adjustment method using hyperparameters, the present invention can obtain more useful generalized knowledge and learn more detailed deep representations in the training stage with the help of CNN network and Adapter module. At the same time, the target language is regarded as a single task, and the current method is further used to evaluate and adjust the loss in the fine-tuning stage to improve the recognition performance of the target language. The overall formula is as follows, where Conv and Adapter in g(X) are both trained in the pre-training stage.

Based on formula 10, the present invention can automatically and accurately adjust the weight of the training loss during the training of the MAML algorithm, and effectively update the parameters of the model based on the adjusted weight, so that the MAML algorithm in the present invention is more stable and has better performance during the training process than other meta-learning methods. Although the VGG-CNN network and the Adapter module are introduced into the model, there is almost no increase in the computational cost, and the speed is basically the same as the original MAML algorithm.

The technical solution of the present invention and its alternatives:

1. Basic MAML algorithm

和用于评估此次训练性能的验证集

The model-independent meta-learning algorithm MAML can be applied to all model learning that relies on gradient descent because it does not introduce unnecessary parameters. The overall process of the MAML algorithm is similar to the pre-training-fine-tuning process. It is based on the learning of pre-training data, so that the target data can be learned more effectively in the fine-tuning stage. However, unlike ordinary model training tasks, in addition to the source speech data D _source used for training and the target speech data D _{tar get} used for subsequent fine-tuning, the MAML algorithm divides different speech data into different tasks according to their corresponding languages, and each task i is further divided into a support set used to simulate model training during the training process.

and a validation set for evaluating the performance of this training

中随机选取k个数据对当前模型进行梯度更新，用θ表示模型的参数，则元训练过程如公式1所示：In the training phase, the MAML algorithm is divided into two phases: meta-training and meta-testing. In the meta-training phase, for task i, the algorithm will select

Randomly select k data to update the gradient of the current model, and use θ to represent the parameters of the model. The meta-training process is shown in Formula 1:

是相应的交叉熵损失；a则是对应的学习率。在相同的模型参数下，对所有的任务同步进行上述的训练。随后在元测试阶段，分别抽取对应

上k个数据评估训练后参数为θ′_i的模型性能，之后将模型的参数充值为当前轮次的初始值θ，再利用所有任务验证集的损失之和更新模型当前轮次的初始化参数，如公式2所示：Among them, θ′ _i is the parameter of the updated model for training corresponding task i;

is the corresponding cross entropy loss; a is the corresponding learning rate. Under the same model parameters, the above training is performed on all tasks simultaneously. Then, in the meta-test phase, the corresponding

The performance of the model with parameters θ′ _i after training is evaluated on the last k data, and then the parameters of the model are recharged to the initial value θ of the current round, and then the sum of the losses of all task validation sets is used to update the initialization parameters of the model in the current round, as shown in Formula 2:

Among them, β is the corresponding learning rate. The model parameters obtained after updating through formula 2 will be used as the model initialization parameters for the next round to repeat the above update process. The overall process can be summarized as the following formula 3:

It can be found from the above formula that the updating process involves the quadratic derivative of the model parameter θ, which greatly increases the computational complexity of the algorithm. The subsequent first-order derivative approximation method, as shown in Formula 4, replaces the outer layer's derivative of θ′ _i with the derivative of θ, which greatly reduces the computational cost and has almost no effect. The MAML algorithm of the present invention is further innovated on this basis.

After repeated iterations of the above process, very good initialization parameters can be obtained. On this basis, even if the target data D _target is low-resource, the model can achieve very good performance after a small amount of gradient updates.

2. The weight-adaptive MAML algorithm of the present invention

In the meta-test phase, the MAML algorithm adds up the evaluation losses of all tasks to update the model's initialization parameters. However, different tasks have different sensitivities to the same loss function, and simply adding up the losses will further amplify the negative impact of this sensitivity. At the same time, the MAML algorithm's unique dual-layer gradient update method and the small amount of data used in a single update have been proven to be less stable after a large number of experiments. Due to the above two reasons, the algorithm's update of the overall model parameters occasionally produces large fluctuations, and may even damage the model's performance during the fine-tuning phase.

The present invention aims to achieve loss weight adaptation in the meta-test phase by utilizing variance uncertainty, so that it can automatically adjust the corresponding weights to achieve the best effect. At the same time, through this adaptive adjustment process, the corresponding adaptive module can learn more useful generalized deep representations, and apply them to the fine-tuning phase to assist in the training of target data, so that the model can obtain better performance.

Homoscedastic uncertainty, also known as task-related uncertainty, is a quantity that remains constant for the same task but varies between different tasks. Homoscedastic uncertainty can obtain the relative confidence between different tasks in multi-task learning, reflecting the uncertainty of different tasks. It was first used in multi-task learning as a weight to measure the loss of different tasks to enhance the stability of the loss. Therefore, similar to multi-task learning, meta-learning can use it as a basis for evaluating the loss weights of different tasks.

Speech recognition can generally be considered as a regression problem based on Gaussian distribution:

是基于模型参数θ以及模型输入

的模型输出，L表示输出向量的长度，T表示输入音频数据的时域维度，F则是音频的频域维度；

是对应输入的真实标签，

是与同方差不确定性相关的方差标量，本发明基于该标量可以间接反映不同任务之间相对分布的特点，使用其来作为衡量不同任务之间损失权重的基础。基于极大似然估计以及交叉熵损失可以得到：in,

is based on the model parameters θ and the model input

The model output is L, which represents the length of the output vector, T represents the time domain dimension of the input audio data, and F represents the frequency domain dimension of the audio.

is the true label of the corresponding input,

It is a variance scalar related to homoscedastic uncertainty. Based on the fact that this scalar can indirectly reflect the relative distribution between different tasks, this paper uses it as the basis for measuring the loss weights between different tasks. Based on maximum likelihood estimation and cross entropy loss, we can get:

The fluctuation of MAML algorithm performance is mainly concentrated on the validation set loss in the meta-test phase. The present invention mainly uses homoscedastic uncertainty to adjust the loss weight in the meta-test phase. Therefore, based on the above formula and the formula of the meta-test phase of the MAML algorithm, the validation set loss corresponding to N tasks in the meta-test phase can be further obtained:

以及

是对应任务i在元测试阶段的验证集损失以及用于微调损失权重方差标量。模型输入

是由B个输入音频数据x组成的，

则是其对应的真实标签。相比较普通MAML算法，本发明的

在原先简单加和的基础上通过基于同方差不确定性的σ_i进行微调，强化不同任务间的差异性，强化损失的稳定性，从而进一步地使模型在训练过程中学习到更加泛化的语音特征，提高模型的整体识别性能。Among them, θ′ _i is the model parameter obtained after the meta-training phase for the corresponding task i,

as well as

is the validation set loss of task i in the meta-test phase and the variance scalar used to fine-tune the loss weight. Model input

is composed of B input audio data x,

is the corresponding true label. Compared with the common MAML algorithm, the

On the basis of the original simple addition, fine-tuning is performed based on _σi based on homoscedastic uncertainty to enhance the differences between different tasks and the stability of the loss, so that the model can further learn more generalized speech features during the training process and improve the overall recognition performance of the model.

I.，Gurevych，I.，&Ruder，S.(2020).MAD-X：An Adapter-Based Framework for Multi-Task Cross-LingualTransfer(arXiv：2005.00052).arXiv)基于验证集计算得到，模块首先对输入数据进行均值化操作，随后对其进行下采样，在经过ReLU激活函数后通过升采样操作使其恢复原有维度，最后将操作后的数据与最开始的输入数据进行残差链接，得到最后的输出结果。On this basis, the present invention further replaces the hyperparameter σ _i, which is updated with training and serves as a fine-tuning parameter and a regularization term, with a shared Adapter module (Pfeiffer, J.,

I., Gurevych, I., & Ruder, S. (2020). MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (arXiv: 2005.00052). arXiv) is calculated based on the validation set. The module first averages the input data, then downsamples it, and restores its original dimension through upsampling after the ReLU activation function. Finally, the operated data is residually linked with the initial input data to obtain the final output result.

Take the logarithm of the obtained value to prevent it from being negative, and use g(X) to represent the model in VGG-CNN, Adpater, and the process of taking the mean and logarithm. Further, the calculation formula of _σi and the weight adjustment formula are as follows:

Compared with the adjustment method using hyperparameters, the present invention can obtain more useful generalized knowledge and learn more detailed deep representations in the training stage with the help of CNN network and Adapter module. At the same time, the target language is regarded as a single task, and the current method is further used to evaluate and adjust the loss in the fine-tuning stage to improve the recognition performance of the target language. The overall formula is as follows, where Conv and Adapter in g(X) are both trained in the pre-training stage.

Based on formula 10, the present invention can automatically and accurately adjust the weight of the training loss during the training process of the MAML algorithm, and effectively update the parameters of the model based on the adjusted weight, so that the MAML algorithm in the present invention is more stable and has better performance during the training process than other meta-learning methods. See Figure 3 for a schematic diagram of the training process of the present invention.

Claims (1)

1. A loss weight self-adaptive element learning method in low-resource speech recognition is characterized by comprising the following steps:

1) Classifying a plurality of languages used for training according to the languages, and dividing a training set and a verification set;

2) Dividing the voice data of the training set and the verification set of each language type into a supporting set and a query set;

3) Counting the texts of all the voice data, and manufacturing a training dictionary;

4) Writing codes to construct a voice recognition model consisting of a transducer model, a VGG-CNN network and an Adapter module;

5) Model pre-training stage is carried out;

5.1 In each training iteration, for the training set voice data of each language, firstly extracting k data from the supporting set of the training set, extracting features through a VGG-CNN network, then sending the features into a transducer model for training, and updating model parameters of the current iteration turn;

5.2 Extracting k data from the query set of the corresponding language, extracting the characteristics through the VGG-CNN network, sending the characteristics into the recognition performance of the transducer model test model to obtain the corresponding query set loss, sending the characteristics into the Adapter module, and obtaining the variance required by adjusting the loss weight after the average value and the logarithm are taken;

5.3 Utilizing the obtained variance to adjust the query set loss of the corresponding language in the current turn through a formula;

5.4 After the training operation for the single language is completed, initializing the parameters of the model to the numerical value at the beginning of the current training round;

6) Repeating the steps 5.1) to 5.4) for training sets and verification sets of other languages, further obtaining the adjusted query set loss of all the languages for training in the current iteration round, and summing the query set loss to obtain an average value;

7) Gradient descent updating is carried out on all parameters of the model at the beginning of the current iteration round of the model by utilizing the query set loss average value operated in the step 6);

8) After updating the model parameters, taking the current parameters of the model as the model initialization parameters of the next iteration round, and repeating the steps 5) to 7);

9) After each time of training a certain Epoch, testing a current trained model by using voice data of a verification set, storing model parameters in the process of optimal performance after the test, and ending the training process when the model parameters stored for a certain time are the same optimal values after a plurality of tests;

10 Preparing training set, verification set and test set of language to be identified;

11 Training the model stored in the step 9) by using a training set of languages to be identified;

11.1 In each iteration round, extracting voice data of one Batch number from a training set, extracting features through a VGG-CNN network, sending the features into a recognition performance of a transducer model test model to obtain corresponding loss, sending the features into an Adapter module, and obtaining variances required by adjusting loss weights after average and logarithmic operations;

11.2 Using the obtained variance to adjust the training loss of the required identification language in the current turn through a formula;

11.3 Updating the model parameters using the loss after alignment obtained in step 11.2);

12 After each Epoch is trained, testing the current trained model by utilizing the voice data of the target language verification set, and storing the model with optimal performance;

13 After repeatedly training a plurality of epochs, ending the training process aiming at the target language, thus obtaining the voice recognition model with excellent recognition performance.

Images