CN111640425A - Model training and intention recognition method, device, equipment and storage medium - Google Patents

Abstract

The present application discloses a method, device, device and storage medium for model training and intent recognition, and relates to the technical field of artificial intelligence. Among them, the model training method is: according to the training task data set, perform precipitation training on the underlying network of the pre-training model at least twice to obtain a strengthened model of the pre-training model; wherein, the training objects of each precipitation training include at least the underlying network and prediction. layer network, including successively decreasing middle and high-level networks; take at least two networks in the reinforcement model as target networks, and build a distillation model according to the target network, wherein the target network includes a feature recognition network and the prediction layer network; feature recognition The network includes at least the underlying network; by strengthening the target network of the model, the target knowledge of the training task data set is extracted; according to the target knowledge and the training task data set, the distillation model is trained to obtain the target learning model, so as to improve the prediction efficiency of the target learning model and accuracy.

Description

A method, device, device and storage medium for model training and intent recognition

technical field

The embodiments of the present application relate to the field of computer technology, in particular to artificial intelligence technology.

Background technique

With the development of artificial intelligence technology, deep learning models are more and more widely used in the field of human-computer interaction. As a kind of deep learning model, the pre-training model has a complex structure and huge model parameters, so the pre-training model may take a long time and slow in the running phase. In order to improve the response speed of the pre-training model, the prior art usually requires developers to manually select a network layer with a smaller weight value from the pre-training model, and cut them out from the pre-training model to realize the pre-training model. Compression to reduce the structural complexity of the pre-trained model. However, the pre-training model tailored in this way is greatly affected by human factors and has low accuracy, which seriously affects the effect of human-computer interaction and needs to be improved urgently.

SUMMARY OF THE INVENTION

Provided are a model training and intent recognition method, apparatus, device and storage medium.

According to a first aspect, a model training method based on knowledge distillation is provided, the method comprising:

According to the training task data set, the underlying network of the pre-training model is subjected to precipitation training at least twice to obtain an enhanced model of the pre-training model; wherein, the training objects of each pre-training training include at least the underlying network and the prediction layer a network, and includes successively decreasing middle and high-level networks, and the pre-training model includes the bottom-layer network, at least one of the middle and high-level networks, and the prediction layer network from bottom to top;

At least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks, wherein the target network includes a feature identification network and the prediction layer network; the feature identification network includes at least the the underlying network;

Extract the target knowledge of the training task data set through the target network of the reinforcement model;

According to the target knowledge and the training task data set, the distillation model is trained to obtain a target learning model.

According to a second aspect, an intent recognition method is provided, the method comprising:

Obtain user voice data collected by human-computer interaction equipment;

Input the user voice data into the target learning model to obtain the user intent recognition result output by the target learning model; wherein, the target learning model is trained based on the model training method based on knowledge distillation described in any embodiment of the present application to determine;

A response result of the human-computer interaction device is determined according to the user intention recognition result.

According to a third aspect, a model training device based on knowledge distillation is provided, the device comprising:

The precipitation training module is used to perform precipitation training on the underlying network of the pre-training model at least twice according to the training task data set to obtain the enhanced model of the pre-training model; wherein, the training objects of each pre-training training at least include all the bottom layer network and the prediction layer network, and include successively decreasing middle and high level networks, and the pre-training model includes the bottom layer network, at least one of the middle and high level networks and the prediction layer network from bottom to top;

a distillation model building module, configured to use at least two networks in the enhanced model as target networks, and build a distillation model according to the target networks, wherein the target network includes a feature recognition network and the prediction layer network; The feature identification network includes at least the underlying network;

a target knowledge extraction module, used for extracting the target knowledge of the training task data set through the target network of the reinforcement model;

The distillation model training module is used for training the distillation model according to the target knowledge and the training task data set to obtain a target learning model.

According to a fourth aspect, an intention recognition device is provided, the device comprising:

A voice data acquisition module, used to acquire user voice data collected by human-computer interaction equipment;

An intent recognition module, used to input the user voice data into a target learning model to obtain a user intent recognition result output by the target learning model; wherein, the target learning model is based on the knowledge-based method described in any embodiment of the present application The model training method of distillation is determined by training;

A response result determination module, configured to determine a response result of the human-computer interaction device according to the user intention identification result.

According to a fifth aspect, an electronic device is provided, the electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the knowledge-based implementation of any of the embodiments herein Distilled model training method or intent recognition method.

According to a sixth aspect, a non-transitory computer-readable storage medium having computer instructions stored thereon is provided. The computer instructions are used to cause the computer to execute the knowledge distillation-based model training method or the intent recognition method described in any embodiment of the present application.

The technology according to the embodiments of the present application solves the problem of low accuracy of manual compression of the pre-training model in the prior art, and can automatically compress and train a high-precision target learning model at low cost, so as to improve the effect of human-computer interaction.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present application. in:

1A is a flowchart of a model training method based on knowledge distillation provided according to an embodiment of the present application;

1B is a schematic diagram of a network structure of a pre-training model provided according to an embodiment of the present application;

2 is a flowchart of another model training method based on knowledge distillation provided according to an embodiment of the present application;

3 is a flowchart of another model training method based on knowledge distillation provided according to an embodiment of the present application;

Fig. 4-Fig. 5 is the flow chart of two kinds of model training methods based on knowledge distillation provided according to the embodiment of the present application;

6A is a flowchart of another model training method based on knowledge distillation provided according to an embodiment of the present application;

6B is a schematic structural diagram of the principle of training a distillation model according to an embodiment of the present application;

7 is a flowchart of a model training method based on knowledge distillation provided according to an embodiment of the present application;

FIG. 8 is a flowchart of an intent recognition method provided according to an embodiment of the present application;

9 is a schematic structural diagram of a video processing apparatus provided according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of an intention recognition device provided according to an embodiment of the present application;

FIG. 11 is a block diagram of an electronic device used to implement the knowledge distillation-based model training method or the intent recognition method according to the embodiment of the present application.

Detailed ways

Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

FIG. 1A is a flowchart of a model training method based on knowledge distillation provided according to an embodiment of the present application; FIG. 1B is a schematic diagram of a network structure of a pre-trained model provided by an embodiment of the present application. This embodiment is applicable to a situation where a pre-training model with a complex network structure is compressed and trained into a target learning model with a simple network structure based on the knowledge distillation technology. This embodiment may be implemented by a knowledge distillation-based model training apparatus configured in an electronic device, and the apparatus may be implemented by software and/or hardware. As shown in Figures 1A-1B, the method includes:

S101, according to the training task data set, perform precipitation training on the underlying network of the pre-training model at least twice to obtain an enhanced model of the pre-training model.

The task training data set in the embodiment of the present application may be a prediction task that needs to be performed according to the pre-training model, and sample data related to the prediction task is obtained as the training task data set. For example, if the prediction task that the pre-training model needs to perform is to perform intent recognition on user voice data in shopping platform A, then all historical user voice data in shopping platform A can be obtained at this time, and related processing (such as tagging, deleting, etc.) can be performed. After processing invalid data, etc.), the training task dataset corresponding to the prediction task is obtained.

The pre-training model in this embodiment of the present application may be a high-precision model that is built based on a deep learning architecture and has been trained with massive data and can perform a certain learning task. The pre-training model usually has a deep network layer, Each layer of the network has a wide dimension and many model parameters. The pre-training model may be trained by the user using a large amount of sample data, or may be directly obtained from the pre-training model database, which is not limited in this embodiment. Optionally, the pre-training model may include a bottom-up network, at least one middle and high-level network, and a prediction layer network from the bottom up. The bottom layer network and the middle and high-level networks are used for feature identification; the prediction layer network is used for task prediction according to the identified features. Among them, the bottom network is usually used to identify simple features, and the middle and high-level networks are usually used to abstract complex features from simple features. For example, if the pre-trained model is a bert model for intent recognition, the underlying network of the bert model is usually used to identify simpler grammatical features; the middle and high-level networks are usually used to abstract complex features from grammatical features. The prediction layer network is used for task prediction based on the features identified by the underlying network and the middle and high-level networks. Optionally, the pre-training model in this embodiment of the present application may be a bert model.

Exemplarily, the pre-training model 1 shown in FIG. 1B is composed of 12 network layers, wherein the first network layer to the third network layer are the bottom network 10, the fourth network layer to the 11th network layer are the middle and high-level networks 11, The twelfth network layer is the prediction layer network 12, wherein the middle and high-level network 11 further includes a middle-level network 110 (ie, the 4th network layer to the 7th network layer) and a high-level network 111 (ie, the 8th network layer to the 11th network layer).

Optionally, under normal circumstances, the complex features abstracted by the mid-to-high-level network of the pre-training model have a low correlation with the prediction task itself. To accurately complete the prediction task, the underlying network is mainly relied on. Therefore, this operation can perform multiple precipitation training on the underlying network of the pre-training model, and continuously adjust the training object (ie, the network layer that needs to be trained in the pre-training model) during the process of multiple precipitation training. Among them, the training objects of each precipitation training include at least the bottom network and the prediction layer network, and include the gradually decreasing middle and high level networks. That is to say, although this embodiment focuses on the precipitation training of the underlying network of the pre-training model, in order to ensure the accuracy of the training results, the objects of each training must at least include the underlying network and the prediction layer network. , and with the increase of the number of precipitation training, the number of layers of the middle and high-level networks included in the training object shows a decreasing trend. For example, assuming that the underlying network 10 of the pre-training model 1 shown in FIG. 1B is subjected to five precipitation trainings, the training objects of the five precipitation trainings all include the underlying network 10 and the prediction layer network 12. For the middle layer network, the first The training objects of the second precipitation training may include all network layers; the training objects of the second precipitation training may be reduced to include the 4th network layer to the ninth network layer; the training objects of the third precipitation training may be reduced to include the fourth network layer again. From the network layer to the seventh network layer, it decreases in turn. By the fifth precipitation training, the training object may have been reduced to not include the middle and high-level network 11 .

Specifically, in this step, when the underlying network of the pre-training model is deeply trained at least twice according to the training task data set, a part of the training task data set may be input to the pre-training model each time, and this part of the training task data set may be used for the training task. The second training object is subjected to a precipitation training, and then the pre-trained model after multiple precipitation training is used as a reinforcement model. Since the middle and high-level networks in the training object in this step decrease successively with the increase of training times, this operation can perform more accurate training updates on the underlying network as the number of precipitation training increases, so that the parameters of the underlying network become more and more more precise. That is to say, compared with the pre-training model, the network structure of the reinforcement model in this embodiment has not changed. If the pre-training model is the bert model, the reinforcement model after precipitation training is also the bert model, but the network parameters of the underlying network are more Precise.

S102, at least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks.

Wherein, the target network may be a network required to complete the current prediction task selected from the networks included in the reinforcement model. The target network includes a feature identification network and a prediction layer network; the feature identification network is a network used for feature identification, and the feature identification network in this embodiment of the present application includes at least an underlying network. Optionally, in addition to the underlying network, the feature identification network may also include part or all of the middle and high-level networks, which is not limited in this embodiment.

Optionally, in this embodiment, at least two networks may be selected from the reinforcement model as the target network, wherein, if two networks are selected, the two networks are the bottom layer network and the prediction layer network of the reinforcement model, and the target network at this time. The feature recognition network in the network only includes the bottom network; if three or more networks are selected, the remaining networks can be selected from the middle and high-level networks on the basis of selecting the bottom network and the prediction layer network. In addition to the underlying network, the characteristic network also includes at least one middle and high-level network. It should be noted that whether to use the middle and high-level network of the enhanced model as the feature recognition network of the target network can be determined by combining factors such as the actual prediction task and the type of target knowledge to be extracted subsequently. This embodiment is not limited.

Optionally, when the distillation model is constructed according to the target network in this embodiment, a distillation model that also includes the target network may be constructed according to the target network. It should be noted that the network type of the target network of the distillation model constructed in this step should be the same as the target network type of the reinforcement model. Specifically, the distillation model also includes the prediction layer network and the feature recognition network. Regarding the feature recognition network, if the feature recognition network selected from the reinforcement model only includes the underlying network, the feature recognition network of the constructed distillation model will also only contain Including the bottom network; if the feature recognition network selected from the reinforcement model includes not only the bottom network, but also the middle network in the middle and high-level networks, the feature recognition network of the constructed distillation model also includes the bottom network and the middle network.

Optionally, when the distillation model is constructed according to the target network in this operation, the network layer structure of the target network of the reinforcement model can be combined to construct a distillation model with the same structure as the reinforcement model, that is, the isomorphism model of the reinforcement model. For example, if the reinforcement model is a bert model, the constructed distillation model is a bert model that only includes the target network structure in the reinforcement model. The structure of the constructed distillation model can also be different from that of the reinforcement model, but it also includes the target network type of the reinforcement model, that is, the heterogeneous model of the reinforcement model. For example, the reinforcement model is a bert model, and the constructed distillation model is a CNN model, but the CNN model also includes the same type of target network as the reinforcement model. The specific method of how to construct the isomorphic or heterogeneous distillation model will be described in detail in the following embodiments.

It should be noted that the distillation model constructed by this operation can be a machine learning model or a small model based on a neural network. The distillation model is characterized by few parameters, block inference speed, and good portability.

S103, extract the target knowledge of the training task data set by strengthening the target network of the model.

Among them, the target knowledge can be the result obtained after the target network in the reinforcement model processes the training task data set, and the target knowledge is used for subsequent injection into the distillation model as a supervision signal during the training of the distillation model.

Optionally, when extracting the target knowledge of the training task data set in this step, the training task data set may be used as the input of the reinforcement model, the first data feature representation output by the feature recognition network of the reinforcement model is obtained, and the prediction of the reinforcement model is obtained. The first prediction probability representation output by the layer network; and the acquired first data feature representation and the first prediction probability representation are used as the target knowledge of the training task data set. Specifically, the training task dataset may be divided into multiple copies according to a preset size, such as batch_size. Then, input the divided training task data into the reinforcement model, run the reinforcement model, and obtain the feature representation output by the feature recognition network of the reinforcement model as the first data feature representation. Among them, if the feature recognition network has only the underlying network, the first data feature representation is only the feature representation output by the underlying network; if the feature recognition network includes the underlying network and a part of middle and high-level networks, the first data feature representation includes not only the features output by the underlying network representation, and also includes the feature representation of the output of the high-level network in this part. Obtain the feature representation of the network output of the prediction layer of the reinforcement model, such as the prediction probability value, as the first prediction probability representation, and then use the acquired first data feature representation and the first prediction probability representation as the target corresponding to the training task data input this time. Knowledge.

S104, train the distillation model according to the target knowledge and the training task data set to obtain a target learning model.

Optionally, in this operation, the target knowledge obtained in S103 is used as a supervision signal for training the distillation model, and based on the training task data set, the distillation model is induced to train, so that the target knowledge can be transferred to the distillation model during the training process, so as to improve the performance of the distillation model. The prediction task that makes the distillation model learn to strengthen the model. Specifically, this step may be to calculate the soft supervision label according to the data feature representation and prediction probability representation in the target knowledge, and the data feature representation and prediction probability representation obtained by processing the training task data with the distillation model, and process the training task data according to the distillation model. As a result, the hard-supervised labels are calculated, and then the software-supervised labels are combined with the hard-supervised labels to perform distillation training on the distillation model with less training task data and more efficient learning efficiency. How to calculate the hard supervised labels and software degree labels, and how to perform distillation training according to these two supervised labels will be described in detail in the following embodiments.

Optionally, in this step, the distillation model is trained, and the trained distillation model is the target learning model. Since the target learning model is obtained by performing knowledge distillation on the pre-training model, the target learning model can accurately execute the pre-training model. Compared with the pre-training model, the target learning model has a simpler structure, so it takes less time and responds faster when performing the prediction task.

Optionally, in this embodiment, after the target learning model is obtained through training, the target learning model can be deployed in the actual human-computer interaction field to perform online task prediction. Preferably, if the pre-training model and the target learning model are models used for intention recognition, correspondingly, after the distillation model is trained according to the target knowledge and the training task data set, and the target learning model is obtained, this application implements For example, the target learning model may also be deployed in the human-computer interaction device, so as to perform intent recognition on the user's speech data obtained in real time by the human-computer interaction device. Specifically, after the target learning model is deployed in the human-computer interaction device, after the human-computer interaction device obtains the user's voice data, it will transmit the user's voice data to the target learning model, and the target learning model will respond to the input user voice. The data is used for intention recognition, and the intention recognition result is fed back to the human-computer interaction device. The human-computer interaction device will generate a response result corresponding to the user's voice data according to the intention recognition result of the target learning model, and feed it back to the user. The solution of the embodiment of the present application is to train the target learning model by means of knowledge distillation. Compared with the pre-training model, the network structure is simpler, and the prediction effect can be approximated to the complex pre-training model, which can achieve fast and accurate Intent recognition is performed to meet the real-time response requirements of human-computer interaction devices.

In the technical solution of this embodiment, according to the training task data set, the bottom network, the prediction layer network and the successively decreasing middle and high-level networks are used as training objects, and the bottom network of the pre-training model is subjected to precipitation training at least twice to obtain a strengthened model; From the target network identified in the reinforcement model, the network builds the distillation model. Then, by strengthening the target network of the model, the target knowledge of the training task data set is extracted; based on the extracted target knowledge and the training task data set, the distillation model is trained to obtain the target learning model. In this embodiment, the underlying network of the pre-training model is subjected to multiple precipitation training in the manner of successively decreasing the middle and high-level networks, which can make the parameters of the underlying network of the pre-training model more accurate. Subsequent at least build a distillation model based on the accurate underlying network and prediction layer network after precipitation, and perform distillation training on the distillation model based on the extracted target knowledge, so that the target learning model distilled from the pre-training model simplifies the network structure at the same time. , retains the prediction accuracy of the pre-trained model, and also improves the generalization ability of the model. And the whole distillation process is not affected by human factors. Deploying the target learning model into human-computer interaction equipment can perform tasks quickly and accurately to meet the real-time response requirements of human-computer interaction equipment.

Optionally, the pre-training model in the embodiment of the present application is a model that has been trained to perform a certain prediction task. When the prediction task covers a relatively wide field, the pre-training model may be able to perform task prediction in multiple fields. But for one of these fields, the prediction effect may not be very good. For example, if the pre-trained model is a model for intent recognition, it can perform intent recognition on user voices in many fields such as shopping, business processing, and smart furniture control, but for some specific fields, the prediction effect may not be effective. not very accurate. In view of this situation, in this embodiment, before performing precipitation training on the underlying network of the pre-training model at least twice according to the training task data set, perform domain training on the pre-training model according to the training domain data set, and update the pre-training model. Train the model.

Specifically, the training domain data set may be the work domain to be deployed based on the pre-training model, and sample data related to this domain is specially obtained as the training domain data set. For example, if the pre-training model needs to execute the intent of the user's voice in the shopping domain At this time, all the voice data of each shopping platform can be processed by related processing (such as tagging, deleting invalid data, etc.) to obtain a training field data set corresponding to the field. Input the training domain data set into the pre-training model to update and train the pre-training model for this domain, and fine-tune the parameters of the pre-training model, so that the updated pre-training model can more accurately perform prediction tasks in this domain. In this embodiment, domain training is performed on the pre-training model according to the training domain data set, and after the pre-training model is updated, the precipitation training operation of S101 is performed on the updated pre-training model. The advantage of this setting is that the pre-training is greatly improved. The prediction accuracy of the model in the domain of its prediction task. It provides a guarantee for the subsequent distillation of an accurate target learning model based on the pre-training model.

2 is a flowchart of another model training method based on knowledge distillation provided according to an embodiment of the present application; the present embodiment is further optimized on the basis of the above-mentioned embodiment, and provides a data set according to a training task, The specific situation of performing precipitation training at least twice for the underlying network of the pre-trained model is introduced. As shown in Figure 2, the method includes:

S201: Divide the training task data set to determine multiple training data subsets.

Optionally, this operation may be to divide the training task data set into multiple training data subsets according to a preset precipitation strategy, such as the number of network layers for each knowledge extraction. For example, if the pre-training model is the model shown in FIG. 1B , and the precipitation strategy is to extract the knowledge of one layer of the network at a time, then the training task data set can be divided into 12 parts at this time. Among them, the number of divisions of the training data subset is less than or equal to the total number of layers of the pre-training model. For example, when the total number of layers of the pre-training model is N, the number K of the training data subsets divided in this step may be equal to half of the total number of layers N. Optionally, the number of training data in the divided training data subsets may be the same or different, which is not limited in this embodiment.

S202, according to the set precipitation training times, determine the training objects corresponding to each training data subset.

Among them, the training object may be the network layer that needs to be trained in the pre-training model each time the precipitation training is performed. The training objects corresponding to each training data subset in the embodiments of the present application are different. Specifically, the training objects corresponding to each training data subset include the underlying network, the middle and high-level networks, and the prediction layer network of the pre-training model, and the number of layers of the middle and high-level networks included is inversely proportional to the order of precipitation training. And the middle and high-level networks included in each training object are network layers adjacent to the underlying network and continuous upwards. That is to say, in the training objects corresponding to each training data subset, the underlying network and the prediction layer network remain unchanged, and the number of layers in the middle and high-level networks moves backward with the precipitation training order corresponding to the training data subsets. The number of layers decreases from top to bottom. Optionally, and based on the increase of the number of precipitation training, the number of layers of the middle and high-level networks included in the training object is decreased to zero. In this way, as the number of precipitation training increases, only the underlying network is finally updated and trained.

Optionally, in this embodiment, when determining the training objects corresponding to each training data subset, the underlying network and the prediction layer network are unchanged, and the total number of layers of the pre-training model and the corresponding training data subsets can be used. The precipitation training sequence is determined by determining the number of middle and high-level network layers in the training object corresponding to each training data subset. For example, if the total number of layers of the pre-training model is N, and the precipitation training sequence corresponding to a certain training data subset is the kth time, then the highest layer of the middle and high-level network included in the training object corresponding to the training data subset The number is S=N-2*k, that is, the middle and high-level networks below the S layer are the training objects corresponding to the training data set.

S203 , according to each training data subset, perform a precipitation training on the training objects corresponding to the training data subset in the pre-training model, to obtain an enhanced model of the pre-training model.

Optionally, after determining the training objects corresponding to each of the divided training data subsets, each training data subset may be sequentially input into the pre-training model according to the precipitation training sequence corresponding to each training data subset. , using the input training data subset to train each network layer corresponding to the current training object in the pre-training model, and update the parameters of each network layer corresponding to the training object. Since the number of layers of the middle and high-level networks in the training objects corresponding to the training data subsets in this embodiment gradually decreases with the increase of the number of precipitation training, the updated parameters of the mid- and high-level networks in the process of multiple precipitation training will be more and more The less and less, the training process is gradually concentrated to the underlying network. After multiple precipitation training, the underlying network of the pre-trained model will become more and more accurate. At this time, the pre-trained model after multiple precipitation training can be used as a reinforcement model. .

S204, at least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks.

Wherein, the target network includes a feature identification network and the prediction layer network; the feature identification network includes at least a bottom layer network.

S205, extract the target knowledge of the training task data set by strengthening the target network of the model.

S206, train the distillation model according to the target knowledge and the training task data set to obtain a target learning model.

In the technical solution of this embodiment, the training task data set is divided into multiple training data subsets, and each training data subset is determined based on the principle that the number of layers of the middle and high-level networks in the training object is inversely proportional to the order of precipitation training For the training objects, according to each training data subset after division, a precipitation training is performed on the training objects to obtain the reinforcement model. According to the reinforcement model, the network constructs a distillation model and extracts target knowledge; then, based on the extracted target knowledge and training task data set, the distillation model is trained to obtain a target learning model. Based on the principle that the number of layers of the middle and high-level networks in the training objects is inversely proportional to the sequence of precipitation training, the present embodiment determines the training objects for each precipitation training, so that the parameters of the underlying network of the pre-trained model after multiple precipitation training are more Precise. It provides a new idea for the knowledge precipitation operation in the knowledge distillation process. It provides a guarantee for the subsequent operation of the knowledge distillation training target learning model.

Fig. 3 is a flowchart of another model training method based on knowledge distillation provided according to an embodiment of the present application. This embodiment is further optimized on the basis of the above-mentioned embodiment, and provides the bottom layer of the pre-training model. In the process of multiple precipitation training of the network, the specific situation of when to get the reinforcement model of the pre-trained model is introduced. As shown in Figure 3, the method includes:

S301, according to the training task data set, successively perform precipitation training on the underlying network of the pre-training model.

It should be noted that, in this step, the specific implementation of successive precipitation training for the underlying network of the pre-training model has been described in detail in the above embodiments, and will not be repeated here.

S302, test the pre-trained model after precipitation training according to the test task data set.

The test task data set may be test data used to test whether the pre-trained model after precipitation training can accurately complete the prediction task. Optionally, the sample data related to the prediction task can be obtained according to the prediction task to be performed by the pre-training model, and then divided into two parts, one part is used as the training task data set of the embodiment of the present application, and the other part is used as the training task data set of the embodiment of the present application. The test task dataset of the embodiment of the present application.

Optionally, in this embodiment, the test task data set may be input into the pre-training model after multiple precipitation training in S301, to obtain the prediction result output by the pre-training model after precipitation training based on the test task data, and finally according to the test task. The real labels in the data analyze the prediction results, calculate the evaluation index value that characterizes whether the output result of the pre-training model after multiple precipitation training is accurate, and use the evaluation index value as the test result. Optionally, the evaluation index value may be determined according to the prediction task, for example, may be the accuracy rate, precision rate, and recall rate of the output results of the pre-training model after multiple precipitation training.

Optionally, in order to ensure the accuracy of the test results, in this embodiment, multiple sets of test task data sets may be used to perform multiple tests on the pre-trained model after precipitation training, and the final test result may be determined according to the multiple test results.

S303, if the test result satisfies the precipitation end condition, the pre-trained model after the precipitation training is used as the reinforcement model.

Wherein, the precipitation end condition may be judging whether the pre-training model after multiple precipitation training satisfies the judgment condition as a reinforcement model. Specifically, it may be an index threshold corresponding to the evaluation index value in the test result.

Optionally, in this embodiment, the pre-training model after the precipitation training can be tested in S302, and the obtained test result (that is, the evaluation index value) is compared with the index threshold in the precipitation end condition, if the evaluation index value satisfies the index threshold. , it means that the test result satisfies the precipitation end condition, and the pre-training model after precipitation training can be used as the reinforcement model; if the test result does not meet the precipitation end condition, it is necessary to return to S301 to continue according to the training task data set, for the pre-training model The underlying network performs precipitation training successively until the test result meets the precipitation end condition.

S304, at least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks.

Wherein, the target network includes a feature identification network and the prediction layer network; the feature identification network includes at least a bottom layer network.

S305, extract the target knowledge of the training task data set by strengthening the target network of the model.

S306, train the distillation model according to the target knowledge and the training task data set to obtain a target learning model.

In the technical solution of this embodiment, according to the training task data set, the underlying network of the pre-training model is subjected to multiple precipitation training, and then the pre-trained model after precipitation training is tested according to the test task data set. The pre-trained model after precipitation training is used as a reinforcement model. Then, according to the reinforcement model, the network constructs a distillation model and extracts target knowledge; and based on the extracted target knowledge and training task data set, the distillation model is trained to obtain a target learning model. In this embodiment, the pre-training model after knowledge precipitation is tested to determine whether the knowledge precipitation achieves the expected effect of precipitation training. Only when the expected effect is achieved can it be used as a reinforcement model, which ensures the accuracy of the underlying network parameters of the obtained reinforcement model. sex. It provides a guarantee for the subsequent operation of the knowledge distillation training target learning model.

Optionally, the above-mentioned embodiment introduces the process of determining when the reinforcement model of the pre-training model is obtained in the process of performing multiple precipitation training on the underlying network of the pre-training model. Similarly, this embodiment is based on the target knowledge and In the training task data set, a similar method can also be used in the process of training the distillation model to determine whether the distillation model has been trained, and the target learning model can be obtained. Specifically: in the embodiment of the present application, when performing training on the distillation model according to the target knowledge and the training task data set, and obtaining the target learning model, the specific execution is: according to the target knowledge and the training task data set, the distillation model is trained; According to the test task data set, the trained distillation model is tested; if the test result meets the training end condition, the trained distillation model is used as the target learning model. It should be noted that the process of testing the trained distillation model according to the training task data set is similar to the process of testing the pre-trained model after precipitation training according to the training task data set described in the above embodiment. The test task data set is input into the trained distillation model, and the evaluation index value is calculated according to the predicted result of the output of the trained distillation model and the true label of the test task data set. If the evaluation index value meets the index threshold in the training end condition, It means that the test result of the trained distillation model satisfies the training end condition, and the trained distillation model can be used as the target learning model. The advantage of this setting is to test the trained distillation model to determine whether the task prediction accuracy of the trained distillation model achieves the expected effect. Only when the expected effect is achieved can it be used as the final target learning model. The accuracy of the target learned model distilled by the knowledge distillation technique.

4 to 5 are flowcharts of two knowledge distillation-based model training methods provided according to the embodiments of the present application. On the basis of the above-mentioned embodiments, the present embodiment further optimizes the An introduction to two specific implementations of the distillation model.

Optionally, Figure 4 shows an implementable implementation of constructing a distillation model with the same structure as the reinforcement model according to the target network, specifically:

S401, according to the training task data set, perform precipitation training on the underlying network of the pre-training model at least twice to obtain an enhanced model of the pre-training model.

Among them, the training objects of each precipitation training include at least the bottom network and the prediction layer network, and include the middle and high-level networks that gradually decrease, and the pre-training model includes the bottom layer network, at least one of the middle and high-level networks and the prediction layer network from bottom to top.

S402, at least two networks in the reinforcement model are used as target networks, and network structural blocks of the target network are acquired.

Wherein, the target network includes a feature identification network and a prediction layer network of the enhanced model; the feature identification network includes at least a bottom network of the enhanced model. Optionally, a part or all of the mid- and high-level networks of the enhanced model may also be included. Since the network structure of the distillation model constructed in this embodiment is simpler than that of the reinforcement model, in general, the feature recognition network of the target network in this embodiment of the present application usually does not include or only includes a small number of middle and high-level networks. The network structure block may be obtained by encapsulating the network structure of one or more network layers in the reinforcement model. For example, it is assumed that the reinforcement model in this embodiment is obtained by precipitation training the pre-training model shown in FIG. 1B . , the network structure of the enhanced model should also be shown in Figure 1B. At this time, the network structure of the first network layer to the third network layer in Figure 1B can be encapsulated as the network structure block of the underlying network 10; The network structure to the 7th network layer is encapsulated as the network structure block of the middle network layer 110; the network structure of the 8th network layer to the 11th network layer is encapsulated as the network structure block of the high-level network 111; the network structure of the 12th network layer is encapsulated It is the network structure block of the prediction layer network 12 .

Optionally, if a distillation model with the same structure as the reinforcement model is to be constructed, the network structure block corresponding to the target network in the reinforcement model can be obtained after the target network is selected from the reinforcement model. For example, taking the underlying network and the prediction layer network in the reinforcement model as the target network, the network structure block of the underlying network and the network structure block of the prediction layer network can be used as the network structure block of the target network.

S403, construct a distillation model with the same structure as the reinforcement model according to the acquired network structure blocks.

Optionally, since the target network corresponds to at least two networks in the reinforcement model, the acquired network structure blocks are also network structure blocks of at least two networks. The order from the bottom to the top in the model is arranged, and the output of the lower network structure block is used as the input of the upper network structure block adjacent to it, so as to form a new model composed of the target network, the new model is The constructed distillation model.

For example, assuming that S402 acquires the network structure blocks of the underlying network 10 and the prediction layer network 12 in FIG. 1B , since the underlying network 10 is located below the prediction layer network 12 , the network structure block of the underlying network 10 can be set at this time. Below the network structure block of the prediction layer network 12, and connect the output of the network structure block of the underlying network 10 with the input of the network structure block of the prediction layer network 12, thereby generating a network structure block of the underlying network 10 and the prediction layer network. A distillation model composed of 12 network building blocks. Similarly, if S402 obtains the network structure blocks of the bottom layer network 10, the middle layer network 110 and the prediction layer network 12, then at this time, the network structure block of the bottom layer network 10 may be located at the bottom, and the network structure block of the middle layer network 110 may be located at the bottom. Located in the middle, the network structure block of the prediction layer network 12 is located at the top, the output of the network structure block of the bottom network 10 is connected to the input of the network structure block of the middle layer network 110, and the output of the network structure block of the middle layer network 110 is connected to the prediction layer. The input of the network structure blocks of the network 12 , thereby generating a distillation model composed of the network structure blocks of the bottom layer network 10 , the network structure blocks of the middle layer network 110 , and the network structure blocks of the prediction layer network 12 .

S404, extract the target knowledge of the training task data set by strengthening the target network of the model.

S405, train the distillation model according to the target knowledge and the training task data set to obtain a target learning model.

Optionally, Fig. 5 shows an implementable manner of constructing a distillation model with a structure different from that of the reinforcement model according to the target network, specifically:

S501 , according to the training task data set, perform precipitation training on the underlying network of the pre-training model at least twice to obtain an enhanced model of the pre-training model.

Among them, the training objects of each precipitation training include at least the bottom network and the prediction layer network, and include the middle and high-level networks that gradually decrease, and the pre-training model includes the bottom layer network, at least one middle and high-level network and the prediction layer network from bottom to top.

S502, at least two networks in the reinforcement model are used as target networks.

Optionally, the process of selecting the target network from the reinforcement model has been introduced in the foregoing embodiment, and is not repeated in this embodiment.

S503, according to the target network, a neural network model with a structure different from that of the reinforcement model is selected as the distillation model.

Among them, the output layer network of the neural network model is of the same type as the prediction layer network in the target network, and the non-output layer network of the neural network model is of the same type as the feature recognition network in the target network. The so-called prediction layer network type means that the type of the network belongs to the prediction type, that is, the type of task prediction. The types of so-called feature recognition networks include: bottom-level networks, middle-level networks, and high-level networks.

Optionally, since the structure of the distillation model constructed in this embodiment is different from that of the reinforcement model, at this time, a neural network model with a simple structure that can be used to implement the prediction task can be selected as the distillation model according to requirements. Among them, the structure of the neural network model that can be selected as the distillation model is usually relatively simple and the number of layers is small, but the output layer of the neural network model needs to be consistent with the type of the prediction layer network in the target network, and the non-output layer network and the feature in the target network. Identify the same type of network. That is, the output layer of the neural network model needs to be a network that can perform task prediction, and its non-output layer needs to be consistent with the type of the feature recognition network of the target network. For example, if the type of the feature recognition network of the target network is the underlying network, then The type of the non-output layer of the neural network model should also be the bottom network; if the type of the feature recognition network of the target network is the bottom network and the middle network, the type of the non-output layer of the neural network model should also be the bottom network and the middle network. .

The distillation model constructed in this step has few structural units and few layers, so it is usually related to a heterogeneous model with a complex structural reinforcement model. For example, assuming that the reinforcement model is the bert model, the CNN model can be selected as the distillation model at this time.

S504, extract the target knowledge of the training task data set by strengthening the target network of the model.

S505, the distillation model is trained according to the target knowledge and the training task data set to obtain the target learning model.

The technical solutions of the embodiments of the present application provide that in the process of training the target learning model of the pre-trained model based on the knowledge distillation technology, two distillations with the same or different structure as the reinforcement model are constructed according to the target network of the reinforcement model after precipitation training. The specific implementation of the model. If a distillation model with the same structure as the reinforcement model is constructed, since the distillation model retains the network structure blocks of the reinforcement model, it is easier for the isomorphic distillation model to obtain the prediction effect of the reinforcement model through distillation training; if a distillation model with the same structure as the reinforcement model is constructed When the heterogeneous distillation model can learn different features from the reinforcement model, the generalization ability of the model can be improved. The embodiments of the present application can be selected according to actual needs, and have strong flexibility.

FIG. 6A is a flowchart of another model training method based on knowledge distillation provided according to an embodiment of the present application; FIG. 6B is a schematic structural diagram of the principle of training a distillation model according to an embodiment of the present application. In this embodiment, further optimization is performed on the basis of the above-mentioned embodiment, and the specific situation of training the distillation model according to the target knowledge and the training task data set is introduced. As shown in Figures 6A-6B, the method includes:

S601, according to the training task data set, perform precipitation training on the underlying network of the pre-training model at least twice to obtain an enhanced model of the pre-training model.

Among them, the training objects of each precipitation training include at least the bottom network and the prediction layer network, and include the middle and high-level networks that gradually decrease, and the pre-training model includes the bottom layer network, at least one middle and high-level network and the prediction layer network from bottom to top.

S602, at least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks.

The target network includes a feature identification network and a prediction layer network; the feature identification network includes at least a bottom layer network.

Exemplarily, it is assumed that from the reinforcement model shown in Fig. 6B, the selected target network is the bottom layer network, the middle layer network and the prediction layer network, and the distillation model shown in Fig. 6B is constructed according to these three networks.

S603, extract the target knowledge of the training task data set by strengthening the target network of the model.

Exemplarily, this operation may be to input the training data of a preset size in the training task data set, such as batch_size, into the reinforcement model shown in FIG. 6B, and obtain the feature representation (knowledge_seq _l ) and the middle layer of the output of the underlying network of the reinforcement model. The feature representation (knowledge_seq _m ) output by the network is used as the first data feature representation (knowledge_seq); the feature representation (knowledge_predict) output by the prediction layer network of the enhanced model is obtained as the first prediction probability representation. The first data feature representation and the first prediction probability representation obtained in this step are the extracted target knowledge.

S604, input the training task data set into the distillation model, and determine the soft supervision label and the hard supervision label according to the processing result and target knowledge of the distillation model on the training task data set.

Among them, soft-supervised labels and hard-supervised labels are two kinds of supervision signals in the process of training the distillation model. The soft-supervised labels are calculated based on the extracted target knowledge, and the hard-supervised labels are calculated based on the actual labels in the training task dataset.

Optionally, in this embodiment, the training task data set may be input into the distillation model, and the distillation model may process the input training data set to obtain the output results of each network layer of the distillation model. On the one hand, the output results are used to combine Target knowledge determines soft-supervised labels. On the other hand, it is used to compute hard-supervised labels in combination with relevant information from the training task dataset. The specific determination process includes the following three sub-steps:

S6041: Input the training task data set into the distillation model to obtain the second data feature representation output by the feature recognition network of the distillation model and the second prediction probability representation output by the prediction layer network of the distillation model.

Specifically, after inputting the training data with a preset size in the training task data, such as batch_size, into the distillation model, the output prediction result (that is, the feature representation) of the prediction layer network in the distillation model is obtained as the second prediction probability representation; if the distillation model In the feature recognition network in , there is only the underlying network, then the feature representation output by the underlying network is obtained as the second data feature representation; if the feature recognition network in the distillation model includes some middle and high-level networks in addition to the underlying network, then obtain The feature representation of the bottom layer network and the output of the high-level network in this part is combined as the second data feature representation. Exemplarily, as shown in FIG. 6B , the training task dataset is input into the distillation model. Since the feature recognition network of the target network in FIG. 6B includes the bottom layer network and the middle layer network, the distillation model needs to be processed after the training task dataset. , the feature representation (samll_seq _l ) output by the underlying network and the feature representation (samll_seq _m ) output by the middle layer network are used as the second data feature representation (samll_seq); the feature representation (small_predict) output by the prediction layer network of the distillation model is used as the second prediction probability representation.

S6042, according to the target knowledge, the second data feature representation and the second predicted probability representation, determine the soft supervision label.

Optionally, since the target knowledge is constituted by the first data feature representation and the first prediction probability representation, this embodiment may be based on a preset algorithm, the first data feature representation, the first prediction probability representation, the second data feature representation, the second data feature representation The feature representation and the second predicted probability representation are computed to obtain soft-supervised labels. The specific calculation algorithm is not limited in this embodiment. For example, the mean variance of the first data feature representation and the second data feature representation in the target knowledge can be used as the data feature label; the mean variance represented by the first prediction probability representation and the second prediction probability representation in the target knowledge can be used as the probability prediction label. Then, according to the weight value of the feature recognition network of the enhanced model, label fusion is performed on the data feature label and the probability prediction label to obtain a soft supervision label. In this embodiment, the soft-supervised label is determined based on the output feature representation of the reinforcement model and the distillation model based on the same training task data set, so that the determined soft-supervised label is more accurate, thereby improving the accuracy of the subsequently trained target learning model. .

Specifically, the data feature label may be calculated according to the following formula (1), the probability prediction label may be calculated according to the following formula (2), and finally the soft supervision label may be calculated according to the following formula (3).

loss_i=MSE(knowledge_seq,small_seq) (1)

loss_p=MSE(knowledge_predict,small_predict) (2)

loss_soft=W _i *loss_i+loss_p (3)

Among them, loss_i is the data feature label; MSE() is the mean variance function; knowledge_seq is the first data feature representation; small_seq is the second data feature representation; loss_p is the probability prediction label; knowledge_predict is the first prediction probability representation; small_predict is the second Prediction probability representation; _{loss_soft} is the soft supervision label; Wi is the weight value of the feature recognition network.

Optionally, when the feature recognition network includes multiple networks (such as the bottom network and the middle network), the first data feature representation and the second data feature representation are both composed of feature representations output by multiple network layers, which can be For the feature representation output by each network layer, a data feature label is calculated according to formula (1). For example, as shown in FIG. 6B , the first data feature representation includes: knowledge_seq _l and knowledge_seq _m , and the second data feature representation includes: samll_seq _l and samll_seq _m . At this time, the data feature label loss_i _l of the underlying network can be calculated according to knowledge_seq _l and samll_seq _l , and the data feature label loss_im of the middle-layer network can be calculated according to knowledge_seq _m and _{samll_seq m .} Correspondingly, when calculating the soft supervision label at this time, the product of the weight value of each network and its data feature label and the probability prediction label may be summed to obtain the final soft supervision label. For example, for the scene shown in FIG. 6B , the calculation formula of the soft supervision label may be loss_soft=W _l *loss_i _l +W _m * _{loss_im} +loss_p.

S6043: Determine a hard-supervised label according to the second prediction probability representation and the training task dataset information.

The training task dataset information includes: the number of training samples, the number of training labels, and the actual label value in the training task dataset.

Optionally, this sub-step may calculate hard supervised labels according to the following formula (4).

Among them, loss_hart is the hard supervision label; N is the number of training samples in the training task dataset; M is the number of training labels, i is the ith training sample; c is the cth training label; y _ic is the ith sample belonging to the ith The actual label value of the c training labels; small_predict _ic is the probability value that the i-th training sample output by the prediction network layer of the distillation model belongs to the c-th training label. Optionally, the value of y _ic can be 0 or 1.

The present embodiment S6041-S6043 determines the soft-supervised label according to the output feature representation based on the reinforcement model and the distillation model based on the same training task data set, and determines the hard-supervised label according to the actual label value of the training task data and the prediction probability of the distillation model, It provides a new idea for the determination of soft supervised labels and hard supervised labels, and improves the accuracy of soft and hard supervised labels.

S605: Determine the target label according to the soft-supervised label and the hard-supervised label.

Among them, the target label is the final label value determined for the training of the supervised distillation model after combining the characteristics of the soft-supervised label and the hard-supervised label. Optionally, this step determines the target label according to the following formula (5):

loss=alpha*loss_soft+(1-alpha)*loss_hart (5)

Among them, loss is the target label, alpha is the parameter variable; loss_soft is the soft-supervised label; loss_hart is the hard-supervised label.

The optional parameter variables in the above formula (5) may be constants set based on preset rules, or may be variables that are trained together with the distillation model. This embodiment is not limited.

S606, according to the target label, iteratively update the parameters of the distillation model to obtain the target learning model.

Optionally, this embodiment may update and adjust the parameters of the distillation model according to the target label determined in S605 and according to preset rules, such as the back-propagation algorithm (BP algorithm), so as to complete one iteration of the parameters of the distillation model. renew. Then, from the training task data set, obtain the next set of preset sizes, such as batch_size training data, and input it into the distillation model, return to perform the operations of S603-S606, and perform the next iterative update on the parameters of the distillation model, thus completing Train the distillation model. After training the distillation model for many times, you can test the trained distillation model through the test task dataset. If the training end condition is met, the distillation model has been trained, and the trained distillation model can be used as the target learning model.

In the technical solution of this embodiment, a strengthened model is obtained by precipitation training the underlying network of the pre-training model, a distillation model is constructed and target knowledge is extracted; the soft supervision label is determined according to the processing result of the distillation model on the task training data and the extracted target knowledge and hard supervision labels, and then determine the target label based on the soft and hard supervision labels to iteratively update the parameters of the distillation model to obtain the target learning model. In this embodiment, the soft-supervised label and the hard-supervised label are combined to train the distillation model, so that the trained distillation model can approach the prediction effect of the pre-trained model and also improve the generalization ability of the distillation model. In order to better meet the real-time response requirements of human-computer interaction equipment.

FIG. 7 is a flowchart of a model training method based on knowledge distillation provided according to an embodiment of the present application. This embodiment provides a preferred example on the basis of the above-mentioned embodiments. Specifically, as shown in FIG. 7 , the method includes:

S701, obtaining a pre-training model.

Optionally, the pre-training model obtained in this step is a model that has been trained based on a large number of training samples, and the pre-training model can better complete the online prediction task.

S702, perform domain training on the pre-training model according to the training domain data set, and update the pre-training model.

S703, perform successive precipitation training on the underlying network of the pre-training model according to the training task data set.

Among them, the training objects of each precipitation training include at least the bottom network and the prediction layer network, and include the middle and high-level networks that gradually decrease, and the pre-training model includes the bottom layer network, at least one middle and high-level network and the prediction layer network from bottom to top.

S704, test the pre-trained model after precipitation training according to the test task data set.

S705, it is judged whether the test result satisfies the precipitation end condition, if yes, go to S706, if not, go back to go to S702.

Optionally, if the test result satisfies the precipitation end condition, it means that the precipitation training has achieved the expected effect, and S706 can be executed to use it as the reinforcement model; otherwise, it means that the precipitation training is not sufficient, and it is necessary to return to S702 based on the training domain data set, and pre-training is performed. The parameters of the model are updated and adjusted.

S706, if the test result satisfies the precipitation end condition, the pre-trained model after the precipitation training is used as the reinforcement model.

S707, taking at least two networks in the reinforcement model as target networks, and constructing a distillation model according to the target networks.

S708, extract the target knowledge of the training task data set by strengthening the target network of the model.

S709, train the distillation model according to the target knowledge and the training task data set.

S710, test the trained distillation model according to the test task data set.

S711, determine whether the test result satisfies the training end condition, if yes, execute S712, if not, return to execute S709.

S712, if the test result satisfies the training end condition, use the trained distillation model as the target learning model.

The technical solution of the embodiment of the present application provides a specific implementation solution for distilling the target learning model from the pre-training model based on the knowledge distillation technology. The target learning model distilled from the solution retains the accurate prediction ability of the pre-training model while retaining the accurate prediction ability of the pre-training model. , simplifies the network structure branches and improves the generalization ability of the model. Deploying the target learning model into human-computer interaction equipment can realize fast and accurate task execution to meet the real-time response requirements of human-computer interaction equipment.

FIG. 8 is a flowchart of an intent recognition method provided according to an embodiment of the present application. This embodiment is applicable to the case of performing intention recognition based on the target learning model trained in the above-mentioned embodiments. This embodiment may be implemented by an intention recognition apparatus configured in an electronic device, and the apparatus may be implemented by software and/or hardware. Optionally, the electronic device may be a human-computer interaction device or a server that communicates and interacts with the human-computer interaction device. The human-computer interaction device may be a smart robot, a smart speaker, a smart phone, and the like. As shown in Figure 8, the method includes:

S801, acquiring user voice data collected by a human-computer interaction device.

Optionally, the human-computer interaction device in this embodiment of the present application may collect user voice data in the environment in real time through a voice collecting device (eg, a microphone) configured inside the device. If the execution subject of this embodiment is a human-computer interaction device, the human-computer interaction device can directly perform the following operations of S802 after collecting the user's voice data. If the execution body of this embodiment is the server that communicates and interacts with the human-computer interaction device, after the human-computer interaction device collects the user's voice data, it will transmit the user's voice data to its communication and interaction server, and the server will After the user voice data is acquired, the following operations of S802 are performed.

S802 , input the user speech data into the target learning model to obtain the user intent recognition result output by the target learning model.

The target learning model in this embodiment is determined based on the training of the model training method based on knowledge distillation described in any of the above embodiments. And the target learning model of this embodiment is a model for performing intention recognition.

Optionally, after the human-computer interaction device or the server that communicates and interacts with it obtains the user's voice data, it will input the obtained user's voice data into the target learning model. The algorithm during training analyzes and predicts the user's voice data online, and outputs the user's intent recognition result. At this time, the human-computer interaction device or the server that communicates and interacts with it obtains the user's intent recognition result output by the target learning model.

S803: Determine a response result of the human-computer interaction device according to the user intention identification result.

Optionally, the human-computer interaction device or the server that communicates and interacts with it will determine the target human-computer interaction response rule corresponding to the user intent recognition result based on the obtained user intent recognition result, and based on the target human-computer interaction response rule, Determine the response result this time, and feed back the response result to the user, so as to realize human-computer interaction based on the user's voice data.

The technical solution of the embodiment of the present application is to deploy the target learning model for intention recognition trained based on the model training method based on knowledge distillation described in any of the above embodiments to a human-computer interaction device or a device that communicates and interacts with the human-computer interaction device. In the server, the human-computer interaction device or the server that communicates and interacts with it can obtain user voice data and input it into the target learning model, and determine the response result based on the user intent recognition result output by the target learning model. The target learning model deployed in the human-computer interaction device or the server that communicates and interacts with the embodiment of the present application is trained by means of knowledge distillation, and its network structure is simpler than that of the pre-training model, and the prediction effect can be approximated With complex pre-trained models, it can realize fast and accurate intent recognition to meet the real-time response requirements of human-computer interaction devices.

9 is a schematic structural diagram of a video processing apparatus provided according to an embodiment of the present application. This embodiment is applicable to a situation where a pre-training model with a complex network structure is compressed and trained into a target learning model with a simple network structure based on knowledge distillation technology. The apparatus can implement the model training method based on knowledge distillation described in any embodiment of the present application, and the apparatus 900 specifically includes the following:

The precipitation training module 901 is configured to perform precipitation training on the underlying network of the pre-training model at least twice according to the training task data set to obtain a strengthened model of the pre-training model; wherein, the training objects of each pre-training training at least include The bottom layer network and the prediction layer network include successively decreasing middle and high level networks, and the pre-training model includes the bottom layer network, at least one of the middle and high level networks and the prediction layer network from bottom to top;

A distillation model building module 902, configured to use at least two networks in the enhanced model as target networks, and build a distillation model according to the target networks, wherein the target network includes a feature identification network and the prediction layer network; The feature identification network includes at least the underlying network;

A target knowledge extraction module 903, configured to extract the target knowledge of the training task data set through the target network of the reinforcement model;

The distillation model training module 904 is configured to train the distillation model according to the target knowledge and the training task data set to obtain a target learning model.

Further, the bottom layer network and the middle and high-level networks are used for feature identification; the prediction layer network is used for task prediction according to the identified features.

Further, the precipitation training module 901 includes:

a data subset dividing unit for dividing the training task data set to determine multiple training data subsets;

The training object determination unit is used to determine the training objects corresponding to each training data subset according to the set precipitation training times; wherein, the training objects corresponding to each training data subset include the underlying network of the pre-training model, the middle High-level network and prediction layer network, and the number of layers of the middle and high-level network included is inversely proportional to the order of precipitation training;

a precipitation training unit, configured to perform a precipitation training on the training objects corresponding to the training data subset in the pre-training model according to each training data subset;

Wherein, the number of divisions of the training data subset is less than or equal to the total number of layers of the pre-training model.

Further, the middle and high-level networks included in each of the training objects are network layers that are adjacent to the underlying network and are continuous upwards; and based on the increase in the number of times of precipitation training, the number of layers of the middle and high-level networks included in the training objects decreases. zero.

Further, the precipitation training module 901 is specifically used for:

Perform precipitation training on the underlying network of the pre-training model one by one according to the training task data set;

According to the test task data set, test the pre-trained model after precipitation training;

If the test result satisfies the precipitation end condition, the pre-trained model after the precipitation training is used as the reinforcement model.

Further, the device also includes:

The domain training model is used to perform domain training on the pre-training model according to the training domain data set before performing precipitation training on the underlying network of the pre-training model at least twice according to the training task data set, and update the pre-training model.

Further, the distillation model building module 902 is specifically used for:

Using at least two networks in the enhanced model as target networks, and acquiring network building blocks of the target network;

According to the obtained network structure block, a distillation model with the same structure as the reinforcement model is constructed.

Further, the distillation model building module 902 is also specifically used for:

using at least two networks in the enhanced model as target networks;

According to the target network, a neural network model with a different structure from the reinforcement model is selected as the distillation model, wherein the output layer network of the neural network model is consistent with the type of the prediction layer network in the target network, and the neural network The non-output layer network of the model is consistent with the type of the feature recognition network in the target network.

Further, the target knowledge extraction module 903 is specifically used for:

Using the training task data set as the input of the reinforcement model, obtain the first data feature representation of the feature recognition network output of the reinforcement model, and the first prediction probability representation of the prediction layer network output of the reinforcement model;

The acquired first data feature representation and the first predicted probability representation are used as target knowledge of the training task data set.

Further, the distillation model training module 904 includes:

Supervised label determination unit, used to input the training task dataset into the distillation model, and determine soft-supervised labels and hard-supervised labels according to the processing results of the distillation model on the training task dataset and the target knowledge Label;

a target label determination unit, configured to determine a target label according to the soft supervision label and the hard supervision label;

A model parameter updating unit, configured to iteratively update the parameters of the distillation model according to the target label.

Further, the supervision label determination unit specifically includes:

The output acquisition subunit is used to input the training task data set into the distillation model to obtain the second data feature representation of the feature recognition network output of the distillation model, and the second data feature representation of the prediction layer network output of the distillation model. Predicted probability representation;

a soft label determination subunit, configured to determine a soft supervision label according to the target knowledge, the second data feature representation and the second predicted probability representation;

A hard label determination subunit, configured to determine a hard supervised label according to the second prediction probability representation and the training task dataset information.

Further, the training task data set information includes: the number of training samples, the number of training labels and the actual label value in the training task data set.

Further, the soft label determination subunit is specifically used for:

Taking the mean variance of the first data feature representation in the target knowledge and the second data feature representation as a data feature label;

Taking the mean variance of the first prediction probability representation in the target knowledge and the second prediction probability representation as a probability prediction label;

According to the weight value of the feature recognition network of the enhanced model, label fusion is performed on the data feature label and the probability prediction label to obtain a soft-supervised label.

Further, the distillation model training module 904 is specifically used for:

According to the target knowledge and the training task data set, the distillation model is trained;

Test the trained distillation model according to the test task dataset;

If the test result satisfies the training end condition, the trained distillation model is used as the target learning model.

Further, the pre-training model is a bert model.

Further, the pre-training model and the target learning model are models used for intention recognition;

Correspondingly, the device further includes:

The model deployment module is used for deploying the target learning model to the human-computer interaction device, so as to perform intention recognition on the user voice data obtained by the human-computer interaction device in real time.

In the technical solution of this embodiment, according to the training task data set, the bottom network, the prediction layer network and the successively decreasing middle and high-level networks are used as training objects, and the bottom network of the pre-training model is subjected to precipitation training at least twice to obtain a strengthened model; From the target network identified in the reinforcement model, the network builds the distillation model. Then, by strengthening the target network of the model, the target knowledge of the training task data set is extracted; based on the extracted target knowledge and the training task data set, the distillation model is trained to obtain the target learning model. In this embodiment, the underlying network of the pre-training model is subjected to multiple precipitation training in the manner of successively decreasing the middle and high-level networks, which can make the parameters of the underlying network of the pre-training model more accurate. Subsequent at least build a distillation model based on the accurate underlying network and prediction layer network after precipitation, and perform distillation training on the distillation model based on the extracted target knowledge, so that the target learning model distilled from the pre-training model simplifies the network structure at the same time. , retains the prediction accuracy of the pre-trained model, and also improves the generalization ability of the model. And the whole distillation process is not affected by human factors. Deploying the target learning model into human-computer interaction equipment can perform tasks quickly and accurately to meet the real-time response requirements of human-computer interaction equipment.

FIG. 10 is a schematic structural diagram of an intent recognition apparatus provided according to an embodiment of the present application. This embodiment is applicable to the situation of performing intent recognition based on the target learning model trained in the above-mentioned embodiments. The device can implement the intent recognition method described in any embodiment of the present application, and the device 1000 specifically includes the following:

A voice data acquisition module 1001, configured to acquire user voice data collected by a human-computer interaction device;

Intent recognition module 1002, configured to input the user voice data into a target learning model to obtain a user intention recognition result output by the target learning model; wherein, the target learning model is based on the knowledge-based knowledge described in any of the above embodiments The model training method of distillation is determined by training;

The response result determination module 1003 is configured to determine the response result of the human-computer interaction device according to the user intention identification result.

Further, the apparatus is configured in the human-computer interaction device, or a server that communicates and interacts with the human-computer interaction device.

The technical solution of the embodiment of the present application is to deploy the target learning model for intention recognition trained based on the model training method based on knowledge distillation described in any of the above embodiments to a human-computer interaction device or a device that communicates and interacts with the human-computer interaction device. In the server, the human-computer interaction device or the server that communicates and interacts with it can obtain user voice data and input it into the target learning model, and determine the response result based on the user intent recognition result output by the target learning model. The target learning model deployed in the human-computer interaction device or the server that communicates and interacts with the embodiment of the present application is trained by means of knowledge distillation, and its network structure is simpler than that of the pre-training model, and the prediction effect can be approximated With complex pre-trained models, it can realize fast and accurate intent recognition to meet the real-time response requirements of human-computer interaction devices.

According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

As shown in FIG. 11 , it is a block diagram of an electronic device for a model training method or an intent recognition method based on knowledge distillation according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.

As shown in FIG. 11, the electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or otherwise as desired. The processor may process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices may be connected, each providing some of the necessary operations (eg, as a server array, a group of blade servers, or a multiprocessor system). In FIG. 11, a processor 1101 is used as an example.

The memory 1102 is the non-transitory computer-readable storage medium provided by the present application. Wherein, the memory stores instructions executable by at least one processor, so that the at least one processor executes the knowledge distillation-based model training method or the intent recognition method provided by the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing the computer to execute the knowledge distillation-based model training method or the intent recognition method provided by the present application.

As a non-transitory computer-readable storage medium, the memory 1102 can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as those corresponding to the knowledge distillation-based model training method or the intent recognition method in the embodiments of the present application. Program instructions/modules (for example, the precipitation training module 901, the distillation model building module 902, the target knowledge extraction module 903, and the distillation model training module 904 shown in FIG. 9; or the speech data acquisition module 1001 shown in FIG. 10, intent Identification module 1002 and response result determination module 1003). The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1102, that is, to implement the knowledge distillation-based model training method or intent recognition in the above method embodiments. method.

The memory 1102 may include a stored program area and a stored data area, wherein the stored program area can store an operating system and an application program required by at least one function; data created by the use of the device, etc. Additionally, memory 1102 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 1102 may optionally include memory located remotely relative to the processor 1101, and these remote memories may be connected to the electronic device of the knowledge distillation-based model training method or the intent recognition method through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The electronic device for the model training method or the intent recognition method based on knowledge distillation may further include: an input device 1103 and an output device 1104 . The processor 1101 , the memory 1102 , the input device 1103 and the output device 1104 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 11 .

The input device 1103 can receive input numerical or character information, and generate key signal input related to user settings and function control of the electronic device based on the knowledge distillation-based model training method or intent recognition method, such as touch screen, keypad, mouse, track Input devices such as pads, touchpads, pointing sticks, one or more mouse buttons, trackballs, joysticks, etc. Output devices 1104 may include display devices, auxiliary lighting devices (eg, LEDs), haptic feedback devices (eg, vibration motors), and the like. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

These computational programs (also referred to as programs, software, software applications, or codes) include machine instructions for programmable processors, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or apparatus for providing machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memories, programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.

In the technical solution of this embodiment, according to the training task data set, the bottom network, the prediction layer network and the successively decreasing middle and high-level networks are used as training objects, and the bottom network of the pre-training model is subjected to precipitation training at least twice to obtain a strengthened model; From the target network identified in the reinforcement model, the network builds the distillation model. Then, by strengthening the target network of the model, the target knowledge of the training task data set is extracted; based on the extracted target knowledge and the training task data set, the distillation model is trained to obtain the target learning model. In this embodiment, the underlying network of the pre-training model is subjected to multiple precipitation training in the manner of successively decreasing the middle and high-level networks, which can make the parameters of the underlying network of the pre-training model more accurate. Subsequent at least build a distillation model based on the accurate underlying network and prediction layer network after precipitation, and perform distillation training on the distillation model based on the extracted target knowledge, so that the target learning model distilled from the pre-training model simplifies the network structure at the same time. , retains the prediction accuracy of the pre-trained model, and also improves the generalization ability of the model. And the whole distillation process is not affected by human factors. Deploying the target learning model into human-computer interaction equipment can perform tasks quickly and accurately to meet the real-time response requirements of human-computer interaction equipment.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present application can be performed in parallel, sequentially or in different orders, and as long as the desired results of the technical solutions disclosed in the present application can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (38)

1. A model training method based on knowledge distillation, the method comprising: According to the training task data set, the underlying network of the pre-training model is subjected to precipitation training at least twice to obtain an enhanced model of the pre-training model; wherein, the training objects of each pre-training training include at least the underlying network and the prediction layer a network, and includes successively decreasing middle and high-level networks, and the pre-training model includes the bottom-layer network, at least one of the middle and high-level networks, and the prediction layer network from bottom to top; At least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks, wherein the target network includes a feature identification network and the prediction layer network; the feature identification network includes at least the the underlying network; Extract the target knowledge of the training task data set through the target network of the reinforcement model; According to the target knowledge and the training task data set, the distillation model is trained to obtain a target learning model. 2 . The method according to claim 1 , wherein the bottom layer network and the middle and high-level networks are used for feature identification; and the prediction layer network is used for task prediction according to the identified features. 3 . 3. The method according to claim 1, wherein, according to the training task data set, the bottom network of the pre-training model is subjected to precipitation training at least twice, comprising: dividing the training task data set to determine multiple training data subsets; According to the set precipitation training times, the training objects corresponding to each training data subset are determined; wherein, the training objects corresponding to each training data subset include the underlying network, the middle and high-level network and the prediction layer network of the pre-training model, And the number of layers of the middle and high-level networks included is inversely proportional to the order of precipitation training; According to each training data subset, perform a precipitation training on the training objects corresponding to the training data subset in the pre-training model; Wherein, the number of divisions of the training data subset is less than or equal to the total number of layers of the pre-training model. 4. The method according to claim 3, wherein the middle and high-level networks included in each of the training objects are network layers that are adjacent to the underlying network and are continuous upwards; and based on the increase in the number of times of precipitation training, the training objects The number of layers of the mid- and high-level networks included in decrements to zero. 5. The method according to claim 1, wherein, according to the training task data set, the bottom network of the pre-training model is subjected to precipitation training at least twice to obtain the reinforcement model of the pre-training model, comprising: Perform precipitation training on the underlying network of the pre-training model one by one according to the training task data set; According to the test task data set, test the pre-trained model after precipitation training; If the test result satisfies the precipitation end condition, the pre-trained model after the precipitation training is used as the reinforcement model. 6. The method according to claim 1, wherein, according to the training task data set, before the bottom network of the pre-training model is subjected to precipitation training at least twice, the method further comprises: Domain training is performed on the pre-trained model according to the training domain data set, and the pre-trained model is updated. 7. The method according to claim 1, wherein at least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks, comprising: Using at least two networks in the enhanced model as target networks, and acquiring network building blocks of the target network; According to the obtained network structure block, a distillation model with the same structure as the reinforcement model is constructed. 8. The method according to claim 1, wherein at least two networks in the reinforcement model are used as target networks, and a distillation model is constructed according to the target networks, comprising: using at least two networks in the enhanced model as target networks; According to the target network, a neural network model with a different structure from the reinforcement model is selected as the distillation model, wherein the output layer network of the neural network model is consistent with the type of the prediction layer network in the target network, and the neural network The non-output layer network of the model is consistent with the type of the feature recognition network in the target network. 9. The method according to claim 1, wherein, extracting the target knowledge of the training task data set through the target network of the reinforcement model, comprising: Using the training task data set as the input of the reinforcement model, obtain the first data feature representation of the feature recognition network output of the reinforcement model, and the first prediction probability representation of the prediction layer network output of the reinforcement model; The acquired first data feature representation and the first predicted probability representation are used as target knowledge of the training task data set. 10. The method of claim 1, wherein the distillation model is trained according to the target knowledge and the training task dataset, comprising: Inputting the training task data set into the distillation model, and determining a soft-supervised label and a hard-supervised label according to the processing result of the distillation model on the training task data set and the target knowledge; determining a target label according to the soft supervision label and the hard supervision label; According to the target label, the parameters of the distillation model are iteratively updated. 11. The method according to claim 10, wherein the training task data set is input into the distillation model, and according to the processing result of the distillation model on the training task data set and the target knowledge, determine Soft-supervised labels and hard-supervised labels, including: Inputting the training task data set into the distillation model, to obtain the second data feature representation of the feature recognition network output of the distillation model, and the second prediction probability representation of the prediction layer network output of the distillation model; determining a soft-supervised label according to the target knowledge, the second data feature representation and the second predicted probability representation; Hard-supervised labels are determined based on the second predicted probability representation and the training task dataset information. 12. The method according to claim 11, wherein the training task data set information comprises: the number of training samples, the number of training labels and the actual label value in the training task data set. 13. The method of claim 11, wherein determining a soft-supervised label based on the target knowledge, the second data feature representation, and the second predicted probability representation comprises: Taking the mean variance of the first data feature representation in the target knowledge and the second data feature representation as a data feature label; Taking the mean variance of the first prediction probability representation in the target knowledge and the second prediction probability representation as a probability prediction label; According to the weight value of the feature recognition network of the enhanced model, label fusion is performed on the data feature label and the probability prediction label to obtain a soft-supervised label. 14. The method according to claim 1, wherein, according to the target knowledge and the training task data set, the distillation model is trained to obtain a target learning model, comprising: According to the target knowledge and the training task data set, the distillation model is trained; Test the trained distillation model according to the test task dataset; If the test result satisfies the training end condition, the trained distillation model is used as the target learning model. 15. The method of any one of claims 1-14, wherein the pretrained model is a bert model. 16. The method of any one of claims 1-14, wherein the pretrained model and the target learning model are models for intent recognition; Correspondingly, after the distillation model is trained according to the target knowledge and the training task data set to obtain a target learning model, the method further includes: The target learning model is deployed in a human-computer interaction device to perform intention recognition on the user's voice data acquired by the human-computer interaction device in real time. 17. An intent recognition method, the method comprising: Obtain user voice data collected by human-computer interaction equipment; Input the user speech data into the target learning model to obtain the user intent recognition result output by the target learning model; wherein, the target learning model is based on the knowledge distillation-based model training method described in any one of claims 1-16 determined by training; A response result of the human-computer interaction device is determined according to the user intention recognition result. 18. The method according to claim 17, wherein the execution body of the method is the human-computer interaction device or a server that communicates and interacts with the human-computer interaction device. 19. A model training device based on knowledge distillation, the device comprising: The precipitation training module is used to perform precipitation training on the underlying network of the pre-training model at least twice according to the training task data set to obtain the enhanced model of the pre-training model; wherein, the training objects of each pre-training training at least include all the bottom layer network and the prediction layer network, and include successively decreasing middle and high level networks, and the pre-training model includes the bottom layer network, at least one of the middle and high level networks and the prediction layer network from bottom to top; a distillation model building module, configured to use at least two networks in the enhanced model as target networks, and build a distillation model according to the target networks, wherein the target network includes a feature recognition network and the prediction layer network; The feature identification network includes at least the underlying network; a target knowledge extraction module, used for extracting the target knowledge of the training task data set through the target network of the reinforcement model; The distillation model training module is used for training the distillation model according to the target knowledge and the training task data set to obtain a target learning model. 20. The apparatus according to claim 19, wherein the bottom layer network and the middle and high-level networks are used for feature identification; and the prediction layer network is used for task prediction according to the identified features. 21. The apparatus of claim 19, wherein the precipitation training module comprises: a data subset dividing unit for dividing the training task data set to determine multiple training data subsets; The training object determination unit is used to determine the training objects corresponding to each training data subset according to the set precipitation training times; wherein, the training objects corresponding to each training data subset include the underlying network of the pre-training model, the middle High-level network and prediction layer network, and the number of layers of the middle and high-level network included is inversely proportional to the order of precipitation training; a precipitation training unit, configured to perform a precipitation training on the training objects corresponding to the training data subset in the pre-training model according to each training data subset; Wherein, the number of divisions of the training data subset is less than or equal to the total number of layers of the pre-training model. 22. The device according to claim 21, wherein the middle and high-level networks included in each of the training objects are network layers adjacent to the underlying network and continuous upward; and based on the increase in the number of times of the precipitation training, the training objects The number of layers of the mid- and high-level networks included in decrements to zero. 23. The device according to claim 19, wherein the precipitation training module is specifically used for: Perform precipitation training on the underlying network of the pre-training model one by one according to the training task data set; According to the test task data set, test the pre-trained model after precipitation training; If the test result satisfies the precipitation end condition, the pre-trained model after the precipitation training is used as the reinforcement model. 24. The apparatus of claim 19, further comprising: The domain training model is used to perform domain training on the pre-training model according to the training domain data set before performing precipitation training on the underlying network of the pre-training model at least twice according to the training task data set, and update the pre-training model. 25. The apparatus of claim 19, wherein the distillation model building block is specifically used to: Using at least two networks in the enhanced model as target networks, and acquiring network building blocks of the target network; According to the obtained network structure block, a distillation model with the same structure as the reinforcement model is constructed. 26. The apparatus of claim 19, wherein the distillation model building block is further specifically used for: using at least two networks in the enhanced model as target networks; According to the target network, a neural network model with a different structure from the reinforcement model is selected as the distillation model, wherein the output layer network of the neural network model is consistent with the type of the prediction layer network in the target network, and the neural network The non-output layer network of the model is consistent with the type of the feature recognition network in the target network. 27. The apparatus according to claim 19, wherein the target knowledge extraction module is specifically used for: Using the training task data set as the input of the reinforcement model, obtain the first data feature representation of the feature recognition network output of the reinforcement model, and the first prediction probability representation of the prediction layer network output of the reinforcement model; The acquired first data feature representation and the first predicted probability representation are used as target knowledge of the training task data set. 28. The apparatus of claim 19, wherein the distillation model training module comprises: Supervised label determination unit, used to input the training task dataset into the distillation model, and determine soft-supervised labels and hard-supervised labels according to the processing results of the distillation model on the training task dataset and the target knowledge Label; a target label determination unit, configured to determine a target label according to the soft supervision label and the hard supervision label; A model parameter updating unit, configured to iteratively update the parameters of the distillation model according to the target label. 29. The apparatus according to claim 28, wherein the supervisory label determination unit specifically comprises: The output acquisition subunit is used to input the training task data set into the distillation model to obtain the second data feature representation of the feature recognition network output of the distillation model, and the second data feature representation of the prediction layer network output of the distillation model. Predicted probability representation; a soft label determination subunit, configured to determine a soft supervision label according to the target knowledge, the second data feature representation and the second predicted probability representation; A hard label determination subunit, configured to determine a hard supervised label according to the second prediction probability representation and the training task dataset information. 30. The apparatus according to claim 29, wherein the training task data set information comprises: the number of training samples, the number of training labels and the actual label value in the training task data set. 31. The apparatus according to claim 29, wherein the soft label determination subunit is specifically used for: Taking the mean variance of the first data feature representation in the target knowledge and the second data feature representation as a data feature label; Taking the mean variance of the first prediction probability representation in the target knowledge and the second prediction probability representation as a probability prediction label; According to the weight value of the feature recognition network of the enhanced model, label fusion is performed on the data feature label and the probability prediction label to obtain a soft-supervised label. 32. The apparatus of claim 19, wherein the distillation model training module is further used to: According to the target knowledge and the training task data set, the distillation model is trained; Test the trained distillation model according to the test task dataset; If the test result satisfies the training end condition, the trained distillation model is used as the target learning model. 33. The apparatus of any one of claims 19-32, wherein the pretrained model is a bert model. 34. The apparatus of any one of claims 19-32, wherein the pretrained model and the target learning model are models for intent recognition; Accordingly, it also includes: The model deployment module is used for deploying the target learning model to the human-computer interaction device, so as to perform intention recognition on the user voice data obtained by the human-computer interaction device in real time. 35. An intent recognition device, the device comprising: A voice data acquisition module, used to acquire user voice data collected by human-computer interaction equipment; Intention recognition module, used for inputting the user voice data into the target learning model, to obtain the user intention recognition result output by the target learning model; wherein, the target learning model is based on any one of claims 1-16 based on the The model training method of knowledge distillation is determined by training; A response result determination module, configured to determine a response result of the human-computer interaction device according to the user intention identification result. 36. The apparatus according to claim 35, wherein the apparatus is configured in the human-computer interaction device, or a server that communicates and interacts with the human-computer interaction device. 37. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-16 The model training method based on knowledge distillation, or perform the intent recognition method described in any one of claims 17-18. 38. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the knowledge distillation-based model training method of any one of claims 1-16, or to perform The intent recognition method of any one of claims 17-18.

Images