CN111666763A - Network structure construction method and device for multitask scene - Google Patents

Abstract

The present application relates to artificial intelligence, and provides a network structure construction method, device, device and storage medium for multi-task scenarios, including: acquiring a training set, and inputting training sub-text data corresponding to each target semantic task step by step to be determined The multi-task network model of the network structure, obtain the sub-prediction results corresponding to each target semantic task, adjust the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained; obtain the search space corresponding to the multi-task network model , obtain the validation set, adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space, and divide the hidden state vector of the multi-task network model into multiple ordered sub-hidden states during the search. vector, until the output result of the multi-task network model on the validation set satisfies the convergence condition, the target structure parameters are obtained, and the trained multi-task network model is obtained.

Description

Network structure construction method and device for multi-task scene

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a network structure construction method, apparatus, computer equipment and storage medium for multi-task scenarios.

Background technique

Machine learning (ML, Machine Learning) is a branch of artificial intelligence. The purpose of machine learning is to allow machines to learn based on prior knowledge, so as to have the logical ability to classify and judge. Machine learning models represented by neural networks continue to develop and are increasingly applied to various industries.

The multi-task learning mechanism is widely used in the application of modern artificial intelligence products. Multitasking refers to the need to obtain the corresponding recognition results for different tasks. The original solution is to train a model for each subtask. After deployment, each model needs to be trained once. The training is time-consuming and the prediction speed is slow. Engineers manually tried different neural network architectures themselves and then settled on the target architecture based on the performance on the validation set. Due to the complexity of network architecture learning in multi-task scenarios, it is difficult to manually design a very good neural network structure. Traditional model structure automatic search methods are mainly aimed at classification problems, and cannot be directly applied to model structure automatic search in multi-task scenarios. The model structure of the multi-task scene is constructed by the method of manual continuous trial, which has high complexity, low efficiency and large system resource occupancy rate.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a network structure construction method, device, computer equipment and storage medium for multi-task scenarios to automatically discover the most suitable network architecture for existing multi-task scenario datasets. Links effectively reduce resource consumption during differentiable search, make search convergence faster and more stable, improve efficiency and reduce system resource occupancy.

A method for constructing a network structure for a multi-task scenario, the method comprising:

Obtaining a training set, the training set includes training subsamples corresponding to multiple different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

The training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and the sub-prediction results corresponding to each target semantic task are obtained. Describe the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained;

Obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the structure of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set parameter, the hidden state vector of the multi-task network model is divided into a plurality of ordered sub-hidden state vectors during the search, the sub-hidden state vectors corresponding to the current search are obtained in a preset order, and the sub-hidden state vectors are divided into sub-hidden state vectors. The vector input corresponds to the network layer for training, and the updated multi-task network model is obtained, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step is returned, until the multi-task network model is in The output result on the verification set satisfies the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matched network parameters.

In one of the embodiments, the searchable network search space is shared in at least one of the following ways:

Matrix parameter sharing of multi-head attention in the differentiable network search space;

During the search of the pooling layer of the multi-task network model, the parameters of the mapping network are shared based on multiple operators of the capsule network;

The connection relationship between the nodes of the multi-task network model is acquired, the nodes with the same starting node are formed into a node set, and the operators corresponding to the nodes in different node sets share parameters.

In one embodiment, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and the sub-prediction results corresponding to each target semantic task are obtained including:

Perform word segmentation on the current training sub-text data corresponding to the current target semantic task, map each word segmentation to the corresponding vector, and form a vector set;

The encoder extracts semantic features from the vector set, and obtains a sub-prediction result corresponding to the current target semantic task according to the semantic features, wherein the current target semantic task is one of the target semantic tasks.

In one embodiment, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and the sub-prediction results corresponding to each target semantic task are obtained including:

Calculate the similarity between the current training sub-text data corresponding to the current target semantic task and the candidate text in the database, and obtain similar sub-text data matching the current training sub-text data;

Inputting the first vector set corresponding to the current training sub-text data into the first encoder to extract semantic features to obtain first semantic features, and inputting the second vector set corresponding to the similar sub-text data into the second encoder to extract semantic features to obtain the second semantic feature;

A sub-prediction result corresponding to the current target semantic task is obtained according to the first semantic feature and the second semantic feature.

In one of the embodiments, the weights of the first encoder and the second encoder are shared.

In one embodiment, the adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction result and the corresponding training sub-label data includes:

Obtain the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtain the sub-difference corresponding to each target semantic task;

Obtain the task weight corresponding to each target semantic task, and weight each sub-difference according to the task weight to obtain the statistical sub-difference;

The network parameters of the multi-task network model are adjusted according to the statistical sub-differences.

A network structure construction device for a multi-task scenario, the device comprising:

an acquisition module, used for acquiring a training set, the training set includes training subsamples corresponding to a plurality of different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

The network parameter adjustment module is used to input the training sub-text data corresponding to each target semantic task step by step into the multi-task network model of the network structure to be determined, and obtain the sub-prediction results corresponding to each target semantic task. According to the sub-prediction results and the corresponding training The difference of the sub-label data adjusts the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained;

The network structure building module is used to obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the corresponding current target network parameters by searching the differentiable network search space according to the verification set. The structural parameters of the multi-task network model, the hidden state vector of the multi-task network model is divided into a plurality of ordered sub-hidden state vectors during the search, and the sub-hidden states corresponding to the current search are obtained in a preset order. vector, input the sub-hidden state vector into the corresponding network layer for training, obtain the updated multi-task network model, and return to the network parameter adjustment module until the output result of the multi-task network model on the verification set satisfies the convergence condition, and the target is obtained. Structural parameters, obtaining network parameters matching the target structural parameters, and obtaining a trained multi-task network model according to the target structural parameters and the matching network parameters.

In one of the embodiments, the network structure building block is further used to search the differentiable network search space through at least one of the following sharing methods:

Matrix parameter sharing of multi-head attention in the differentiable network search space;

During the search of the pooling layer of the multi-task network model, the parameters of the mapping network are shared based on multiple operators of the capsule network;

The connection relationship between the nodes of the multi-task network model is acquired, the nodes with the same starting node are formed into a node set, and the operators corresponding to the nodes in different node sets share parameters.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Obtaining a training set, the training set includes training subsamples corresponding to multiple different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

The training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and the sub-prediction results corresponding to each target semantic task are obtained. Describe the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained;

Obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the structure of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set parameter, the hidden state vector of the multi-task network model is divided into a plurality of ordered sub-hidden state vectors during the search, the sub-hidden state vectors corresponding to the current search are obtained in a preset order, and the sub-hidden state vectors are divided into sub-hidden state vectors. The vector input corresponds to the network layer for training, and the updated multi-task network model is obtained, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step is returned, until the multi-task network model is in The output result on the verification set satisfies the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matched network parameters.

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Obtaining a training set, the training set includes training subsamples corresponding to multiple different target semantic tasks, and the training subsamples include training subtext data and training sublabel data;

The training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and the sub-prediction results corresponding to each target semantic task are obtained. Describe the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained;

Obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the structure of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set parameter, the hidden state vector of the multi-task network model is divided into a plurality of ordered sub-hidden state vectors during the search, the sub-hidden state vectors corresponding to the current search are obtained in a preset order, and the sub-hidden state vectors are divided into sub-hidden state vectors. The vector input corresponds to the network layer for training, and the updated multi-task network model is obtained, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step is returned, until the multi-task network model is in The output result on the verification set satisfies the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matched network parameters.

The above-mentioned network structure construction method, device, computer equipment and storage medium for multi-task scenarios have the ability to automatically discover the network architecture most suitable for existing multi-task scenario data sets, and can improve multi-task systems without manually trying many different models. It can effectively reduce the resource consumption during differentiable search through partial links, and make the search convergence faster and more stable. While improving the system accuracy, it reduces the cost of manpower and computing resources required for system development, improves efficiency and reduces system resources. Occupancy rate.

Description of drawings

1 is an application environment diagram of a method for constructing a network structure for a multi-task scenario in one embodiment;

2 is a schematic flowchart of a method for constructing a network structure for a multi-task scenario in one embodiment;

3 is a structural block diagram of an apparatus for constructing a network structure for a multi-task scenario in one embodiment;

FIG. 4 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The method for constructing a network structure for a multi-task scenario provided by this application can be applied to the application environment shown in FIG. 1 . FIG. 1 is an application environment diagram for the operation of a method for constructing a network structure in a multi-task scenario in one embodiment. As shown in FIG. 1 , the application environment includes a terminal 110 and a server 120 . The terminal and the server communicate through a network, and the communication network may be a wireless or wired communication network, such as an IP network, a cellular mobile communication network, etc., wherein the number of terminals and servers is not limited.

Wherein, the terminal 110 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented as an independent server or a server cluster composed of multiple servers. A training set can be obtained at the terminal 110 or the server 120, and the training set includes training subsamples corresponding to a plurality of different target semantic tasks, and the training subsamples include training subtext data and training sublabel data; The data is input into the multi-task network model of the network structure to be determined step by step, the sub-prediction results corresponding to each target semantic task are obtained, and the network parameters of the multi-task network model are adjusted according to the difference between the sub-prediction results and the corresponding training sub-label data, until Obtain the current target network parameters corresponding to the current network structure; obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the current target network parameters according to the verification set by searching the differentiable network search space. The structural parameters of the multi-task network model. When searching, the hidden state vector of the multi-task network model is divided into multiple ordered sub-hidden state vectors, and the sub-hidden state vectors corresponding to the current search are obtained in a preset order. The sub-hidden state vector is input to the corresponding network layer for training, and an updated multi-task network model is obtained, and the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step is returned. The output result of the task network model on the verification set satisfies the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matched network parameters.

In one embodiment, as shown in FIG. 2 , a method for constructing a network structure for a multi-task scenario is provided, and the method is applied to the terminal 110 or the server 120 in FIG. 1 as an example for description, including the following steps:

Step 210: Obtain a training set, where the training set includes training subsamples corresponding to multiple different target semantic tasks, and the training subsamples include training subtext data and training sublabel data.

Among them, training sub-samples corresponding to multiple different target semantic tasks constitute a training set, and target semantic tasks are multiple different types of tasks corresponding to multi-task scenarios. For example, tasks of semantic analysis type include entity recognition, sentence classification, and intent recognition. Sentence pair similarity and other tasks. The number of target semantic tasks corresponds to the target recognition result of the multi-task network model of the network structure to be determined, wherein the multi-task network model may be a semantic analysis network. If the target recognition result of the semantic analysis network includes entity recognition and purpose recognition of the input text, the target semantic task includes entity recognition task and purpose recognition task. For example, when receiving a question from the user "how to eat metformin", it is necessary to identify the entity "metformin" and the intent of the sentence, that is, the user wants to ask the usage and dosage.

Specifically, different target semantic tasks have corresponding training subsamples to adapt to multi-task scenarios. For example, the first target semantic task corresponds to the first training subsample, the second target semantic task corresponds to the second training subsample, and each training subsample corresponds to the second training subsample. Both include training sub-text data and training label data, where the training sub-label data is the task recognition result corresponding to the training text data for which the corresponding task result has been determined, and this task recognition result is used as the training sub-label data corresponding to the target semantic task.

Step 220, input the training sub-text data corresponding to each target semantic task step by step into the multi-task network model of the network structure to be determined, obtain the sub-prediction results corresponding to each target semantic task, and obtain the sub-prediction results corresponding to each target semantic task. The network parameters of the multi-task network model are adjusted differentially until the current target network parameters corresponding to the current network structure are obtained.

Specifically, the training sub-text data in the training sample is input into the multi-task network model of the network structure to be determined step by step with the target semantic task as the unit, wherein the step means that the training sub-text data corresponding to the first target semantic task is input first, Obtain the first sub-prediction result corresponding to the first target semantic task, and then input the training sub-text data corresponding to the second target semantic task to obtain the second sub-prediction result corresponding to the second target semantic task, until the corresponding The training sub-text data is sequentially input step by step to obtain the corresponding sub-prediction results. Each sub-prediction result has corresponding training sub-label data, so as to calculate the sub-difference corresponding to each target semantic task, construct a loss function according to each sub-difference, and then backpropagate in the direction that minimizes the loss function to adjust the multi-task network. network parameters of the model and continue training until the end of training condition is met. By minimizing the training loss, the current target network parameters related to the structure of the multi-task network model, i.e. the current optimal weight w are obtained.

Step 230: Obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set, and search for When the hidden state vector of the multi-task network model is divided into multiple ordered sub-hidden state vectors, the sub-hidden state vectors corresponding to the current search are obtained in a preset order, and the sub-hidden state vectors are input into the corresponding network. layer training to obtain an updated multi-task network model, and return to the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined, until the output of the multi-task network model on the validation set The result satisfies the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matched network parameters.

Among them, the search space is defined, and the search space contains various operators, including LSTM (Long Short-TermMemory, long short-term memory network), gated recurrent unit GRU, one-dimensional convolution, multi-head attention (multi-head attention), etc., where the size of the kernel of the operator can be 1, 3, 5, etc., where the number of attention heads can be 1, 2, 4, 8, etc.

Specifically, the multi-task network model is regarded as a stack of multiple cells, and a cell is a directed graph composed of N ordered nodes connected by directed edges to continuously relax the search space, while Each directed edge (i, j) represents an operator, which is regarded as a mixture of all sub-operations, which can be realized by superposition of softmax weights. The following is the softmax formula:

Among them, o(x) is the weight of each sub-operation that is randomly initialized. Because of the needs of training, it cannot be limited whether it is a number between 0 and 1. The formula is to convert it to a number between 0-1, so that the weights of all sub-operations add up to 1. The sub-operations of the directed edge (i,j) are mixed with weight α ^(i,j) . The dimension is |O|, that is, the total number of sub-operations between directed edges (i, j); o() represents the current sub-operation. Update the structural parameters and network parameters, and learn the optimal weight parameters. The optimization goal is a two-layer Bi-level optimization problem, that is,

The optimization method is cross gradient descent, updating the w network parameters once along the gradient of L _train (w _k-1 , α _k-1 ) to w _k-1 , and following L _train (w _k , α _k-1 ) to α The gradient of _k-1 updates the structural parameter α of the multi-task network model once.

When searching, the hidden states of the multi-task network model are partially linked. For example, the hidden state vector includes 300 dimensions, and the 300 dimensions are divided into 6 ordered sub-hidden state vectors. Each sub-hidden state vector includes 50 dimensions. Each time the parameters are adjusted, a sub-implicit state vector is selected, that is, 50-dimension is selected, and a differentiable search is performed in one step. The network layer is trained to obtain an updated multi-task network model, and the steps of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the multi-task network model is on the verification set. The output result satisfies the convergence condition, the target structure parameters are obtained, the network parameters matching the target structure parameters are obtained, and the trained multi-task network model is obtained according to the target structure parameters and the matched network parameters. After the optimization, the operator corresponding to the target structure parameter with the largest weight is activated, and the other operators are removed to obtain the trained multi-task network model.

The above-mentioned network structure construction method for multi-task scenarios has the ability to automatically discover the network architecture that is most suitable for the existing multi-task scenario datasets. It does not need to manually try many different models to improve the accuracy of the multi-task system. Differentiable resource consumption during search, and make search convergence faster and more stable, while improving system accuracy, it reduces the cost of manpower and computing resources required for system development, improves efficiency and reduces system resource occupancy.

In one embodiment, searching the differentiable network search space is performed by at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; when searching the pooling layer of the multi-task network model, capsule-based Multiple operators of the network share the parameters of the mapping network; obtain the connection relationship between the nodes of the multi-task network model, group the nodes with the same starting node into a node set, and parameterize the operators corresponding to the nodes in different node sets shared.

Specifically, the three matrices (W_Q, W_K, W_V) of the multi-head attention can be shared by parameters, and the four operators based on the capsule network can share the parameters of a mapping network. For example, the operator of 1->2 can be shared with 3->4; the rule that can be shared is that the nodes do not share the same starting point, that is, they can be shared, and the nodes with the same starting node are formed into a node set, and different node sets The operators corresponding to the nodes in the parameter share parameters. Parameter sharing effectively reduces resource consumption during differentiable search, and makes search convergence faster and more stable.

In one embodiment, in step 220, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and obtaining the sub-prediction results corresponding to each target semantic task includes: The corresponding current training sub-text data is segmented, and each segment is mapped to the corresponding vector to form a vector set; the semantic feature is extracted from the vector set through the encoder, and the sub-prediction result corresponding to the current target semantic task is obtained according to the semantic feature. The semantic task is one of the respective target semantic tasks.

Specifically, to perform word segmentation on the current training sub-text data corresponding to the current target semantic task, a custom word segmentation algorithm may be used, and the word segmentation algorithms for different target semantic tasks may be different. To map each word segment to the corresponding vector, a custom mapping algorithm can be used. When the current target semantic task is a different target semantic task, the corresponding encoders can be different or the same, so that different semantic features can be extracted for different target semantic tasks, and the sub-prediction results corresponding to the current target semantic task can be obtained according to the semantic features.

In this embodiment, the current training sub-text data corresponding to the current target semantic task is first segmented and mapped to the corresponding vector to form a vector set, and then the encoder extracts semantic features from the vector set to obtain the sub-prediction corresponding to the current target semantic task As a result, the diversification of word segmentation and encoder improves the convenience of obtaining corresponding sub-prediction results for each target semantic task, and different word segmentation algorithms and encoders can be flexibly configured for different target semantic tasks.

In one embodiment, in step 220, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and obtaining the sub-prediction results corresponding to each target semantic task includes: calculating the current target semantic task The similarity corresponding to the corresponding current training sub-text data and the candidate text in the database is obtained, and the similar sub-text data matching the current training sub-text data is obtained; the first vector set corresponding to the current training sub-text data is input into the first encoder to extract The first semantic feature is obtained from the semantic feature, and the second vector set corresponding to the similar sub-text data is input into the second encoder to extract the semantic feature to obtain the second semantic feature; according to the first semantic feature and the second semantic feature, the current target semantic task corresponding sub-prediction results.

Specifically, the candidate texts in the database can be texts with relatively standard expressions. Similar sub-text data corresponding to the training sub-text data can be obtained through similarity search. Because the expressions are relatively standard, it is more effective to facilitate subsequent extraction to obtain semantic features. By combining the two The different semantic features obtained by the encoder are combined to obtain the sub-prediction result corresponding to the current target semantic task, which improves the accuracy of the sub-prediction result. The first encoder may be referred to as a premise encoder, and the second encoder may be referred to as a hypothesis encoder.

When the current target semantic task is a different target semantic task, the first encoder and the second encoder can be shared, and the sharing of the encoders improves resource utilization and improves training efficiency. Since one input text forms two input texts, the corresponding target semantic tasks can also include semantic tasks based on text pairs, such as question-and-answer sentence task, sentence similarity calculation task, and probability task of another sentence under the condition of one sentence Wait.

In this embodiment, similar sub-text data matched by the training sub-text data is obtained, and the sub-prediction result corresponding to the current target semantic task is obtained by combining different semantic features obtained by the two encoders, which improves the accuracy of the sub-prediction result. It also improves the diversity of target semantic task forms.

In one embodiment, the weights of the first encoder and the second encoder are shared.

Specifically, the weight sharing refers to the sharing of convolution kernel parameters, that is, the convolution kernel parameters of the first encoder are the same as the convolution kernel parameters of the second encoder. Through weight sharing, the number of parameters is reduced, and the multi-task system mechanism, as well as the weight sharing of the promise encoder and the hypothesis encoder, reduces the memory usage and cost when the multi-task system is deployed.

In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task; The sub-differences corresponding to the tasks are obtained; the task weights corresponding to each target semantic task are obtained, and the statistical sub-differences are obtained by weighting each sub-difference according to the task weights; the network parameters of the multi-task network model are adjusted according to the statistical sub-differences.

Specifically, the task weight corresponding to the target semantic task represents the importance of the target semantic task, and the larger the task weight, the higher the importance corresponding to the task. For example, for a text, the main task is to recognize the entity of the text, and the secondary task is to recognize the usage of the entity in the text, then the task weight corresponding to the entity recognition task is greater than the corresponding task weight of the entity usage recognition. The sub-differences are weighted by the task weights, so that the weighting coefficient corresponding to the important tasks is large, so that when the network parameters of the multi-task network model are adjusted according to the statistical sub-differences, the important tasks have a higher degree of influence when adjusting the parameters.

In this embodiment, by configuring the corresponding task weights for each target semantic task, the degree of influence of different target semantic tasks on the adjustment of the network parameters of the multi-task network model can be flexibly controlled.

It should be understood that although the various steps in the flowchart of FIG. 2 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIG. 2 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence is also not necessarily sequential, but may be performed alternately or alternately with other steps or sub-steps of other steps or at least a portion of a phase.

In one embodiment, as shown in FIG. 3, a network structure construction device for a multi-task scenario is provided, including: an acquisition module 310, a network parameter adjustment module 320, and a network structure construction module 330, wherein:

The obtaining module 310 is configured to obtain a training set, where the training set includes training sub-samples corresponding to a plurality of different target semantic tasks, and the training sub-samples include training sub-text data and training sub-label data.

The network parameter adjustment module 320 is used to input the training sub-text data corresponding to each target semantic task step by step into the multi-task network model of the network structure to be determined, and obtain the sub-prediction results corresponding to each target semantic task. The differences in the training sub-label data adjust the network parameters of the multi-task network model until the current target network parameters corresponding to the current network structure are obtained.

The network structure building module 330 is used to obtain the search space corresponding to the multi-task network model, form a differentiable network search space, obtain a verification set, and adjust the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set When searching, divide the hidden state vector of the multi-task network model into multiple ordered sub-hidden state vectors, obtain the sub-hidden state vectors corresponding to the current search in a preset order, and divide the sub-hidden state vectors into sub-hidden state vectors. The network layer corresponding to the vector input is trained, the updated multi-task network model is obtained, and the network parameter adjustment module is returned until the output result of the multi-task network model on the verification set meets the convergence condition, the target structure parameters are obtained, and the obtained parameters match the target structure parameters. According to the network parameters of the target structure parameters and the matching network parameters, the trained multi-task network model is obtained.

In one embodiment, the network structure building module 330 is also used to search the differentiable network search space through at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; pooling of multi-task network models When searching the layer, based on the multiple operators of the capsule network, the parameters of the mapping network are shared; the connection relationship between the nodes of the multi-task network model is obtained, and the nodes with the same starting node are formed into a node set. The operator corresponding to the node of , performs parameter sharing.

In one embodiment, the network parameter adjustment module 320 is further configured to perform word segmentation on the current training sub-text data corresponding to the current target semantic task, map each word segmentation to a corresponding vector, and form a vector set; extract semantics from the vector set through the encoder feature, and obtain the sub-prediction result corresponding to the current target semantic task according to the semantic feature, wherein the current target semantic task is one of each target semantic task.

In one embodiment, the network parameter adjustment module 320 is further configured to calculate the similarity between the current training sub-text data corresponding to the current target semantic task and the candidate text in the database to obtain similar sub-text data matching the current training sub-text data ; Input the first vector set corresponding to the current training sub-text data into the first encoder to extract semantic features to obtain the first semantic feature, and input the second vector set corresponding to the similar sub-text data into the second encoder to extract the semantic features to obtain the second semantic feature feature; obtain the sub-prediction result corresponding to the current target semantic task according to the first semantic feature and the second semantic feature.

In one embodiment, the weights of the first encoder and the second encoder are shared.

In one embodiment, the network parameter adjustment module 320 is further configured to obtain sub-prediction results and training sub-label data corresponding to each target semantic task, obtain sub-differences corresponding to each target semantic task; obtain the task weight corresponding to each target semantic task , each sub-difference is weighted according to the task weight to obtain the statistical sub-difference; the network parameters of the multi-task network model are adjusted according to the statistical sub-difference.

For the specific definition of the apparatus for constructing a network structure used in a multi-task scenario, reference may be made to the above definition of the method for constructing a network structure for a multi-task scenario, which will not be repeated here. Each module in the above-mentioned apparatus for constructing a network structure for a multi-task scenario may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 4 . The computer device includes a processor, memory, a network interface, and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The computer device's database is used to store the training set. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a network structure for a multi-task scenario is implemented.

Those skilled in the art can understand that the structure shown in FIG. 4 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components. In some embodiments, the computer device may be a terminal.

In one embodiment, a computer device is provided, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the processor implements the following steps: acquiring a training set, where the training set includes a plurality of different target semantic tasks The corresponding training sub-samples include training sub-text data and training sub-label data; input the training sub-text data corresponding to each target semantic task step by step into the multi-task network model of the network structure to be determined, and obtain the corresponding target semantic tasks. The sub-prediction results of the sub-prediction results, adjust the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data, until the current target network parameters corresponding to the current network structure are obtained; Obtain the search corresponding to the multi-task network model space, form a differentiable network search space, obtain a verification set, and adjust the structural parameters of the multi-task network model corresponding to the current target network parameters by searching the differentiable network search space according to the verification set. The hidden state vector is divided into multiple ordered sub-hidden state vectors, the sub-hidden state vectors corresponding to the current search are obtained in a preset order, and the sub-hidden state vectors are input into the corresponding network layers for training, and the updated sub-hidden state vectors are obtained. The multi-task network model returns the step of inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the output results of the multi-task network model on the verification set meet the convergence conditions, and the target is obtained. Structural parameters, obtain the network parameters matching the target structural parameters, and obtain the trained multi-task network model according to the target structural parameters and the matched network parameters.

In one embodiment, searching the differentiable network search space is performed by at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; when searching the pooling layer of the multi-task network model, capsule-based Multiple operators of the network share the parameters of the mapping network; obtain the connection relationship between the nodes of the multi-task network model, group the nodes with the same starting node into a node set, and parameterize the operators corresponding to the nodes in different node sets shared.

In one embodiment, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and obtaining the sub-prediction results corresponding to each target semantic task includes: The sub-text data is trained for word segmentation, and each word segmentation is mapped to the corresponding vector to form a vector set; the semantic feature is extracted from the vector set through the encoder, and the sub-prediction result corresponding to the current target semantic task is obtained according to the semantic feature. The current target semantic task is One of each target semantic task.

In one embodiment, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and obtaining the sub-prediction results corresponding to each target semantic task includes: calculating the current target semantic task corresponding to the current target semantic task. The similarity corresponding to the candidate text in the training sub-text data and the database is obtained, and the similar sub-text data matching the current training sub-text data is obtained; the first vector set corresponding to the current training sub-text data is input into the first encoder to extract semantics The first semantic feature is obtained from the feature, and the second vector set corresponding to the similar sub-text data is input into the second encoder to extract the semantic feature to obtain the second semantic feature; according to the first semantic feature and the second semantic feature, the sub-set corresponding to the current target semantic task is obtained. forecast result.

In one embodiment, the weights of the first encoder and the second encoder are shared.

In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtaining the corresponding sub-prediction results and training sub-label data for each target semantic task. The sub-differences corresponding to the target semantic tasks are obtained; the task weights corresponding to each target semantic task are obtained, and the statistical sub-differences are obtained by weighting each sub-difference according to the task weights; the network parameters of the multi-task network model are adjusted according to the statistical sub-differences.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: acquiring a training set, where the training set includes training corresponding to a plurality of different target semantic tasks Sub-sample, training sub-sample includes training sub-text data and training sub-label data; input the training sub-text data corresponding to each target semantic task step by step into the multi-task network model of the network structure to be determined, and obtain the sub-prediction corresponding to each target semantic task As a result, the network parameters of the multi-task network model are adjusted according to the difference between the sub-prediction results and the corresponding training sub-label data, until the current target network parameters corresponding to the current network structure are obtained; the search space corresponding to the multi-task network model is obtained to form A differentiable network search space is obtained, a verification set is obtained, the structural parameters of the multi-task network model corresponding to the current target network parameters are adjusted by searching the differentiable network search space according to the verification set, and the hidden state of the multi-task network model is searched. The vector is divided into multiple ordered sub-hidden state vectors, the sub-hidden state vectors corresponding to the current search are obtained in a preset order, and the sub-hidden state vectors are input into the corresponding network layers for training to obtain an updated multi-task network. model, returning the step of inputting the training subtext data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step, until the output result of the multi-task network model on the verification set satisfies the convergence condition, and the target structure parameters are obtained, Obtain the network parameters matching the target structure parameters, and obtain the trained multi-task network model according to the target structure parameters and the matched network parameters.

In one embodiment, searching the differentiable network search space is performed by at least one of the following sharing methods: matrix parameter sharing of multi-head attention in the differentiable network search space; when searching the pooling layer of the multi-task network model, capsule-based Multiple operators of the network share the parameters of the mapping network; obtain the connection relationship between the nodes of the multi-task network model, group the nodes with the same starting node into a node set, and parameterize the operators corresponding to the nodes in different node sets shared.

In one embodiment, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and obtaining the sub-prediction results corresponding to each target semantic task includes: The sub-text data is trained for word segmentation, and each word segmentation is mapped to the corresponding vector to form a vector set; the semantic feature is extracted from the vector set through the encoder, and the sub-prediction result corresponding to the current target semantic task is obtained according to the semantic feature. The current target semantic task is One of each target semantic task.

In one embodiment, the training sub-text data corresponding to each target semantic task is input into the multi-task network model of the network structure to be determined step by step, and obtaining the sub-prediction results corresponding to each target semantic task includes: calculating the current target semantic task corresponding to the current target semantic task. The similarity corresponding to the candidate text in the training sub-text data and the database is obtained, and the similar sub-text data matching the current training sub-text data is obtained; the first vector set corresponding to the current training sub-text data is input into the first encoder to extract semantics The first semantic feature is obtained from the feature, and the second vector set corresponding to the similar sub-text data is input into the second encoder to extract the semantic feature to obtain the second semantic feature; according to the first semantic feature and the second semantic feature, the sub-set corresponding to the current target semantic task is obtained. forecast result.

In one embodiment, the weights of the first encoder and the second encoder are shared.

In one embodiment, adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction results and the corresponding training sub-label data includes: obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task, and obtaining the sub-prediction results and training sub-label data corresponding to each target semantic task; The sub-differences corresponding to the tasks are obtained; the task weights corresponding to each target semantic task are obtained, and the statistical sub-differences are obtained by weighting each sub-difference according to the task weights; the network parameters of the multi-task network model are adjusted according to the statistical sub-differences.

The present application can be applied to smart government affairs and smart security, thereby promoting the construction of smart cities.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, all It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be noted that, for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (10)

1. A method of network fabric construction for a multitasking scenario, the method comprising:

acquiring a training set, wherein the training set comprises training subsamples corresponding to a plurality of different target semantic tasks, and the training subsamples comprise training subsample data and training subsample label data;

inputting training sub-text data corresponding to each target semantic task into a multi-task network model of a network structure to be determined step by step to obtain a sub-prediction result corresponding to each target semantic task, and adjusting network parameters of the multi-task network model according to the difference between the sub-prediction result and corresponding training sub-label data until current target network parameters corresponding to the current network structure are obtained;

obtaining a search space corresponding to the multitask network model, forming a micro-network search space, obtaining a verification set, adjusting the structural parameters of the multitask network model corresponding to the current target network parameters by searching the micro-network search space according to the verification set, dividing the hidden state vector of the multitask network model into a plurality of ordered sub-hidden state vectors during searching, obtaining the sub-hidden state vector corresponding to the current searching according to a preset sequence, inputting the sub-hidden state vectors into the corresponding network layer for training to obtain an updated multitask network model, returning to the step of inputting the training sub-text data corresponding to each target semantic task step by step into the multitask network model with the network structure to be determined until the output result of the multitask network model on the verification set meets the convergence condition, obtaining the target structural parameters, and obtaining the network parameters matched with the target structural parameters, and obtaining a trained multi-task network model according to the target structure parameters and the matched network parameters.

2. The method of claim 1, wherein the searching the micro-searchable space is shared by at least one of:

sharing matrix parameters of multi-head attention in the micro-network search space;

sharing parameters of a mapping network based on a plurality of operational characters of the capsule network during searching of the pooling layer of the multitask network model;

and acquiring the connection relation among the nodes of the multitask network model, forming a node set by the nodes with the same initial node, and sharing parameters of operators corresponding to the nodes in different node sets.

3. The method according to claim 1, wherein the step-by-step inputting of the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined to obtain the sub-prediction result corresponding to each target semantic task comprises:

performing word segmentation on current training sub-text data corresponding to a current target semantic task, and mapping each word segmentation to a corresponding vector to form a vector set;

semantic features are extracted from the vector set through an encoder, and a sub-prediction result corresponding to the current target semantic task is obtained according to the semantic features, wherein the current target semantic task is one of the target semantic tasks.

4. The method according to claim 1, wherein the step-by-step inputting of the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined to obtain the sub-prediction result corresponding to each target semantic task comprises:

calculating the similarity between the current training subfile data corresponding to the current target semantic task and the candidate text in the database to obtain similar subfile data matched with the current training subfile data;

inputting a first vector set corresponding to the current training sub-text data into a first encoder to extract semantic features to obtain first semantic features, and inputting a second vector set corresponding to the similar sub-text data into a second encoder to extract semantic features to obtain second semantic features;

and obtaining a sub-prediction result corresponding to the current target semantic task according to the first semantic feature and the second semantic feature.

5. The method of claim 4, wherein the weights of the first encoder and the second encoder are shared.

6. The method of claim 1, wherein adjusting the network parameters of the multitask network model according to the difference between the sub-prediction results and the corresponding training sub-label data comprises:

acquiring sub-prediction results and training sub-label data corresponding to each target semantic task to obtain sub-differences corresponding to each target semantic task;

acquiring task weights corresponding to the target semantic tasks, and weighting the sub-differences according to the task weights to obtain statistical sub-differences;

and adjusting the network parameters of the multitask network model according to the statistical sub-difference.

7. A network structure construction apparatus for a multitasking scenario, characterized in that the apparatus comprises:

the acquisition module is used for acquiring a training set, wherein the training set comprises training subsamples corresponding to a plurality of different target semantic tasks, and the training subsamples comprise training subsample data and training subsample label data;

the network parameter adjusting module is used for inputting the training sub-text data corresponding to each target semantic task into the multi-task network model of the network structure to be determined step by step to obtain a sub-prediction result corresponding to each target semantic task, and adjusting the network parameters of the multi-task network model according to the difference between the sub-prediction result and the corresponding training sub-label data until the current target network parameters corresponding to the current network structure are obtained;

a network structure constructing module for acquiring the search space corresponding to the multitask network model, forming a micro-network search space, acquiring a verification set, adjusting the structural parameters of the multitask network model corresponding to the current target network parameters by searching a micro-network searching space according to the verification set, dividing the hidden state vector of the multitask network model into a plurality of ordered sub-hidden state vectors during searching, acquiring the sub-hidden state vector corresponding to the current searching according to a preset sequence, inputting the sub-hidden state vector into a corresponding network layer for training to obtain an updated multitask network model, returning to a network parameter adjusting module until the output result of the multitask network model on the verification set meets a convergence condition to obtain the target structural parameters, and acquiring the network parameters matched with the target structural parameters, and obtaining a trained multi-task network model according to the target structure parameters and the matched network parameters.

8. The apparatus of claim 7, wherein the network structure building module is further configured to search the micro-network search space by at least one of:

sharing matrix parameters of multi-head attention in the micro-network search space;

sharing parameters of a mapping network based on a plurality of operational characters of the capsule network during searching of the pooling layer of the multitask network model;

and acquiring the connection relation among the nodes of the multitask network model, forming a node set by the nodes with the same initial node, and sharing parameters of operators corresponding to the nodes in different node sets.

9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

Images