WO2020237688A1 - Method and device for searching network structure, computer storage medium and computer program product - Google Patents

Abstract

Provided is a method for searching network structure, including: the defining search space step: (step S12) determining the search space of the neural network model to be searched for the network structure, wherein the search space defines a variety of operations on the operation layer between every two nodes in the neural network model; the pre-training step: (step S14) training the general map of the search space with the preset parameters of the first network structure according to the first network structure to obtain the general map with the pre-training parameters, wherein the general map is composed of operations; the training step: (step S16) training the general map with the pre-training parameters according to the first network structure and updating the first network structure according to the feedback amount of the first network structure. The present application also discloses a network structure search device, a computer storage medium and a computer program product.

Description

Method and device for searching network structure, computer storage medium and computer program product Technical field

This application relates to the field of machine learning, and in particular to a method and device for network structure search, computer storage media, and computer program products.

Background technique

Among related technologies, machine learning algorithms, especially deep learning algorithms, have been rapidly developed and widely used in recent years. As the application scenarios and model structures become more and more complex, it is more and more difficult to obtain the optimal model in the application scenarios. Among them, the efficient network structure search based on weight sharing can be used (Efficient Neural Architecture Search via Parameter Sharing). , ENAS) to improve the efficiency of network structure search (Neural Architecture Search, NAS). However, the network structure searched by ENAS often has a larger bias, that is, the network structure searched by ENAS always tends to operate with a larger kernel size. This leads to larger model parameters, making it difficult to debug and train. In addition, the bias of the controller means that the local optimal solution to which the controller converges cannot be fully explored (explore) the search space. Such a controller does not have high credibility and cannot guarantee that the searched model is the global optimum.

Summary of the invention

The embodiments of the present application provide a method and device for searching a network structure, a computer storage medium, and a computer program product.

The network structure search method in the implementation manner of this application includes:

The step of defining search space: determining the search space of the neural network model to be searched for the network structure, the search space defining a variety of operations on the operation layer between every two nodes in the convolutional neural network;

Pre-training step: training the general map of the search space according to the first network structure with preset parameters of the first network structure to obtain the general map with pre-training parameters, the general map being constituted by the operation;

Training step: training the general map with the pre-training parameters according to the first network structure and updating the first network structure according to the amount of feedback of the first network structure.

The network structure search device of the embodiment of the present application includes a processor and a memory, the memory stores one or more programs, and the processor is used to define a search space: determine the search space of the neural network model to be searched for the network structure, The search space defines a variety of operations on the operation layer between every two nodes in the convolutional neural network; and is used for pre-training: according to the first network structure, preset parameters of the first network structure Training the general map of the search space to obtain the general map with pre-training parameters, the general map being composed of the operations; and for training: training the general map with the pre-training parameters according to the first network structure The overall graph and the update of the first network structure according to the feedback amount of the first network structure.

The computer storage medium of the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a computer, the computer executes the above-mentioned method.

A computer program product containing instructions according to an embodiment of the present application, when the instructions are executed by a computer, the computer executes the above-mentioned method.

In the network structure search method and device, computer storage medium, and computer program product of the embodiment of this application, before optimizing the general map and the first network structure, the general map is first performed with the preset parameters of the fixed first network structure. Pre-training makes the overall picture with pre-training parameters fully trained. After the pre-training is completed, let go of the parameters of the first network structure to train the general map and the first network structure, thereby optimizing the network structure and the first network structure, and avoiding the bias caused by optimizing the first network structure when training from scratch. Improve the credibility of the first network structure and ensure that the searched model is globally optimal.

The additional aspects and advantages of the embodiments of the present application will be partly given in the following description, and part of them will become obvious from the following description, or be understood through the practice of the embodiments of the present application.

Description of the drawings

The above and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

FIG. 1 is a schematic flowchart of a method for searching a network structure according to an embodiment of the present application;

2 is a schematic diagram of modules of a network structure search device according to an embodiment of the present application;

FIG. 3 is a schematic diagram of the principle of a method for searching a network structure in related technologies;

4 is a schematic diagram of a general diagram of a network structure search method according to an embodiment of the present application;

FIG. 5 is a schematic flowchart of a network structure search method according to another embodiment of the present application;

FIG. 6 is a schematic flowchart of a network structure search method according to another embodiment of the present application;

FIG. 7 is a schematic flowchart of a network structure search method according to another embodiment of the present application;

FIG. 8 is a schematic diagram of the principle of a network structure search method according to another embodiment of the present application;

FIG. 9 is a schematic flowchart of a network structure search method according to still another embodiment of the present application;

FIG. 10 is a schematic flowchart of a network structure search method according to still another embodiment of the present application;

FIG. 11 is a schematic flowchart of a method for searching a network structure according to still another embodiment of the present application;

FIG. 12 is a schematic diagram of the penalty effect of the network structure search method in the embodiment of the present application.

Symbol description of main components:

The network structure search device 10, the memory 102, the processor 104, and the communication interface 106.

Detailed ways

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions. The following embodiments described with reference to the accompanying drawings are exemplary, and are only used to explain the present application, and cannot be understood as a limitation to the present application.

In the description of this application, it should be understood that the terms "first" and "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, "multiple" means two or more than two, unless otherwise specifically defined.

In the description of this application, it should be noted that the terms "installation", "connection", and "connection" should be interpreted broadly unless otherwise clearly specified and limited. For example, it can be a fixed connection or a detachable connection. Connected or integrally connected; it can be mechanically connected, or electrically connected or can communicate with each other; it can be directly connected, or indirectly connected through an intermediate medium, it can be the internal communication of two components or the interaction of two components relationship. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

The following disclosure provides many different embodiments or examples for realizing different structures of the present application. To simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the application. In addition, the present application may repeat reference numerals and/or reference letters in different examples. Such repetition is for the purpose of simplification and clarity, and does not indicate the relationship between the various embodiments and/or settings discussed.

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions. The following embodiments described with reference to the accompanying drawings are exemplary, and are only used to explain the present application, and cannot be understood as a limitation to the present application.

Please refer to FIG. 1 and FIG. 2, an embodiment of the present application provides a method and device 10 for searching a network structure.

The network structure search method in the implementation manner of this application includes:

Define search space Step S12: Determine the search space of the neural network model to be searched for the network structure, the search space defines a variety of operations on the operation layer between every two nodes in the neural network model;

Pre-training step S14: training a whole graph of the search space with preset parameters of the first network structure according to the first network structure to obtain a total graph of pre-training parameters, the total graph being composed of operations;

Training step S16: training the general map with pre-training parameters according to the first network structure and updating the first network structure according to the feedback amount (ACC) of the first network structure.

The network structure search apparatus 10 of the embodiment of the present application includes a processor 104 and a memory 102. The memory 102 stores one or more programs. When the programs are executed by the processor, the processor 104 is used to execute the steps of defining the search space. : Determine the search space of the neural network model to be searched for the network structure. The search space defines a variety of operations on the operation layer between every two nodes in the neural network model; and is used to perform pre-training steps: according to the first network The structure trains the total map of the search space with the preset parameters of the first network structure to obtain the total map of pre-training parameters, the total map is composed of operations; and is used to perform the training step: training the total map with pre-training parameters according to the first network structure Figure and update the first network structure according to the feedback amount of the first network structure.

In other words, the network structure search method of the embodiment of the present application can be implemented by the network structure search apparatus 10 of the embodiment of the present application. Specifically, step S12, step S14, and step S16 may be implemented by the processor 104.

In the method and device 10 for searching the network structure of the embodiment of the present application, before optimizing the network structure and the first network structure, the overall image is pre-trained with the preset parameters of the fixed first network structure, so that there is pre-training. The overall picture of the parameters is fully trained. After the pre-training is completed, let go of the parameters of the first network structure to train the general map and the first network structure, thereby optimizing the network structure and the first network structure, and avoiding the bias caused by optimizing the first network structure when training from scratch. Improve the credibility of the first network structure and ensure that the searched model is globally optimal.

It should be noted that the number of processors 104 may be one. The number of processors 104 may also be multiple, such as 2, 3, 5, or other numbers. When the number of processors 104 is multiple, the steps of the network structure search method of the present application may be executed by different processors 104, for example, different processors 104 may execute step S14 and step S16 respectively.

Optionally, the network structure search apparatus 10 may further include a communication interface 106 for outputting data processed by the network structure search apparatus 10, and/or input data to be processed by the network structure search apparatus 10 from an external device . For example, the processor 104 is used to control the communication interface 106 to input and/or output data.

In recent years, machine learning algorithms, especially deep learning algorithms, have been rapidly developed and widely used. With the continuous improvement of model performance, the model structure becomes more and more complex. In non-automated machine learning algorithms, these structures need to be manually designed and debugged by machine learning experts, and the process is very complicated. Moreover, as application scenarios and model structures become more and more complex, it becomes more and more difficult to obtain an optimal model in an application scenario. In this case, automated machine learning algorithms (AutoMachine Learning, AutoML) have received extensive attention from academia and industry, especially network architecture search (NAS).

Specifically, network structure search is a technology that uses algorithms to automatically design neural network models. The network structure search is to search out the structure of the neural network model. In the embodiment of this application, the neural network model to be searched for the network structure is Convolutional Neural Networks (CNN).

The problem to be solved by the network structure search is to determine the operations between nodes in the neural network model. Different combinations of operations between nodes correspond to different network structures. Further, the nodes in the neural network model can be understood as the characteristic layer in the neural network model. The operation between two nodes refers to the operation required to transform the feature data on one node into the feature data on the other node. The operations mentioned in this application may be other neural network operations such as convolution operations, pooling operations, or fully connected operations. It can be considered that the operation between two nodes constitutes the operation layer between these two nodes. Generally, there are multiple searchable operations on the operation layer between two nodes, that is, multiple candidate operations. The purpose of network structure search is to determine an operation on each operation layer.

For example, define conv3*3, conv5*5, depthwise3*3, depthwise5*5, maxpool3*3, average pool3*3, etc. as the search space. In other words, each layer operation of the network structure is sampled in these six options.

Refer to Figure 3. After the NAS establishes the search space, it usually uses the first network structure to sample the second network structure in the search space, then trains the second network structure to converge to determine the amount of feedback, and finally uses the amount of feedback to update the first network structure. A network structure.

Specifically, the idea of NAS is to obtain a network structure in the search space through a first network structure, and then obtain the accuracy rate R according to the network structure, and use the accuracy rate R as feedback to update the first network structure. Continue to optimize to get another network structure, and repeat until the best result is obtained. In the example in Figure 3, the first network structure is constructed by Recurrent Neural Network (RNN). Of course, the first network structure can also be constructed by Convolutional Neural Networks (CNN) or long and short-term memory artificial nerves. Network (Long-Short Term Memory, LSTM) construction. The specific method of constructing the first network structure is not limited here.

However, it takes time to train the second network structure to convergence. In this regard, related technologies have emerged a variety of methods to solve the efficiency of NAS, for example, efficient network structure search based on network transformation (Efficient Architecture Search by Network Transformation), and efficient network structure search based on weight sharing (Efficient Neural Architecture Search) via Parameter Sharing, ENAS). Among them, efficient network structure search based on weight sharing is widely used.

Specifically, please refer to Fig. 4. In the process of using the efficient network structure search based on weight sharing, operations are connected into a general map, and the final optimal structure searched is one of the sub-graphs in the general map. In the example of Fig. 4, the overall graph is formed by the operation of nodes. The connection mode of the optimal structure with edges in bold in Fig. 4 is a subgraph of the overall picture.

ENAS adopts a weight sharing strategy. After sampling a network structure every time, it is no longer directly trained to convergence, but a batch is trained. After multiple iterations, the overall graph can finally converge. Please note that the convergence of the graph is not equivalent to the convergence of the network structure.

After training the overall graph, the parameters of the overall graph can be fixed (fix), and then the first network structure is trained. Specifically, the general graph can be sampled to obtain the second network structure, and predictions can be made according to the second network structure to train the parameters of the first network structure and obtain the feedback amount of the first network structure, thereby updating the first network structure .

It can be understood that the efficient network structure search based on weight sharing can save time and improve the efficiency of network structure search because each time the network structure is searched, the parameters that can be shared are shared. For example, in the example in Figure 4, if after searching for node 1, node 3 and node 6 and training the searched network structure, node 1, node 2, node 3 and node 6 are searched this time, then, The relevant parameters of the network structure trained when node 1, node 3 and node 6 are searched can be applied to the training of the network structure searched this time. In this way, it is possible to improve efficiency through weight sharing.

ENAS can increase the efficiency of NAS by more than 1000 times. However, in the actual use process, the following problems will occur: the searched network often has a large bias, that is, the network structure searched through ENAS will always tend to For operations with larger kernel size.

Taking the above search space as an example, define conv3*3, conv5*5, depthwise3*3, depthwise5*5, maxpool3*3, average pool3*3, etc. as the search space, then the first network structure will always search conv5*5 . This will lead to two more serious problems: First, the searched model parameters are larger. In this way, even if the computing power is sufficient, there is no restriction on the model parameters, and the model with more parameters is easy to overfit, the generalization ability decreases, and it is difficult to debug and train. Second, the bias of the first network structure means that the first network structure converges to a local optimal solution, and the search space cannot be fully explored (explore). Such a first network structure does not have high credibility, and cannot guarantee that the model we searched for is the global optimal.

The above-mentioned problems are more serious, and the first network structure needs to be adjusted very finely so that it does not converge to the local optimal solution. Otherwise, the problem will invalidate the ENAS framework. But fine-tuning the first network structure will make ENAS very cumbersome, and the lost original intention of AutoML will become difficult to use. Moreover, fine adjustment cannot guarantee a better first network structure.

Based on this, the method and device 10 for searching the network structure of the embodiment of the present application, before optimizing the general map and the first network structure, pre-train the general map with the preset parameters of the fixed first network structure, so that The master map with pre-training parameters is fully trained. After the pre-training is completed, let go of the parameters of the first network structure to train the general map and the first network structure, thereby optimizing the network structure and the first network structure, and avoiding the bias caused by optimizing the first network structure when training from scratch. Improve the credibility of the first network structure and ensure that the searched model is globally optimal.

It can be understood that in traditional CNN, the neural network structure obtained from the network structure search can generally be trained and verified through sample data. Among them, the sample data includes verification samples and training samples. The verification samples can be used to verify whether the network structure is good or not, and the training samples can be used to train the network structure searched by the network structure search method.

In this embodiment, the training sample may also include a training set (train) and a test set (valid).

In other words, in this embodiment, the training sample can be divided into a training set and a test set. Specifically, the training set and the test set may be divided by the user, or divided by the device 10 for searching the network structure. Wherein, when the training set is divided by the network structure search device 10, the processor 104 may be used to divide the training sample into a training set and a test set.

In this embodiment, the first network structure is constructed by LSTM. When searching the network structure, the training set is used to train the parameters of the search structure, such as the parameters of the structure calculated by conv3*3, sep5*5. Cell is used to search the network structure. In order to see the generalization ability of the searched network structure, the LSTM parameters are trained on the test set. In other words, the training set is used to train the parameters of the search structure, and the test set is used to train the parameters of the LSTM. The verification sample is used to verify whether the network structure searched after training is good.

In an example, the number of training samples is 10, and the training samples are divided into a training set of 8 and a test set of 2, and the training set of 8 is used to train the searched network structure. A test set of 2 is used to train the LSTM.

It should be noted that, like the traditional CNN, the verification sample can be used to verify the network structure in this embodiment, that is, the verification sample in this embodiment can remain unchanged. The training set in the training sample can be used in the pre-training step S14 and the training step S16, that is, the pre-training step S14 and the training step S16 can be trained on the same training set.

Optionally, referring to FIG. 5, step S14 includes:

Step S142: Set the parameters of the first network structure as preset parameters of the first network structure.

Specifically, step S142 may be implemented by the processor 104, that is, the processor 104 may be configured to set the parameters of the first network structure as preset parameters of the first network structure.

In this way, the parameter setting of the first network structure in the pre-training process is realized. At this time, the parameters of the first network structure in the pre-training process are fixed.

Optionally, step S14 further includes:

Step S144: sampling an operation at each operation layer of the search space according to the first network structure and the preset parameters of the first network structure to obtain a sub-picture of the overall picture; and

Step S146: Use a batch of training data of the training set to train the sub-images of the overall image to obtain the overall image with pre-training parameters.

Specifically, step S144 and step S146 can be implemented by the processor 104, that is to say, the processor 104 can be used to sample one sample at each operation layer of the search space with the preset parameters of the first network structure according to the first network structure. Operate to obtain a sub-image of the overall image, and the sub-image used to train the overall image using a batch of training data of the training set to obtain the overall image with pre-training parameters.

In this way, the pre-training of the overall image is realized. In this embodiment, during pre-training, the first network structure samples operations with fixed preset parameters, so that the probability of sampling each operation is uniformly distributed. That is to say, in the pre-training process, when the first network structure samples corresponding operations with fixed preset parameters, the probability of sampling each operation is equal, and the size of the convolution kernel will not affect the first network structure. The probability of sampling to each operation.

Optionally, step S144 and step S146 are iterated until the total number of first iterations is completed.

It can be understood that steps S144 and S146 can perform pre-training on one sub-graph at a time, that is, after each pre-training, there is always a change in the parameters of one sub-graph in the overall picture with pre-training parameters. Therefore, steps S144 and S146 The iterative process allows multiple subgraphs to be pre-trained. By controlling the total number of first iterations of pre-training, each subgraph in the overall graph can be sampled, so that the overall graph can be fully pre-trained, that is, A general map with pre-trained parameters.

In an example, the search space has 5 layers, and each layer has 4 optional operations, which is equivalent to a 4X5 graph. Network structure search needs to select an operation at each layer, which is equivalent to path optimization on the graph. Initially, step S144 randomly samples an operation in each operation layer of the search space, and then connects the sampled operations to obtain a network structure. This network structure is a sub-graph in the general graph of the search space. Step S146 is This network structure is trained on a batch of test data in the training set; then, step S144 is repeated and another operation is randomly sampled at each layer to obtain another network structure. In step S146, this network structure is trained on another batch of test data in the training set.

Among them, the training set can be divided into multiple batches of training data, and step S144 and step S146 are continuously repeated until the multiple batches of training data of the training set are used up, that is, one iteration (epoch). After an epoch, some subgraphs in the overall image were randomly sampled, and the corresponding subgraphs were also pre-trained. Then, the overall image was trained in the same way to complete the second epoch... In this way, multiple iterations were performed. Then sufficient pre-training of the overall picture can be realized.

It can be understood that by setting a suitable total number of first iterations, after pre-training the overall image, each sub-image in the overall image has been trained, and an overall image with pre-training parameters can be obtained.

In this embodiment, the total number of first iterations may be 310 times. It can be understood that, in other embodiments, the value of the total number of second iterations may be 100 times, 200 times, or other values.

Optionally, referring to FIG. 6, step S16 includes:

Training the master map step S162, training the master map with pre-training parameters according to the first network structure; and

Training the first network structure step S164, determining the feedback amount of the first network structure and updating the first network structure according to the feedback amount of the first network structure.

Specifically, step S162 and step S164 can be implemented by the processor 104, that is to say, the processor 104 can be used to train the general map with pre-training parameters according to the first network structure, and to determine the amount of feedback and according to the amount of feedback. Update the first network structure.

In this way, the optimization of the network structure is realized on the basis of pre-training.

Optionally, step S162 and step S164 are iterated until the total number of second iterations is completed.

In this way, the overall graph and the first network structure are alternately optimized. Specifically, the total number of second iterations can enable the overall graph and the first network structure to be fully trained.

Optionally, referring to FIG. 7, step S162 includes:

Step S1622: Sampling an operation in each operation layer of the search space according to the first network structure to obtain a sub-graph of the overall graph with pre-training parameters; and

Step S1624: Use a batch of training data in the training set to train the sampled subgraphs.

Specifically, step S1622 and step S1624 can be implemented by the processor 104, that is, the processor 104 can be used to sample an operation at each operation layer of the search space according to the first network structure to obtain a total of pre-training parameters. A subgraph of the graph, and the subgraph used for training samples using a batch of training data from the training set.

In this way, the training of the master map is realized. In this embodiment, ENAS adopts a weight sharing strategy. After sampling a network structure each time, it is no longer directly trained to convergence, but a batch is trained, and then the first network structure is trained. Please note that the convergence of the graph is not equivalent to the convergence of the network structure.

Since the general map has been trained in the pre-training process, the parameters of the first network structure can be released at this time, that is, the preset parameters of the first network structure may not be sampled in step S1622. In the operation layer of the overall graph of training parameters, the probability of sampling to each operation may be different.

Correspondingly, step S1622 and step S1624 can be performed cyclically. In an example, at the beginning, each layer samples an operation, and then connect the sampled operations to obtain a subgraph, and train this on a batch of data in the training set Subgraph; then, sample one operation for each layer to obtain another subgraph, and then train this subgraph on another batch of data in the training set; then, continue to sample another subgraph for each layer and use another subgraph in the training set Train this subgraph on batch data...until the data in the training set is used up, that is, an epoch is completed.

After an epoch is completed in step S162 of training the general map, the parameters of the general map are fixed, and the first network structure training step S164 is entered to realize the optimization of the general map and the first network structure alternately.

Then, train the overall image in the same way to complete the second epoch, fix the parameters of the overall image, and then train the first network structure.

Then, train the overall graph in the same way to complete the third epoch, fix the parameters of the overall graph, and then train the first network structure... Iterate until the total number of second iterations is completed, to alternate the overall graph and the first network structure optimize.

It should be noted that in the step S162 of executing the training master map, the parameters of the master map change as the training progresses, that is, the master map with pre-training parameters in each iteration continues with the parameters of the master map during the training process. Update until the completion of an epoch.

In the above-discussed embodiment, each time an epoch is completed in the training master map step S162, the parameters of the master map are fixed, and then the training first network structure step S164 is entered. Of course, in other embodiments, the training master map step S162 can be After every 10 epochs, the parameters of the overall graph are fixed, and then the first network structure training step S164 is entered. The foregoing listing of the number of epochs completed in step S162 each time is only an example, and cannot be construed as a limitation of the application. It can be understood that the number of epochs completed each time in step S162 can be set according to actual conditions, and is not specifically limited here.

In this embodiment, the total number of second iterations is 310. It can be understood that, in other embodiments, the value of the total number of second iterations may be 100 times, 200 times, or other values.

It should be noted that, in this embodiment, the total number of first iterations and the total number of second iterations are the same, while in other embodiments, the total number of first iterations and the total number of second iterations may be different. There is no specific limitation here.

Please refer to Figure 8. Each operation layer of the search space corresponds to a time step of the long short-term memory artificial neural network (LSTM). For each time step, the cell of the long short-term memory artificial neural network (Cell) outputs one Hidden state, step S1622 includes:

Mapping the hidden state into a feature vector, the dimension of the feature vector is the same as the number of operations on each operation layer;

Sampling an operation in each operation layer according to the feature vector to obtain a sub-image of the overall image.

Specifically, the processor 104 may be used to map the hidden state into a feature vector, and to sample an operation at each operation layer according to the feature vector to obtain a sub-image of the overall image. Among them, the dimension of the feature vector is the same as the number of operations on each operation layer.

In this way, an operation is sampled in each operation layer of the search space to obtain a sub-graph of the overall image. For example, to search a 20-layer network altogether, 20 time steps are required regardless of jumpers.

In the example in FIG. 8, the solid arrow represents timestep, time 1 represents the first cell of LSTM, time 2 represents the second cell of LSTM, and so on. The square conv3*3 represents the operation of this layer in the model, and the circle represents the connection relationship between the operation layer and the operation layer.

It can be understood that because the calculation of the network structure has a sequence, the logical relationship of the calculation sequence is mapped to the LSTM, which is a small square in Figure 8 from left to right, corresponding to the state of the LSTM cell at each time.

Specifically, at time 1, the hidden state output by the cell is calculated to obtain conv3×3, conv3×3 is used as the input layer of the cell at time 2, and the hidden state output by the cell at time 1 is also used as the input of the cell at time 2. , Circle 1 is calculated.

In the same way, circle 1 is used as the input of the cell at time 3, and the hidden state of the cell output at time 2 is also used as the input of time 3. The convolution sep5×5 is calculated and so on.

Further, the steps of sampling an operation at each operation layer according to the feature vector to obtain the network structure include:

Normalize the feature vector (softmax) to get the probability of each operation of each operation layer;

Sample an operation at each operation layer according to the probability to get the network structure.

In this way, an operation is sampled at each operation layer according to the feature vector to obtain the network structure. Specifically, in the example shown in FIG. 8, the hidden state output by the cell of the LSTM is encoded (encoding), and it is mapped to a vector with a dimension of 6, which undergoes a normalized exponential function (softmax). , Becomes a probability distribution, sampling is performed according to this probability distribution, and the operation of the current layer is obtained. And so on to finally get a network structure. It can be understood that in this example, there is only one input, which contains a total of six operations (3×3 convolution, 5×5 convolution, 3×3 depthwise-separable convolution, 5×5 depthwise-separable convolution, max pooling, 3 ×3average pooling), the dimension of the vector corresponds to the search space, 6 means that the search space has 6 operations to choose from.

Referring to FIG. 9, in step S164, the steps of determining the feedback amount include:

Step S1642: Sample an operation at each operation layer of the search space according to the first network structure to obtain the second network structure;

Step S1644: Use a batch of test data in the test set to predict the second network structure to determine the feedback amount of the first network structure.

Correspondingly, step S1642 and step S1644 can be implemented by the processor 104, that is, the processor 104 can be used to sample an operation at each operation layer of the search space according to the first network structure to obtain the second network structure, and It is used to predict the second network structure using a batch of test data of the test set to determine the feedback amount of the first network structure.

In this way, a prediction can be made on the test set according to the searched second network structure to obtain the feedback amount of the first network structure.

In this embodiment, after the second network structure is searched, the searched second network structure can be predicted on the test set to obtain feedback to update the first network structure according to the aforementioned formula. Please note that the LSTM is not directly trained on the test set.

Optionally, step S164 of training the first network structure loops a preset number of times, and step S164 includes:

In step S1646, the first network structure is updated with the feedback amount of the first network structure determined in each cycle, and the number of feedback amounts of the first network structure is determined to be a preset number in each cycle.

Correspondingly, the processor 104 is configured to cyclically train the first network structure for a preset number of times, and is configured to update the first network structure by using the feedback amount determined in each cycle. Among them, the number of feedback amounts determined in each cycle is a preset number.

In this way, the first network structure is trained. Specifically, in this embodiment, step S16 is looped 50 times, that is to say, the training general map step S162 and the training first network structure step S164 are iteratively performed during the training process, each time the training first network structure step 164 is executed , The first network structure is updated 50 times to realize the optimization of the first network structure. After the step 164 of training the first network structure is completed, the training master map step 162 in the next iteration uses the last updated first network structure to train the master map.

It can be understood that in other examples, the preset number of times may be 10, 20, 30 or other values, and the specific number of the preset times is not limited herein.

Optionally, step S1642 and step S1644 are cyclically performed to obtain a preset number of feedback amounts of the first network structure.

It is understandable that every time you loop, you can sample a second network structure. After this network structure is tested on a batch of training data in the test set, you can get a feedback about the first network structure, and repeat enough times. Then, a preset number of feedback amounts of the first network structure can be obtained.

In an example, the preset number is 20, that is, the feedback amount of the first network structure is 20. It can be understood that in other examples, the preset number may be 10, 15, 25 or other numerical values. The specific value of the preset number is not limited here.

Optionally, the first network structure is constructed according to a long and short-term memory network model, and step S164 is implemented by the following conditional formula:

Wherein, R _k is the k th feedback, θ _c is the short and long term memory parameter network model, a _t t th said operating layer in the sampled _{operation, P (a t | a (} t-1): ₁ ; θ _c ) is the probability of sampling to operation, m is the total number of feedbacks, and T is the number of hyperparameters predicted by the first network structure.

In this way, the first network structure is updated according to the average value of multiple feedback quantities. In this embodiment, T includes an operating layer and jumpers. In other embodiments, other hyperparameters to be optimized may also be included. The specific content of T is not limited here.

In this embodiment, the training of the overall graph and the update of the first network structure are performed in multiple iterations, and the total number of second iterations of alternate training of the overall graph and the first network structure is 310 times. In other words, step S14 and step S16 are iterated 310 times. In this way, a better second network structure can be finally obtained.

In each alternate training, the preset number of cycles of step S16 is 50 times. In other words, in each iteration of step S14 and step S16, the first network structure is updated 50 times and executed 50 times according to the above conditional expression. It can be understood that selecting the number of loops in step S16 to be 50 can reduce the randomness optimization caused by sampling.

Each time step S16 is looped, the number of sampled second network structures is 20 to obtain 20 feedback quantities, and the 20 feedback quantities are substituted into the above conditional expression as R _k to realize the update of the first network structure. In other words, in the above conditional expression, the value of m is 20.

Please note that once the training sample is traversed, it is an iteration. For example, the number of training samples is 10, and each time 2 samples are divided as a batch of the training set to train the subgraph. After 5 batches, the training samples are used up, and one iteration is completed.

Optionally, referring to FIG. 10, in step S164, the step of determining the feedback amount includes:

Step S1646: Adjust the feedback amount of the first network structure according to the preset penalty model.

Correspondingly, step S1646 can be implemented by the processor 104, that is, the processor 104 can be used to adjust the feedback amount of the first network structure according to a preset penalty model.

It can be understood that the pre-training process may not train the overall image to fully converge. At this time, if the overall image fails to fully converge during the pre-training process, the feedback amount is adjusted according to the preset penalty model to avoid the The prejudice caused by optimizing the first network structure when the convergence is insufficient, thereby further improving the credibility of the first network structure, and ensuring that the searched model is the global optimum.

Optionally, referring to FIG. 11, step S1646 includes:

Step S1648: Determine the penalty item according to the preset information, the second network structure, the current number of iterations and the total number of second iterations;

Step S1649: Adjust the feedback amount according to the penalty item.

Correspondingly, step S1648 and step S1649 can be implemented by the processor 104, that is, the processor 104 is used to determine the penalty item according to the preset information, the second network structure, the current number of iterations and the total number of iterations; and Adjust the amount of feedback according to the penalty.

In this way, the feedback amount is adjusted according to the penalty model. It can be understood that in order to avoid the introduction of bias in the optimization of the first network structure in the case of insufficient convergence of the general map during the search process. We add a penalty term to the larger convolution kernel to deal with the amount of feedback during prediction.

Specifically, the preset information in step S1648 may be information input in advance by the user, or may be a result input by another calculation model. The source of the preset information is not limited here.

Please note that in this embodiment, step S162 and step S164 are performed iteratively. The total number of second iterations refers to the total number of iterations of step S12 and step S164. The current iteration number refers to how many iterations the current iteration is during the iteration process of step S162 and step S164. For example, the total number of the second iteration is 310. In the first iteration, the current iteration number is 1; in the second iteration, the current iteration number is 2; in the third iteration, the current iteration number is 3...and so on, step S162 and step S164 iterate 310 times and stop the iteration.

Therefore, the total number of second iterations and the current number of iterations in each iteration can also be determined. The total number of second iterations can be set by the user. The current number of iterations can be accumulated as the iteration progresses.

Please refer to Figure 12 together. As the iteration progresses, the overall graph gradually converges. It can be understood that the value of the penalty term increases with the convergence process of the overall graph, where the overall graph converges in the later stage of training and should be given a normal amount of feedback instead of penalizing the amount of feedback. Therefore, the overall graph converges When, the value of the penalty item is 1. Please note that as the iterations of step S162 and step S164 proceed, the overall graph gradually converges. After the total number of iterations of the second iteration, the overall graph is in a convergent state.

It can be seen from Figure 12 that the value of the penalty term can be gradually transitioned from 0.76 to 1. Correspondingly, the adjusted feedback amount gradually changes from ACC*0.76 to ACC.

In some embodiments, the size of the penalty item may be related to the degree of convergence of the total graph during pre-training. At this time, the preset information may include the total number of first iterations of pre-training, that is, step S1648 may be based on the pre-training. The total number of first iterations of training, the second network structure, the current number of iterations, and the total number of second iterations determine the penalty term.

The embodiment of the present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer executes the method of any of the above embodiments.

The embodiment of the present application also provides a computer program product containing instructions, which when executed by a computer causes the computer to execute the method of any one of the foregoing embodiments.

The computer storage medium and the computer program product of the embodiment of the present application optimize the network structure and the first network structure alternately, and adjust the feedback amount according to the preset penalty model, which can avoid the optimization of the first network structure when the overall map is fully converged. Therefore, the credibility of the first network structure is improved, and the searched model is guaranteed to be globally optimal.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc. .

A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (18)

A method for network structure search, characterized in that it comprises: The step of defining search space: determining the search space of the neural network model to be searched for the network structure, the search space defining various operations on the operation layer between every two nodes in the neural network model; Pre-training step: training the general map of the search space according to the first network structure with preset parameters of the first network structure to obtain the general map with pre-training parameters, the general map being constituted by the operation; Training step: training the general map with the pre-training parameters according to the first network structure and updating the first network structure according to the amount of feedback of the first network structure. The method for searching a network structure according to claim 1, wherein the pre-training step comprises: Sampling one operation at each operation layer of the search space according to the first network structure and preset parameters of the first network structure to obtain a sub-picture of the overall picture; A batch of training data of the training set is used to train the sub-pictures of the overall image to obtain the overall image with the pre-training parameters. The method for searching a network structure according to claim 1, wherein the training step comprises: Training the general map step: training the general map with the pre-training parameters according to the first network structure; The step of training the first network structure: determining the feedback amount and updating the first network structure according to the feedback amount. The method for searching a network structure according to claim 3, wherein the step of training the general map and the step of training the first network structure are performed iteratively. The method for network structure search according to claim 3, wherein the step of training the master map comprises: Sampling one of the operations in each of the operation layers of the search space according to the first network structure to obtain a sub-picture of the overall picture with the pre-training parameters; A batch of training data of the training set is used to train the subgraph. The method for searching the network structure according to claim 3, wherein the step of training the first network structure loops a preset number of times, and updating the first network structure according to the feedback amount, comprising: The first network structure is updated by using the feedback amount determined in each cycle, and the number of feedback amounts determined in each cycle is a preset number. The method for network structure search according to claim 6, wherein determining the feedback amount comprises: Sampling one operation at each operation layer of the search space according to the first network structure to obtain a second network structure; Using a batch of test data in the test set to predict the second network structure to determine the amount of feedback. The network structure search method according to claim 6, wherein the first network structure is constructed according to a long- and short-term memory network model, and the first network structure is updated according to the feedback amount, which is realized by the following conditional expression :

Wherein, R _k is the k th feedback amount, θ _c is the parameter for the short and long term memory network model, a _t t th as the handle layer to the sampling operation, P (a _t | a _(t-1):1 ; θ _c ) is the probability of sampling to the operation, m is the total number of feedback amounts, and T is the number of hyperparameters predicted by the first network structure. A network structure search device, characterized in that it comprises a processor and a memory, the memory stores one or more programs, and when the programs are executed by the processor, the processor is used to execute: The step of defining search space: determining the search space of the neural network model to be searched for the network structure, the search space defining various operations on the operation layer between every two nodes in the neural network model; Pre-training step: training the general map of the search space according to the first network structure with preset parameters of the first network structure to obtain the general map with pre-training parameters, the general map being constituted by the operation; Training step: training the general map with the pre-training parameters according to the first network structure and updating the first network structure according to the amount of feedback of the first network structure. The network structure search device according to claim 9, wherein the processor is configured to use the preset parameters of the first network structure in each of the search spaces according to the first network structure. The operation layer samples one of the operations to obtain a sub-image of the overall image; and is used to train the sub-images of the overall image using a batch of training data of the training set to obtain the overall image with the pre-training parameters . The network structure search device according to claim 9, wherein the processor is used for training a general map: training the general map with the pre-training parameters according to the first network structure; and for training The first network structure: determining the feedback amount and updating the first network structure according to the feedback amount. The network structure search device according to claim 11, wherein the processor is configured to iteratively perform the training master map and the training first network structure. The device for searching a network structure according to claim 11, wherein the processor is configured to sample one of the operations at each operation layer of the search space according to the first network structure to obtain the A sub-picture of the overall picture of the pre-training parameters, and a batch of training data for training the sub-picture using a training set. The network structure search device according to claim 11, wherein the processor is configured to loop the step of training the first network structure for a preset number of times, and is configured to update the feedback amount determined by each loop. In the first network structure, the number of the feedback amounts determined in each cycle is a preset number. The device for searching a network structure according to claim 14, wherein the processor is configured to sample one of the operations at each operation layer of the search space according to the first network structure to obtain the second network structure And for using a batch of test data in the test set to predict the second network structure to determine the amount of feedback. The network structure search device according to claim 14, wherein the first network structure is constructed according to a long and short-term memory network model, and the processor is configured to update the first network structure using the feedback amount , Through the following conditional expression:

Wherein, R _k is the k th feedback amount, θ _c is the parameter for the short and long term memory network model, a _t t th as the handle layer to the sampling operation, P (a _t | a _(t-1):1 ; θ _c ) is the probability of sampling to the operation, m is the total number of feedback amounts, and T is the number of hyperparameters predicted by the first network structure. A computer storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a computer, the computer executes the method according to any one of claims 1 to 8. A computer program product containing instructions, wherein when the instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 8.

Images