CN118196600A - Neural architecture searching method and system based on differential evolution algorithm - Google Patents

Abstract

The present invention provides a neural architecture search method and system based on a differential evolution algorithm. The method first defines a search space, combines all candidate operations together to construct a supernet, and encodes them using continuous numerical values. Next, the network weights of the supernet are trained using training data and their labels. Then, a unique encoding method is designed for each subnet in the supernet, and the network weights are obtained directly from the supernet. Subsequently, the encoding of the subnet is optimized using a differential evolution algorithm. Finally, the network weight optimization of the supernet and the encoding optimization of the subnet are performed alternately. Unlike the method of optimizing subnet encoding using a gradient descent method and a genetic algorithm, the differential evolution algorithm of the present invention can make more full use of the vectorized information of the continuous encoding. Compared with traditional optimization methods, this method has higher efficiency and better search performance, and can find the optimal neural network architecture suitable for the task more quickly.

Description

Neural architecture search method and system based on differential evolution algorithm

Technical Field

The present invention relates to the field of deep neural network technology, and in particular to a neural architecture search method and system based on a differential evolution algorithm.

Background technique

Deep learning is a branch of machine learning that simulates and learns complex nonlinear relationships by building deep neural networks. As deep learning models have achieved outstanding results in tasks such as image recognition and natural language processing, they have attracted widespread attention in scientific research and industrial applications. Traditional neural network architectures are manually designed by experts, and then the architecture is continuously adjusted according to the network effect. This method requires a lot of manpower and expert experience. In order to adapt to the development of deep neural network technology and automatically obtain a network architecture suitable for the target data distribution, neural network architecture search technology came into being. Neural network architecture search is a method for automatically designing neural network architectures. This technology uses search algorithms to automatically discover high-performance neural network architectures in a pre-defined search space, which allows even non-neural network professionals to make good use of neural network technology in their fields. The emergence of this method not only makes the design and optimization of neural networks more efficient, but also provides researchers and engineers from all walks of life with broader application prospects and opportunities.

Neural network architecture search is an important research field that aims to automatically discover the optimal neural network structure to meet the needs of specific tasks. In this process, it is necessary to find an effective combination from the huge search space so that the network can fully learn and perform well. In the early days, neural network architecture search algorithms mainly relied on methods such as reinforcement learning and evolutionary computing. These methods require full training after each sampling of a new architecture, and use its true accuracy as the basis for optimization. Although these algorithms have achieved remarkable results in improving network performance, the efficiency is not ideal because each training requires a lot of time and computing resources.

In order to solve this problem, the differentiable architecture search framework has emerged in recent years. This method makes the entire search space continuous by assigning an architecture weight to each candidate operation, so as to use optimization methods such as gradient descent to adjust the network weight and architecture weight at the same time. When the network converges, the final discretized network is obtained according to the architecture weight value. The emergence of this method has significantly improved the efficiency of neural network architecture search, making network design more flexible and efficient under large-scale tasks. However, this method still has some problems. First, the gradient descent method is difficult to jump out of the current solution when searching for a local optimal solution. Even if the stochastic gradient descent method and the concept of momentum are introduced, this problem cannot be completely solved. Secondly, the differentiable architecture search algorithm is implemented on the entire supernet, and the network architecture weights of all candidate operations are optimized at the same time, which leads to interference between operations in this process, resulting in a certain degree of coupling. Therefore, it is very necessary to propose a global search algorithm based on evolutionary computation methods to jump out of the local optimal solution and evaluate the subnet separately.

Summary of the invention

Purpose of the invention: The technical problem to be solved by the present invention is to provide a neural architecture search method and system based on a differential evolution algorithm in view of the deficiencies in the prior art.

The method comprises:

Step 1: construct a supernet containing M connecting edges and N operations. Each subnet uses a continuous value in a range to assign a weight to each operation on each connecting edge. The weight is used as a code to uniquely represent an individual in a population.

Step 2, use the training data and its labels to train the network weights of the supernet;

Step 3, using differential evolution algorithm to optimize subnet coding;

Step 4: Alternately execute steps 2 and 3 until the error rate of the network architecture within the population converges.

In step 1, each edge in the supernet contains N candidate operations, where N is a natural number; M corresponds to the internal nodes set in the supernet, and the number of internal nodes is H. ;

The supernet includes X subnets, where the value of X is , the supernet needs to be used in the stage of training the network weights using the gradient descent method, that is, the parameters of all candidate operations in the search space are trained. The subnet represents a network with only one candidate operation on all connected edges. It is part of the supernet. The present invention uses a differential evolution algorithm to optimize the encoding of the subnet to ensure that the best subnet architecture is found during the search process. The main function of the supernet is to train the network weights of all candidate operations so as to directly provide the network weights required to evaluate the performance of the subnet.

A supernet is a general term for a hybrid network that includes all candidate operations and topological connections in the search space. Generally speaking, input information is often processed by a single operation, such as a 3*3 convolution to obtain a new feature map, but in a supernet, the input information is processed by multiple operations, such as 3*3 dynamic separable convolution, 3*3 hole convolution, jump connection, and a series of operations, and the feature maps obtained by each are added according to the corresponding element positions to obtain a new hybrid feature map information.

In the present invention, a continuous value in the range of 0 to 1 is used to assign a weight to each operation on each connection edge. This encoding method allows the subnet to be discretized into a conventional neural network according to the size of the weight, that is, candidate operations and connection edges with smaller weights are removed.

In the present invention, the supernet includes M connection edges, and each connection edge in the supernet contains N candidate operations, including 3*3 dynamic separable convolution, 5*5 dynamic separable convolution, 3*3 dilated convolution, 5*5 dilated convolution, 3*3 maximum pooling, 3*3 average pooling, skip connection and empty operation, etc., and the empty operation represents that the connection edge does not exist;

According to the above description, it is known that the supernet in the present invention is a network structure that mixes multiple operations, but the purpose of the present invention is to search through an algorithm to find a subnet that undergoes a single operation. For example, M connecting edges need to determine a unique operation on each connecting edge.

In order to indicate the required subnet, a number in the range of 0 to 1 is needed to represent the importance of each connecting edge, for example, an M*N matrix is used. Therefore, the purpose of the supernet is to store the trained network weights of various operations, and the encoding is for the subnet. In the population optimization method of evolutionary computation, the subnet is also regarded as an individual in the population. When the encoding shows different values, a variety of neural networks can be obtained according to the value size. In addition, it should be noted that N has no limited range. In theory, any convolution and pooling operation can be added to this mode. M corresponds to the set internal nodes. If the internal nodes are set to H, then .

In step 1, specifically, the encoding is in the form of a 14*8 matrix, which represents the 14 candidate connection edges in the supernet and the 8 candidate operations on the connection edges. As the subsequent search space expands, the dimension of the matrix can also be increased, that is, it contains more connection edges and candidate operations.

In step 2, the supernet needs to be trained before the weights of various operations inside it can be used;

When the supernet is initialized, in order to ensure that the sum of the values of all candidate operations on a connection edge is 1 and each operation is equally important, the value of each element in the matrix is , ensuring that the parameters of each operation in the supernet are trained. In the subsequent optimization process, after generating the subnet code, the code of the better individual in the subnet is assigned to the supernet to guide the supernet to optimize the network weight. The internal value of the code will be inclined, and the operation with a larger architecture weight can often be trained more fully. Specifically, when optimizing the supernet, the feature map information on different connection edges will be spliced in the channel dimension when it converges to the same node, expressed as/> , where concat represents the concatenation function, which operates on different feature maps of the input./> and/> To achieve splicing in the channel dimension, the feature map on each connecting edge is a mixture of the feature maps obtained by the candidate operation after processing the input data, expressed as/> , where /> Indicates the architecture weight corresponding to the i-th operation, input indicates the input data, /> It represents the process of extracting input information features by the i-th operation; after such processing, the neural network back propagation update algorithm is optimized to cover all candidate operations in the entire network architecture.

The input data is image information obtained by processing the stem layer (called the initial layer, including a two-dimensional convolution and batch normalization, whose main function is to pre-process the input data of general images, extract features and convert them into feature representation forms suitable for subsequent processing). For general images, directly using them as input will make it difficult for the neural network to extract common abstract information, so the present invention uses the stem layer composed of convolution operations to obtain more representative image information.

The training data is derived from real-life application scenarios. For example, in the medical field, if the type of skin disease needs to be determined based on skin images, the disease type determined by the existing image needs to be assigned as a label. The neural network searched based on the existing image can be trained to perform medical auxiliary processing on the newly generated medical image. After the judgment is completed, the new image is added to the image library to update the network architecture and parameters. After such processing, the neural network back-propagation update algorithm can cover all candidate operations in the entire network architecture, thereby optimizing the weights of all operations in the supernet.

Step 3 includes:

Step 3.1, initialize the population individuals, including subnet coding and scaling factors;

Step 3.2, use the evolution operator to update the scaling factor;

Step 3.3, update the subnetwork encoding using the evolution operator containing the adaptive factor;

Step 3.4, the execution environment selects and determines the subnet encoding and scaling factor of the new generation.

Step 3.1 includes:

Step 3.1.1, according to the definition of N candidate operations for each link, the encoding method of the jth link of the nth individual in the gth generation population is defined as ,/> Represents a connection edge vector encoding,/> Indicates the value at the Nth position in the vector code;/> The /> on a connecting edge in step 2 correspond;

Step 3.1.2, according to the setting that a supernet has M connecting edges, the encoding method of the nth individual in the gth generation population is specified as , where /> represents the result of transposing the code of the Mth edge in step 3.1.1,/> Represents the matrix encoding of the nth individual in the gth generation population;

Step 3.1.3, according to the upper and lower limits of the continuous value, randomly initialize each part of the subnet individual ,/> Indicates a random number between 0 and 1, /> represents the minimum value obtained at the Nth position on the connecting edge of the nth individual in the gth generation population. In the present invention, this value is 0; represents the maximum value obtained at the Nth position on the connecting edge of the nth individual in the gth generation population. In the present invention, this value is 1;

Step 3.1.4: Change the expansion factor of the nth individual in the gth generation population to As a parameter that obtains information from the vector encoding difference, it is initially set to a random number between 0 and 1:/> .

Step 3.2 includes:

Step 3.2.1, traverse the individuals in the population, and randomly select non-repeating individuals from the g-th generation population each time, which are ,/> and/> , the three non-repeating individual scaling factor codes are respectively/> , and/> , the subnet codes of the three non-repeating individuals are respectively/> 、/> and/> ;For the evolution operator, the encoding of one individual is used as the basis vector, and the difference between the encodings of the other two is calculated;

Step 3.2.2, select the scaling factor encoding of 2 individuals from the 3 randomly selected individuals to calculate the vector information difference ;

Step 3.2.3, multiply the calculated vector code difference by the scaling factor of the nth individual in the gth generation, expressed as , where/> It represents the expansion factor after mutation during the g-th generation population traversal. The adaptive factor in step 3.3 is obtained through this mutation operation.

Step 3.3 includes:

Step 3.3.1, select the corresponding subnet code from the three individuals randomly selected in step 3.2.1 and calculate the vector information difference ;

Step 3.3.2, use the remaining random subnet encoding that does not participate in the calculation of the vector information difference plus , and then get the mutated subnet code, expressed as/> , where/> Represents the new subnet code obtained by the mutation operator during the g-th generation population traversal process;

Step 3.3.3, perform crossover on the current traversal individual and the variant individual obtained in step 3.3.2 to obtain the encoding information of the jth part Among them, /> It refers to the jth part of the nth individual in the population encoded from the atomic network/> and the mutated subnet code/> Specifically, each has a 50% probability of obtaining the j-th part of the coded information.

Step 3.4 includes:

Step 3.4.1, discretize the subnet code according to its numerical value: only keep the code with the largest value after optimization on each connection edge, and convert the discretized code into 0-1 code, that is, the code corresponding to the operation retained on each connection edge is 1, and the rest are 0. The formula is expressed as ;

Step 3.4.2, calculate the fitness of the nth individual in the gth generation population in the current population And the fitness of the nth individual obtained after being processed by the evolutionary operator/> ：Use the confusion matrix C to measure the relationship between the prediction results and the true labels in the classification model. Suppose there is an image classification task with L categories. The rows of the confusion matrix represent the predicted categories of the model, and the columns represent the true categories of the image. Set the total number of samples to/> , the number of correctly classified samples is/> , the calculation formula is:

,

in, Represents the element value of the k-th row and k-th column in the confusion matrix C;

,

in, Represents the element value of the kth row and lth column in the confusion matrix, where k and l both represent index numbers; the calculation formula for the accuracy, i.e., fitness, is:

;

Step 3.4.3, compare the nth individual in the gth generation population The corresponding mutated individual/> The higher fitness value is saved to the next generation population:

,

in, Indicates the individuals saved to the next generation, which includes the subnet code and the scaling factor. If the individual fitness function obtained by the evolution operator is higher, the nth individual in the population obtained by evolution will be saved. The subnet code and scaling factor are preserved, otherwise the nth individual in the original g-th generation population is retained/> Subnet coding and scaling factor; Indicates the nth individual in the population obtained by evolution/> Subnet code in /> The fitness of Indicates the nth individual in the original g-generation population/> Subnet code in /> The fitness of .

In step 4, in order to reduce the computing resource overhead, the network weight and subnet coding are optimized alternately until the error rate of the network architecture in the population converges. It is difficult to train the supernet completely at one time because it is unknown how to allocate the training weights of the supernet. Therefore, the present invention provides a strategy for training the supernet guided by excellent individuals in the population, that is, on the one hand, the supernet needs to be trained to provide a basis for optimizing the coding, and on the other hand, the supernet needs to copy the coding information of the excellent individuals in the population to guide the optimization of the supernet network weights, which can be expressed as:

Subject to/> ,

in Indicates the subnet code, /> The subnet code corresponding to the individual with the highest fitness,/> represents the fitness calculation function, W represents the weight in the supernet, /> Indicates the value of weight W when the loss function value is the lowest, Indicates that the maximum value in the brackets is returned /> ,/> It means returning the smallest value in the brackets, W, and dividing the training data set into two parts, denoted as/> and/> , where the data set /> The data in is used to train the supernet network weights, data set/> The data in is used to evaluate the subnetwork performance in the process of optimizing the subnetwork encoding, and Loss represents the cross entropy loss function commonly used in classification tasks.

The present invention also provides a system based on the neural architecture search method based on the differential evolution algorithm, the system comprising:

An encoding module, used to construct a supernet including M connecting edges and N operations, each subnet uses a continuous value ranging from 0 to 1 to assign a weight to each operation on each connecting edge, and the weight is used as a code to uniquely represent an individual in a population;

A training module for training the network weights of the supernet using training data and its labels;

The optimization module is used to optimize the subnet encoding using the differential evolution algorithm.

By introducing evolutionary computation methods, the present invention can better explore the entire search space, thereby avoiding falling into a local optimal solution. At the same time, evaluating the subnets individually can reduce interference between operations and improve the efficiency and accuracy of the search. Such a global search algorithm is expected to solve the limitations of the current differentiable architecture search algorithm and bring new breakthroughs and progress to the field of neural network architecture search. Therefore, this method proposes a method for learning operation weights using a multilayer perceptron based on the input information of each cell. This method has achieved better results than existing methods.

The beneficial effects of the present invention are:

The method of the present invention proposes a neural architecture search algorithm based on a differential evolution algorithm for differentiable neural network architecture search. The algorithm optimizes individual codes in a population through a differential evolution algorithm, and optimizes supernet network weights by a gradient descent method. In this method, we adopt a continuous coding scheme to make the entire search space continuous, which can be used to train all operation weight parameters in the supernet. Each time the differential evolution algorithm is used to evaluate the fitness of individuals in a population, the weights are directly copied from the supernet and the fitness is calculated without the need to completely retrain the network architecture, thereby greatly reducing the computational overhead. The differential evolution algorithm is targeted at continuous coding and can make full use of the vectorized information in the coding, so as to more effectively search for better architectures. In addition, an adaptive factor is designed for the evolution of subnet coding, so that the subnet performance guides the evolution of the scaling factor, thereby assisting the differential evolution algorithm in optimizing the subnet coding. This neural architecture search algorithm based on the differential evolution algorithm brings new methods and ideas to the field of neural network architecture search, and has high practical value and broad application prospects.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and/or other advantages of the present invention will become more clear.

FIG. 1 is an overall framework diagram of the present invention.

FIG. 2 is a diagram describing the variation in the present invention.

FIG3 is a schematic diagram of the super network accuracy obtained through training.

FIG. 4 is a schematic diagram comparing the method of the present invention with other methods.

Detailed ways

As shown in FIG1 , the present invention provides a neural architecture search method and system based on a differential evolution algorithm. The method comprises:

Step 1: construct a supernet containing M connecting edges and N operations. Each subnet uses a continuous value in a range to assign a weight to each operation on each connecting edge. The weight is used as a code to uniquely represent an individual in a population.

Step 2, use the training data and its labels to train the network weights of the supernet;

Step 3, using differential evolution algorithm to optimize subnet coding;

Step 4: Alternately execute steps 2 and 3 until the error rate of the network architecture within the population converges.

In step 1, each edge in the supernet contains N candidate operations, where N is a natural number; M corresponds to the internal nodes set in the supernet, and the number of internal nodes is H. ;

The supernet includes X subnets, where the value of X is , the supernet needs to be used in the stage of training the network weights using the gradient descent method, that is, the parameters of all candidate operations in the search space are trained. The subnet represents a network with only one candidate operation on all connected edges. It is part of the supernet. The present invention uses a differential evolution algorithm to optimize the encoding of the subnet to ensure that the best subnet architecture is found during the search process. The main function of the supernet is to train the network weights of all candidate operations so as to directly provide the network weights required to evaluate the performance of the subnet.

Supernet is a general term for hybrid networks that include all candidate operations and topological connections in the search space. Generally speaking, input information is often processed by a single operation, such as a 3*3 convolution to obtain a new feature map, but in the supernet, the input information will be processed by multiple operations, such as 3*3 dynamic separable convolution, 3*3 void convolution, jump connection, and a series of operations, and the feature maps obtained by each are added according to the corresponding element positions to obtain a new hybrid feature map information.

In the present invention, a continuous value in the range of 0 to 1 is used to assign a weight to each operation on each connection edge. This encoding method allows the subnet to be discretized into a conventional neural network according to the size of the weight, that is, candidate operations and connection edges with smaller weights are removed.

In the present invention, the supernet includes M connection edges, and each connection edge in the supernet contains N candidate operations, including 3*3 dynamic separable convolution, 5*5 dynamic separable convolution, 3*3 dilated convolution, 5*5 dilated convolution, 3*3 maximum pooling, 3*3 average pooling, skip connection and empty operation, etc., and the empty operation represents that the connection edge does not exist;

According to the above description, it is known that the supernet in the present invention is a network structure that mixes multiple operations, but the purpose of the present invention is to search through an algorithm to find a subnet that undergoes a single operation. For example, M connecting edges need to determine a unique operation on each connecting edge.

In order to indicate the required subnet, a number in the range of 0 to 1 is needed to represent the importance of each connecting edge, for example, an M*N matrix is used. Therefore, the purpose of the supernet is to store the trained network weights of various operations, and the encoding is for the subnet. In the population optimization method of evolutionary computation, the subnet is also regarded as an individual in the population. When the encoding shows different values, a variety of neural networks can be obtained according to the value size. In addition, it should be noted that N has no limited range. In theory, any convolution and pooling operation can be added to this mode. M corresponds to the set internal nodes. If the internal nodes are set to H, then .

In step 1, specifically, the encoding is in the form of a 14*8 matrix, which represents the 14 candidate connection edges in the supernet and the 8 candidate operations on the connection edges. As the subsequent search space expands, the dimension of the matrix can also be increased, that is, it contains more connection edges and candidate operations.

In step 2, the supernet needs to be trained before the weights of various operations inside it can be used;

When the supernet is initialized, in order to ensure that the sum of the values of all candidate operations on a connection edge is 1 and each operation is equally important, the value of each element in the matrix is , ensuring that the parameters of each operation in the supernet are trained. In the subsequent optimization process, after generating the subnet code, the code of the better individual in the subnet is assigned to the supernet to guide the supernet to optimize the network weight. The internal value of the code will be inclined, and the operation with a larger architecture weight can often be trained more fully. Specifically, when optimizing the supernet, the feature map information on different connection edges will be spliced in the channel dimension when it converges to the same node, expressed as/> , where concat represents the concatenation function, which operates on different feature maps of the input./> and/> To achieve splicing in the channel dimension, the feature map on each connecting edge is a mixture of the feature maps obtained by the candidate operation after processing the input data, expressed as/> , where /> Indicates the architecture weight corresponding to the i-th operation, input indicates the input data, /> It represents the process of extracting input information features by the i-th operation; after such processing, the neural network back propagation update algorithm is optimized to cover all candidate operations in the entire network architecture.

The input data is image information obtained by processing the stem layer (called the initial layer, including a two-dimensional convolution and batch normalization, whose main function is to pre-process the input data of general images, extract features and convert them into feature representation forms suitable for subsequent processing). For general images, directly using them as input will make it difficult for the neural network to extract common abstract information, so the present invention uses the stem layer composed of convolution operations to obtain more representative image information.

The training data is derived from real-life application scenarios. For example, in the medical field, if the type of skin disease needs to be determined based on skin images, the disease type determined by the existing image needs to be assigned as a label. The neural network searched based on the existing image can be trained to perform medical auxiliary processing on the newly generated medical image. After the judgment is completed, the new image is added to the image library to update the network architecture and parameters. After such processing, the neural network back-propagation update algorithm can cover all candidate operations in the entire network architecture, thereby optimizing the weights of all operations in the supernet.

Step 3 includes:

Step 3.1, initialize the population individuals, including subnet coding and scaling factors;

Step 3.2, use the evolution operator to update the scaling factor;

Step 3.3, update the subnetwork encoding using the evolution operator containing the adaptive factor;

Step 3.4, the execution environment selects and determines the subnet encoding and scaling factor of the new generation.

Step 3.1 includes:

Step 3.1.1, according to the definition of N candidate operations for each link, the encoding method of the jth link of the nth individual in the gth generation population is defined as ,/> Represents a connection edge vector encoding,/> Indicates the value at the Nth position in the vector code;/> The /> on a connecting edge in step 2 correspond;

Step 3.1.2, according to the setting that a supernet has M connecting edges, the encoding method of the nth individual in the gth generation population is specified as , where /> represents the result of transposing the code of the Mth edge in step 3.1.1,/> Represents the matrix encoding of the nth individual in the gth generation population;

Step 3.1.3, according to the upper and lower limits of the continuous value, randomly initialize each part of the subnet individual ,/> Indicates a random number between 0 and 1, /> represents the minimum value obtained at the Nth position on the connecting edge of the nth individual in the gth generation population. In the present invention, this value is 0; represents the maximum value obtained at the Nth position on the connecting edge of the nth individual in the gth generation population. In the present invention, this value is 1;

Step 3.1.4: Change the expansion factor of the nth individual in the gth generation population to As a parameter that obtains information from the vector encoding difference, it is initially set to a random number between 0 and 1:/> .

Step 3.2 includes:

Step 3.2.1, traverse the individuals in the population, and randomly select non-repeating individuals from the g-th generation population each time, which are ,/> and/> , the three non-repeating individual scaling factor codes are respectively/> , and/> , the subnet codes of the three non-repeating individuals are respectively/> 、/> and/> ;For the evolution operator, the encoding of one individual is used as the basis vector, and the difference between the encodings of the other two is calculated;

Step 3.2.2, select the scaling factor encoding of 2 individuals from the 3 randomly selected individuals to calculate the vector information difference ;

Step 3.2.3, multiply the calculated vector code difference by the scaling factor of the nth individual in the gth generation, expressed as , where/> It represents the expansion factor after mutation during the g-th generation population traversal. The adaptive factor in step 3.3 is obtained through this mutation operation.

Step 3.3 includes:

Step 3.3.1, select the corresponding subnet code from the three individuals randomly selected in step 3.2.1 and calculate the vector information difference ;

Step 3.3.2, use the remaining random subnet encoding that does not participate in the calculation of the vector information difference plus , and then get the mutated subnet code, expressed as/> , where/> Represents the new subnet code obtained by the mutation operator during the g-th generation population traversal process;

Step 3.3.3, perform crossover on the current traversal individual and the variant individual obtained in step 3.3.2 to obtain the encoding information of the jth part Among them, /> It refers to the jth part of the nth individual in the population encoded from the atomic network/> and the mutated subnet code/> Specifically, each has a 50% probability of obtaining the j-th part of the coded information.

Step 3.4 includes:

Step 3.4.1, discretize the subnet code according to its numerical value: only keep the code with the largest value after optimization on each connection edge, and convert the discretized code into 0-1 code, that is, the code corresponding to the operation retained on each connection edge is 1, and the rest are 0. The formula is expressed as ;

Step 3.4.2, calculate the fitness of the nth individual in the gth generation population in the current population And the fitness of the nth individual obtained after being processed by the evolutionary operator/> ：Use the confusion matrix C to measure the relationship between the prediction results and the true labels in the classification model. Suppose there is an image classification task containing L categories. The rows of the confusion matrix represent the predicted categories of the model, and the columns represent the true categories of the image. Set the total number of samples to/> , the number of correctly classified samples is/> , the calculation formula is:

,

in, Represents the element value of the k-th row and k-th column in the confusion matrix C;

,

in, Represents the element value of the kth row and lth column in the confusion matrix, where k and l both represent index numbers; the calculation formula for the accuracy, i.e., fitness, is:

;

Step 3.4.3, compare the nth individual in the gth generation population The corresponding mutated individual/> The higher fitness value is saved to the next generation population:

,

in, Indicates the individuals saved to the next generation, which includes the subnet code and the scaling factor. If the individual fitness function obtained by the evolution operator is higher, the nth individual in the population obtained by evolution will be saved. The subnet code and scaling factor are preserved, otherwise the nth individual in the original g-th generation population is retained/> Subnet coding and scaling factor; Indicates the nth individual in the population obtained by evolution/> Subnet code in /> The fitness of Indicates the nth individual in the original g-generation population/> Subnet code in /> The fitness of .

In step 4, in order to reduce the computing resource overhead, the network weight and subnet coding are optimized alternately until the error rate of the network architecture in the population converges. It is difficult to train the supernet completely at one time because it is unknown how to allocate the training weights of the supernet. Therefore, the present invention provides a strategy for training the supernet guided by excellent individuals in the population, that is, on the one hand, the supernet needs to be trained to provide a basis for optimizing the coding, and on the other hand, the supernet needs to copy the coding information of the excellent individuals in the population to guide the optimization of the supernet network weights, which can be expressed as:

Subject to/> ,

in Indicates the subnet code, /> The subnet code corresponding to the individual with the highest fitness,/> represents the fitness calculation function, W represents the weight in the supernet, /> Indicates the value of weight W when the loss function value is the lowest, Indicates that the maximum value in the brackets is returned /> ,/> It means returning the smallest value in the brackets, W, and dividing the training data set into two parts, denoted as/> and/> , where the data set /> The data in is used to train the supernet network weights, data set/> The data in is used to evaluate the subnetwork performance in the process of optimizing the subnetwork encoding, and Loss represents the cross entropy loss function commonly used in classification tasks.

The present invention also provides a system based on the neural architecture search method based on the differential evolution algorithm, the system comprising:

An encoding module, used to construct a supernet including M connecting edges and N operations, each subnet uses a continuous value ranging from 0 to 1 to assign a weight to each operation on each connecting edge, and the weight is used as a code to uniquely represent an individual in a population;

A training module for training the network weights of the supernet using training data and its labels;

The optimization module is used to optimize the subnet encoding using the differential evolution algorithm.

In a specific embodiment of the present invention, FIG1 takes a search space including 6 connecting edges and 3 candidate operations as an example to introduce in detail the implementation process and details of the neural architecture search method and system based on the differential evolution algorithm proposed in the present invention.

As shown in FIG1 , the neural architecture search method based on the differential evolution algorithm proposed in the present invention includes the following steps:

Step 1: First, construct a supernet as shown in the left sub-graph of Figure 1. The supernet includes 4 nodes representing feature maps. The nodes have a sequence, and the feature map can only be transferred from the previous node to the next node after processing; the supernet has 3 connecting edges representing candidate operation processing, where Op ₁ represents 3*3 convolution, Op ₂ represents 5*5 convolution, and Op ₃ represents skip connection. Taking the input data on node 0 in the supernet as an example, the feature maps obtained by processing the input information of the three candidate operations will be weighted and summed according to the encoded weights, and the obtained feature map information is stored on node 1. For subsequent nodes such as node 2 and node 3, they obtain feature map information from multiple previous nodes. At this time, multiple feature map information needs to be spliced in the channel dimension. It should be noted that, taking the description of Figure 1 as an example, the supernet only assists the training of network weights, and it does not have a specific encoding, and the individuals in the optimized population have the encoding corresponding to all candidate operations in the supernet.

Step 2: The codes in Figure 1 correspond to the subnet individuals one by one. represents the architecture weight, and the number in the brackets represents the node number. When training the supernet, it serves as the weight information in the weighted sum of the feature graph, so that all candidate operations in the supernet participate in the network training. In order to show the advantages of excellent individuals in the population, the encoding weights are assigned to the supernet from the individuals in the population in turn to optimize the supernet network weights. Next, the complete supernet training process is introduced using the input data of a real scene as an example. First, the image data of the target scene needs to be collected; then, a 224*224 pixel image is cut out from the center of the image; then, the cut image is sent to the stem layer for feature extraction; finally, the extracted abstract features are used as the input of the supernet.

Step 3: As shown in FIG1 , in the process of alternating optimization, the subnet coding is optimized using the differential evolution algorithm based on the individual evaluation index within the population, including the following steps:

Step 3.1: Initialize the population individuals as shown in Figure 1, including the following steps:

Step 3.1.1: In this example, each link contains three candidate operations. The encoding method of the jth link of the nth individual in the gth generation population is specified as ;

Step 3.1.2: According to the setting that a supernet has 6 connecting edges, the encoding method of the nth individual is specified as ;

Step 3.1.3: Based on the upper and lower limits of the continuous values, take the first candidate operation on a connecting edge as an example to initialize the subnet individuals. , the number of individuals in the population is initialized to 50.

Step 3.2: Update the scaling factor using the evolution operator:

Step 3.2.1: Traverse the individuals in the population, randomly select 3 individuals from the current population each time, and use the individuals in a certain traversal as For example, three non-repeating individuals are randomly obtained, namely/> ,/> and/> , the subscript indicates the index number of the individual in the population, and all three individuals contain subnet codes/> ,/> and/> , and scaling factor encoding/> ,/> and/> ;

Step 3.2.2: Randomly select two of the three individuals to calculate the vector information difference and multiply it by the scaling factor of the currently traversed individual , and then add the resulting information to the third individual. In this example, the mutation scaling factor /> The specific calculation method is as follows: ;

Step 3.3: Update the subnetwork encoding using the evolution operator containing the adaptive scaling factor:

Step 3.3.1: Screen out the three individuals randomly selected in step 3.2 \/> and/> , and the corresponding subnet code/> ,/> and/> ;

Step 3.3.2: Multiply the scaling factor obtained in step 3.2 by the vector difference of the two subnet codes, and add the third subnet code to get the specific mutated subnet code. , the specific calculation method is/> , Figure 2 visualizes the calculation process, at this time/> special circumstances;

Step 3.3.3: Perform crossover on the current traversal individual and the mutant individual obtained in step 3.2.3. , where j represents the jth part of the subnet code. This part is obtained by crossing the traversed subnet code and its mutated subnet code, each with a half probability of forming a new subnet code.

Step 3.4: Execution environment selection determines the next generation of subnet encoding and scaling factor. The process includes:

Step 3.4.1: Discretize the continuous code of the subnet for performance evaluation. Taking Figure 2 as an example, the new subnet code contains specific values. There are three candidate operations between node 0 and node 1. The architecture weights of Op ₁ , Op ₂ , and Op ₃ are 0.66, 0.84, and 0.64 respectively. The candidate operation with the largest weight value is selected, so the operation between node 0 and node 1 in the discretized architecture is Op ₂ .

Step 3.4.2: Calculate the fitness of the traversed individuals in the current population And the fitness of the individual obtained after being processed by the evolutionary operator/> Specifically, the confusion matrix is used to measure the relationship between the prediction results and the true labels in the classification model. Assuming that there are an image classification task containing L categories, the rows of the confusion matrix represent the predicted categories of the model, and the columns represent the true categories of the image. Assuming the total number of samples is/> , the number of correctly classified samples is/> , then the calculation method of accuracy, i.e. fitness, is / > To calculate the number of correctly classified samples, we need to add up all the elements on the diagonal of the confusion matrix, that is, /> , calculate the total number of samples/> ;

Step 3.4.3: Compare the two fitness values, and save the higher one to the next generation population. , where /> Indicates the individuals saved to the next generation, which includes the subnet code and the scaling factor. That is, if the individual fitness function obtained by the evolution operator is higher, the evolved individual will be saved to the next generation. The subnet code and scaling factor of the subnet are preserved, otherwise the original individual Subnet coding and scaling factor.

Step 4: In order to reduce the computing resource overhead, the present invention adopts the method of alternating optimization of network weights and subnet codes until the error rate of the network architecture in the population converges. It is difficult to fully train the supernet in one go because it is unknown how to allocate the training weights of the supernet. Therefore, the present invention provides a strategy for training the supernet guided by excellent individuals in the population, that is, on the one hand, the training supernet needs to provide a basis for optimizing the coding, and on the other hand, the supernet needs to copy the coding information of the excellent individuals in the population to guide the optimization of the supernet network weights. The process is described as:

Subject to/> ,

in Indicates the subnet code, /> represents the fitness calculation function, W represents the weight in the supernet, /> Indicates the data set divided into parts used to train the supernet, /> Indicates the part of data used to optimize the subnet encoding, and Loss represents the cross entropy loss function commonly used in classification tasks. In the process of continuous iterative optimization, the optimal subnet is finally obtained. , specifically expressed as:

in, Indicates a dynamic separable convolution with a convolution kernel size of 5*5,/> Indicates a dynamic separable convolution with a convolution kernel size of 3*3,/> Indicates a dilated convolution with a kernel size of 3*3, /> Indicates the maximum pooling size of 3*3,/> represents a skip connection, /> Indicates average pooling of size 3*3.

Compared with the prior art, the present invention can alleviate the problem of existing methods falling into local optimality during the search process due to the excellent global search ability of the evolution operator. In addition, the evolution operator of the present invention pays more attention to population diversity during implementation, which can avoid the premature convergence problem caused by the small model trap in the optimization process. In order to facilitate comparison with other algorithms, the performance of the method of the present invention is tested on the public data set CIFAR-10, which contains 10 categories and 60,000 pictures, of which 50,000 are used for training and 10,000 are used for testing. The size of pictures of all categories is 32*32 and the number is equal. The present invention can search for a neural network architecture with an accuracy of up to 97.45% in only 4.8 hours on GPU 2080ti, which is 31.2 hours faster than the existing differentiable architecture search algorithm and 0.45% higher in accuracy. Specifically, the first step is to build a supernet, and sample an operation from each connection edge in the supernet to obtain a subnet, and repeat the sampling step 50 times to obtain the initial population; the accuracy of the supernet trained in step 2 is shown in Figure 3.

In the process of alternately optimizing the supernet network weights and the population individual codes, the present invention compares the average verification accuracy of all individuals in each generation with other methods. As shown in FIG4 , compared with the premature convergence problem of other methods caused by the small model trap (i.e., the orange curve converges prematurely and the accuracy decreases in the subsequent optimization process), the method of the present invention can converge normally to a lower verification error rate.

The present invention provides a neural architecture search method and system based on a differential evolution algorithm. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims (10)

1. The neural architecture searching method based on the differential evolution algorithm is characterized by comprising the following steps of:

Step 1, constructing a super-network comprising M connection sides and N operations, wherein each sub-network uses continuous values in a range to assign a weight to each operation on each connection side, and the weights are used as codes to uniquely represent individuals in a population;

Step 2, training the network weight of the super-network by using the training data and the labels thereof;

step 3, optimizing the subnet code by using a differential evolution algorithm;

and step 4, alternately executing the step 2 and the step 3 until the error rate of the network architecture in the population converges.

2. The method according to claim 1, wherein in step 1, each connection edge in the super network contains N candidate operations, where N is a natural number; m corresponds to the internal nodes arranged in the super network, and if the number of the internal nodes is H, the internal nodes are set；

The super network comprises X sub-networks, and the value of X isThe sub-net table shows a network with only one candidate operation on all connected edges.

3. The method of claim 2, wherein in step 2, the value of each element in the matrix is equal in importance in order to ensure that the sum of the values of all candidate operations on one connection edge is 1 during the initialization of the super-networkIn the subsequent optimization process, after generating the subnet code, assigning the code of the better individual in the subnet to the super-network to guide the super-network to optimize the network weight, and specifically comprises the following steps: when the super network is optimized, the characteristic diagram information on different connection edges is spliced on the channel dimension when converging to the same node, and the characteristic diagram information is expressed as/>Wherein concat represents a stitching function which maps/>, for different features of the inputAnd/>The splicing in the channel dimension is realized, and the characteristic diagrams on each connecting edge are formed by mixing characteristic diagrams obtained by processing input data through candidate operations, and are expressed as/>Wherein/>Representing architecture weight corresponding to the ith operation, input representing input data,/>The i-th operation is represented as a process of extracting the features of the input information.

4. A method according to claim 3, wherein step 3 comprises:

step 3.1, initializing population individuals, including subnet codes and expansion factors;

Step 3.2, updating the expansion factor by using an evolution operator;

Step 3.3, updating the subnet code by using an evolution operator containing the self-adaptive factor;

and 3.4, selecting an execution environment to determine a new generation of subnet coding and a scaling factor.

5. The method of claim 4, wherein step 3.1 comprises:

Step 3.1.1, defining the coding mode of the j-th connecting edge of the nth individual in the g-th generation group as follows according to the definition of N candidate operations of each connecting edge ，/>Representing a code in the form of a concatenated edge vector,A value representing an nth position within the vector code;

Step 3.1.2, according to the setting of M joint edges shared by a super network, defining the coding mode of the nth individual in the g generation group as Wherein/>Representing the result obtained by transposed processing the code of the M-th connecting edge in step 3.1.1,/>Representing the code in the form of an nth individual matrix within the g-th generation population;

step 3.1.3, implementing random initialization of each part in the sub-network unit according to the upper limit and the lower limit of the continuous values ，/>Representing that the random number between 0 and 1 is takenRepresenting the minimum value taken at the nth position on the connecting edge of the nth individual within the g generation population,/>Representing the maximum value obtained at the nth position on the connecting edge of the nth individual in the g generation population;

step 3.1.4, the expansion factors of the nth individuals in the g generation population are calculated As a parameter for acquiring information from the vector code difference, a random number between 0 and 1 is initially set: /(I)。

6. The method of claim 5, wherein step 3.2 comprises:

Step 3.2.1, traversing individuals in the population, randomly selecting non-repeating individuals from the g generation population each time, respectively ，/>And/>The scale factor codes of 3 non-repeating individuals were/>, respectively、/>AndThe subnet codes of 3 non-repeating individuals were/>, respectively、/>And/>; For evolution operators, wherein the codes of one individual are used as basis vectors, and the other two calculate the difference between the codes;

Step 3.2.2, selecting 2 individuals from 3 randomly selected individuals to calculate vector information difference by using the scale factor codes ；

Step 3.2.3, multiplying the calculated vector code difference by the expansion factor of the nth generation individual, expressed asWherein/>And the mutated expansion factor in the traversal process of the g generation population is shown.

7. The method of claim 6, wherein step 3.3 comprises:

Step 3.3.1, selecting corresponding subnet codes from 3 individuals selected randomly in step 3.2.1 to calculate vector information difference ；

Step 3.3.2, adding with the remaining one random subnet code not participating in the calculation of the vector information differenceThen, a mutated subnet code is obtained, expressed as/>Wherein/>Representing a new subnet code obtained by a mutation operator in the traversal process of the g generation population;

step 3.3.3, intersecting the current traversing individual and the variant individual obtained in step 3.3.2 to obtain the coding information of the j-th part 。

8. The method of claim 7, wherein step 3.4 comprises:

Step 3.4.1, discretizing according to the value of the subnet code: only the maximum code value obtained after optimization is reserved on each connecting edge, the discretized coding mode is converted into 0-1 code, namely the corresponding code of the operation reserved on each connecting edge is 1, the rest is 0, and the formula is expressed as ；

Step 3.4.2, calculating the adaptability of the nth individual in the current population to the nth generation populationAnd the fitness/>, of the nth individual obtained after processing via evolution operators: The relation between the prediction result and the real label in the classification model is measured by using an confusion matrix C, an image classification task containing L categories is set, the rows of the confusion matrix represent the prediction category of the model, and the real categories of the image are listed; setting the total number of samples to/>The number of correctly classified samples is/>The calculation formula is as follows:

，

Wherein, Element values representing the kth column of the kth row in the confusion matrix C;

，

Wherein, The element value of the kth row and the kth column in the confusion matrix is represented, and k and l both represent index serial numbers; the calculation formula of the accuracy, namely the Fitness Fitness is as follows:

；

Step 3.4.3 comparing nth individuals within the g generation population Corresponding mutated individuals/>The higher ones are saved into the next generation population:

，

Wherein, Representing individuals saved to the next generation, which comprises a subnet code and a scaling factor, if the individual fitness function obtained by an evolution operator is higher, the nth individual/>, in the population obtained by evolutionThe subnet code and the expansion factor of the population are all preserved, otherwise, the nth individual/>, in the original g generation population is preservedA subnet coding and scaling factor of (a); representing the nth individual/>, within the evolving population Intra subnet coding/>Is used for the degree of adaptation of the system,Representing the nth individual/>, in the original g-th generation populationIntra subnet coding/>Is used for the adaptation degree of the device.

9. The method of claim 8, wherein in step 4, a manner of alternately optimizing network weights and subnet codes is adopted until the intra-population network architecture error rate converges, expressed as:

obeys to/> ，

Wherein the method comprises the steps ofRepresenting subnet encodings,/>Subnet code corresponding to individual representing highest fitness,/>Representing fitness calculation function, W representing weights within the super-network,/>Representing the value of weight W when the loss function value is the lowest,/>Indicating/>, when the value in the return bracket is maximized，/>W, representing the minimum return of the values in brackets, divides the training dataset into two parts, denoted as/>, respectivelyAnd/>Wherein the dataset/>The data in the data set is used for training the weight of the super-network, and the data set/>The data in (2) is used for evaluating the subnet performance in the process of optimizing the subnet coding, and Loss represents a cross entropy Loss function commonly used in classification tasks.

10. The system of a neural architecture search method based on a differential evolution algorithm according to any one of claims 1 to 9, comprising:

The coding module is used for constructing a super-network comprising M connecting edges and N operations, wherein each sub-network uses continuous numerical values ranging from 0 to 1 to assign a weight to each operation on each connecting edge, and the weights are used as codes to uniquely represent individuals in a population;

the training module is used for training the network weight of the super-network by using the training data and the labels thereof;

and the optimizing module is used for optimizing the subnet codes by using a differential evolution algorithm.