CN115100459A - Image classification algorithm based on full attention network structure search - Google Patents

Abstract

The invention discloses an image classification algorithm based on full attention network structure search, which comprises the steps of firstly designing a staged search space, and selecting different self-attention operations at each stage of a network; then, searching by using an automatic supervision searching method, updating the weight parameters and the structural parameters inside the network, and when the automatic supervision searching stage is completed, keeping the structural parameters and using the structural parameters as initial values of the supervision searching stage; and finally, searching by using a supervision searching method, updating the internal weight parameters and the structural parameters of the network, and obtaining the optimal full attention network according to the structural parameters. The searching method provided by the invention can search out a high-performance full attention network structure and simultaneously ensure the required searching efficiency. Experimental results on an image classification task show that the searched full attention network model is superior to the most advanced architecture, and meanwhile, the parameter quantity is greatly reduced.

Description

Image classification algorithm based on full attention network structure search

technical field

The invention relates to the field of image classification in computer vision, and more particularly, to an efficient design of a full attention network structure for image classification tasks based on a full attention network structure search algorithm.

Background technique

In recent years, significant progress has been made in the structural design of attention networks. Self-attention operations have become an important part of neural networks due to their ability to capture long-range feature dependencies and content-based parameter learning mechanisms. Neural networks composed entirely of self-attention operations have achieved good results in various computer vision tasks. Manually designing an excellent full-attention neural network structure is a challenging and complex task that requires a lot of prior knowledge and experience, and consumes a lot of time and resources for experimental verification, which greatly slows down the attention network. development speed.

Network Structure Search (NAS) provides a way to solve the above problems. NAS is an automatic process of neural network structure design. Its goal is to automatically search from a given search space to obtain a network structure with better performance than the manually designed model under certain prior knowledge. The NAS method can not only improve the performance of the model, but also liberate human experts from the tedious task of designing the network structure. In recent years, many important progresses have been made in the related research of NAS. At present, the search algorithms for network structures are mainly divided into three types: methods based on reinforcement learning, methods based on evolutionary algorithms, and differentiable network structure search algorithms based on gradients. They mainly study how to use different search strategies to improve search efficiency and The resulting network performance. Compared with the NAS method based on reinforcement learning and evolutionary algorithm, the gradient-based differentiable network structure search algorithm converges faster, and its optimization objective is more flexible.

Existing NAS methods are not suitable for directly searching full attention networks. First, existing NAS methods usually use a cell-based search space. In the network structure obtained by the search, the cell structures of the shallow and deep layers of the network are the same. This does not apply to self-attention network search, because the effects of self-attention operations at different stages of the network are different. Moreover, existing NAS methods usually use classification tasks as supervision for structure search. Classification tasks require the model to pay more attention to local regions related to label information and do not need to consider content correlations between distant pixels. However, self-attention models focus on capturing long-range content dependencies between pixels to learn rich image representations. Therefore, it is inappropriate to directly use existing NAS methods to search for full-attention network structures.

SUMMARY OF THE INVENTION

In order to solve the problems in the prior art, the present invention provides an image classification algorithm based on full attention network structure search, which solves the problem of inaccuracy of directly using the existing NAS method to search the full attention network in the prior art.

The present invention is achieved through the following technical solutions:

An image classification algorithm based on full attention network structure search, including the following steps:

Design a staged search space in which each stage of the network selects a different self-attention operation;

Search using a self-supervised search method, input images into the network model of the staged search space, update the network internal weight parameters and structural parameters, when the self-supervised search phase is completed, retain the structural parameters and use them as the supervised search phase. initial value;

Use the supervised search method to search, input the image into the network model of the staged search space, update the internal weight parameters and structural parameters of the network, and obtain the optimal full attention network according to the structural parameters.

In the network structure of the staged search space, the first layer is a fixed local self-attention operation, the last two layers are the average pooling layer and the classification layer, and the rest of the middle part is composed of five stages. And there is a fixed pooling operation in the fourth stage, which halves the spatial size of the feature map and doubles the number of channels; each stage has three searchable layers, and for each searchable layer, it needs to be selected from many candidate operations. The operation with the best performance is selected; the candidate operation consists of 7 self-attention operations, including a non-local self-attention operation and 6 local self-attention operations with different hyperparameters. The hyperparameters of the local self-attention operation include Spatial range and number of heads, where 3, 5, and 7 are selected for the spatial range, and 4 and 8 are selected for the number of heads; To sum up, the staged search space contains 15 searchable layers, each of which is performed from 7 candidate operations Selected, the search space contains 715 possible structures.

In the self-supervised search stage, a self-supervised search algorithm based on contextual autoregressive task is designed; the contextual autoregressive task refers to randomly masking multiple regions of the input image and training the network to predict the content information of the missing part. An encoder-decoder structure is used to extract the features of the input image and reconstruct the missing image content. Then, use this task to search the full attention network. Use the full attention network as the feature encoder to extract features from the input image; the network contains two types of learnable parameters: the weight parameter w of the self-attention operation and the structural parameter a corresponding to each candidate operation; the image dataset is divided into are two independent sets, represented by DatasetA and DatasetB respectively; use DatasetA to optimize weight parameters, and use DatasetB to optimize structure parameters; use L1 loss as the loss function, which is defined as follows:

来优化权重参数，然后在DatasetB数据集上通 过梯度下降

来优化结构参数；当自监督搜索阶段完成时，存储 结构参数，并使用它们作为监督搜索阶段的初始值。where M is the number of pixels, p _i is the input pixel, and y _i is the real value; then, the differentiable structure search method is used to alternately optimize the weight parameter w and the structure parameter a; in an iterative way, on the DatasetA data set through gradient descent

to optimize the weight parameters, and then pass gradient descent on the DatasetB dataset

to optimize the structural parameters; when the self-supervised search phase is complete, store the structural parameters and use them as initial values for the supervised search phase.

In the supervised search stage, the differentiable structure search method is used to search on the image classification dataset; the structure parameters obtained in the self-supervised search stage are used as initial values, and the gradient descent method is used to alternately optimize the structure parameter α and the weight parameter w; use cross-entropy Loss as a loss function, which is defined as:

where K is the number of categories, p _k is the predicted probability of belonging to the kth category, and y _k is the category label; when the supervised search process is over, the structure parameters are sorted, and the operation corresponding to the maximum value of the structure parameters is selected to obtain the final system. structure.

The beneficial effects of the present invention are: the search method proposed by the present invention can search for a high-performance full attention network structure, while ensuring the required search efficiency. Experimental results on image classification tasks show that the searched full attention network model outperforms state-of-the-art architectures while greatly reducing the amount of parameters.

Description of drawings

Fig. 1 network structure search flow chart;

Figure 2 is a macroscopic network structure diagram of the search space;

The specific configuration of the candidate operation in Fig. 3; wherein k represents the spatial extent, and h represents the number of heads in the local self-attention operation;

Figure 4. The best full attention network structure obtained by the search;

Figure 5. The performance comparison between the algorithm of the present invention and the existing advanced network on CIFAR-10 and ImageNet; (a) experimental results on the CIFAR data set; (b) experimental results on the ImageNet data set.

Detailed ways

In order to make the technical solutions of the present invention clearer, the present invention is further described below with reference to the accompanying drawings.

The present invention proposes a full-attention network structure search algorithm to efficiently design a full-attention network structure for image classification tasks. We first design a staged search space in which each stage of the network can choose a different self-attention operation. Afterwards, in order to efficiently find the optimal network structure from the search space, a new search strategy is proposed to jointly use self-supervised search and supervised search to search for an efficient full-attention network structure for image classification tasks. The algorithm flow chart can be seen in Figure 1. The specific details of the search algorithm are introduced below:

1. Staged search space

The present invention proposes a staged search space for searching full attention networks. Figure 2 shows the macroscopic network structure of the search space, and the figure shows the number of layers and the dimension of the input at each stage of the network. In the network structure of the search space, the first layer is a fixed local self-attention operation, and the last two layers are the average pooling layer and the classification layer. The rest of the middle part consists of five stages, in the second and fourth stage there is a fixed pooling operation that halves the spatial size of the feature map and doubles the number of channels. Each stage has three searchable layers, and for each searchable layer, the operation with the best performance needs to be selected from many candidate operations. The candidate operation consists of 7 self-attention operations, including one non-local self-attention operation and 6 local self-attention operations with different hyperparameters, and their configurations are listed in Fig. 3. The hyperparameters of the local self-attention operation include the spatial extent and the number of heads, where the spatial extent can be selected from 3, 5 and 7, and the number of heads can be selected from 4 and 8. To sum up, our search space contains 15 searchable layers, each of which can be selected from 7 candidate operations, and the search space contains 7 ¹⁵ possible structures.

2. A search strategy for joint self-supervised search and supervised search

The first step is to prepare the dataset.

In the image classification task, the present invention selects the CIFAR-10 image classification data set and the ImageNet image classification data set to test and compare the algorithm performance. The CIFAR-10 dataset contains images of 10 categories, each with 6,000 images, for a total of 60,000 images, which are divided into 50,000 training images and 10,000 testing images. In the CIFAR dataset, the spatial resolution of each image is 32×32 in size. The ImageNet 2012 dataset contains images of 1000 categories, with 1.28 million training images and 50,000 validation images. For ImageNet images, we crop them to 224×224 size.

The second step is to search using a self-supervised search method.

100 classes were randomly selected from ImageNet's 1000 original classes to construct a training set, and the images were resized to a resolution of 32 × 32. The training set is divided into two equal subsets, one part is used to optimize the network weight parameters and the other part is used to optimize the architecture parameters. In the self-supervised search process, the images are input into the network model of the staged search space, and the network parameters are optimized using a stochastic gradient descent (SGD) optimizer, where the weight parameter of the optimizer is set to 0.0003, and the momentum decay rate is set to 0.9, The initial learning rate of the network is 0.025. Structural parameters are optimized using the Adam optimizer with a learning rate of 3 × 10 ^-5 and a weight decay of 0.001. After 20 iterations, the search ends and the structure parameters are saved.

The third step is to use a supervised search method to search.

Using the structural parameters obtained in the first stage as initialization values, supervised search is performed on the CIFAR-10 dataset. The training images for CIFAR-10 were randomly split into two parts, each containing 25,000 images. One set is used to optimize network weight parameters, and the other set is used to optimize structure parameters. In the supervised search process, the images of the classification dataset are input into the network model of the staged search space, and the weight parameters are optimized using the SGD optimizer with an initial learning rate of 0.025, momentum of 0.9, and weight decay of 0.0003. The weight parameters are optimized using the Adam optimizer with a learning rate of 1×10 ^-4 and a weight decay of 0.001. After 50 iterations, the search ends, and the optimal full attention network is obtained according to the structural parameters. The searched network is shown in Figure 4.

In the fourth step, the network structure is trained and tested on the CIFAR-10 and ImageNet datasets.

First, train the network on CIFAR-10, and use the SGD optimizer to optimize the model. The weight parameter of the optimizer is set to 4×10 ^-4 , the momentum decay rate is set to 0.9, and the initial learning rate is 0.04. According to the cosine decay rule, it gradually decays to 0. After forward conduction and reverse conduction, the parameter weights of the network are updated once, and after 500 iterations, the trained network can be obtained. During the test, the test set images of CIFAR-10 are input into the network model, and the test results are obtained, as shown in Figure 5(a).

Then train the network on ImageNet and use the SGD optimizer to optimize the model, where the weight parameter of the optimizer is set to 3×10 ^-5 , the momentum decay rate is set to 0.9, and the initial learning rate is 0.04, which gradually decays to 0 according to the cosine decay rule. After forward conduction and reverse conduction, the parameter weights of the network are updated once, and after 300 rounds of iterations, the trained network can be obtained. During the test, the test set image of ImageNet is input into the network model, and the test result is obtained, as shown in Figure 5(b).

The present invention compares the searched model with the classification result of the existing advanced model. It can be seen from the experimental results that the algorithm of the present invention greatly improves the accuracy of the full attention network structure obtained by the search in the classification task while ensuring the search efficiency.

Claims (4)

1. an image classification algorithm based on full attention network structure search, is characterized in that, comprises the following steps: Design a staged search space in which each stage of the network selects a different self-attention operation; Search using a self-supervised search method, input images into the network model of the staged search space, update the network internal weight parameters and structural parameters, when the self-supervised search stage is completed, retain the structural parameters and use them as the supervised search stage. initial value; The supervised search method is used to search, the image is input into the network model of the staged search space, the internal weight parameters and structural parameters of the network are updated, and the optimal full attention network is obtained according to the structural parameters. 2. The image classification algorithm based on full attention network structure search according to claim 1, characterized in that, in the network structure of the phased search space, the first layer is a fixed local self-attention operation, and the last two The layers are the average pooling layer and the classification layer, and the rest of the middle part consists of five stages. There is a fixed pooling operation in the second and fourth stages, which halves the spatial size of the feature map and doubles the number of channels. ; There are three searchable layers in each stage. For each searchable layer, the operation with the best performance needs to be selected from many candidate operations; the candidate operation consists of 7 self-attention operations, including a non-local self-attention operation and 6 local self-attention operations with different hyperparameters, the hyperparameters of the local self-attention operation include the spatial range and the number of heads, where 3, 5 and 7 are selected for the spatial range, and 4 and 8 are selected for the number of heads; As mentioned above, the staged search space contains 15 searchable layers, each of which is selected from 7 candidate operations, and the search space contains ⁷¹⁵ possible structures. 3. The image classification algorithm based on full attention network structure search according to claim 1, is characterized in that, in self-supervised search stage, design a kind of self-supervised search algorithm based on context autoregressive task; context autoregressive task refers to Yes, randomly mask multiple regions of the input image and train the network to predict the content information of the missing parts. An encoder-decoder structure is employed to extract the features of the input image and reconstruct the missing image content. Then, use this task to search the full attention network. Use the full attention network as the feature encoder to extract features from the input image; the network contains two types of learnable parameters: the weight parameter w of the self-attention operation and the structural parameter a corresponding to each candidate operation; the image dataset is divided into are two independent sets, represented by DatasetA and DatasetB respectively; use DatasetA to optimize weight parameters, and use DatasetB to optimize structure parameters; use L1 loss as the loss function, which is defined as follows:

where M is the number of pixels, p _i is the input pixel, and y _i is the real value; then, the differentiable structure search method is used to alternately optimize the weight parameter w and the structure parameter a; in an iterative way, on the DatasetA data set through gradient descent

to optimize the weight parameters, and then pass gradient descent on the DatasetB dataset

to optimize the structural parameters; when the self-supervised search phase is complete, store the structural parameters and use them as initial values for the supervised search phase. 4. The image classification algorithm based on full attention network structure search according to claim 1, is characterized in that, in the supervised search stage, use the differentiable structure search method to search on the image classification data set; use the self-supervised search stage to obtain As the initial value, the structure parameter α and the weight parameter w are optimized alternately by the gradient descent method; the cross entropy loss is used as the loss function, which is defined as:

where K is the number of categories, p _k is the predicted probability of belonging to the kth category, and y _k is the category label; when the supervised search process is over, the structure parameters are sorted, and the operation corresponding to the maximum value of the structure parameters is selected to obtain the final system. structure.

Images