TWI764081B - Framework for combining multiple global descriptors for image retrieval - Google Patents

Abstract

The present invention discloses a framework for combining multiple global descriptors for image retrieval. The framework for image retrieval implemented by a computer system includes: a main module that cascades multiple different global descriptors extracted from a convolutional neural network (CNN) (concatenate) to learn; and an auxiliary module to further learn a specific global descriptor among the multiple above-mentioned global descriptors.

Description

A framework for combining multiple global descriptors for image retrieval

The following description refers to the framework of a deep learning model for image retrieval.

Image descriptors based on convolutional neural networks (CNN) are used as common descriptors in computer vision techniques including classification, object detection, and semantic segmentation. In addition, it is also used in very meaningful research such as image captioning and visual question answering.

Recent studies applying CNN-based image descriptors are applicable to temporal-level image retrieval, which is applicable to existing methods that rely on local descriptor matching, and are validated by spatial verification. )rearrange.

Features that result from pooling (average pooling, max pooling, generalized mean pooling, etc.) after CNN can be used in the field of image retrieval as global descriptor. Also, fully connected layers can be added after convolution layers to use the features presented by the FC layers as global descriptors. In this case, the FC layer is used to reduce the dimensionality, and the FC layer can be omitted when there is no need to reduce the dimensionality.

As an example, Korean Granted Patent No. 10-1917369 (registration date: November 05, 2018) discloses an image retrieval technology using a convolutional neural network.

Typical global descriptors generated by global pooling methods include sum pooling of convolution (SPoC), maximum activation of convolution (MAC), and generalized average pooling ( generalized-mean pooling, GeM). Since each global descriptor has different properties, its performance varies depending on the dataset. For example, SPoC activates larger regions on the image representation, conversely, MAC activates more concentrated regions. Variations of typical global descriptors such as weighted sum pooling, weighted value GeM, and regional MAC (R-MAC) exist for enhanced functionality.

Recent research has focused on ensemble techniques for image retrieval. If there are existing combinatorial techniques that educate multiple learners individually and use a combined model combination to improve performance, in recent years there have been multiple ways to improve retrieval performance by combining individually educated method global descriptors. In other words, currently, in the field of image retrieval, different CNN backbone models and global descriptors are used by ensemble to improve retrieval performance.

However, explicitly training different learners (CNN backbone models or global descriptors) for the purpose of combination not only results in longer training time and increased memory consumption, but also requires specially designed policies or losses to control learners The diversity between them makes the training process cumbersome and difficult.

[Problems to be Solved by Invention] Provides a deep learning model framework that can be used with a single model that learns different global descriptors at once.

Provides a method whereby the equivalent effect of combining can be obtained by applying multiple global descriptors without explicitly training multiple learners or controlling for diversity among learners. [Technical means to solve problems]

Provides a framework for image retrieval, implemented by a computer system, including: a main module for a plurality of mutually different global descriptors (global descriptors) extracted from a convolutional neural network (convolution neural network, CNN). ) for concatenation to learn; and an auxiliary module for further learning a specific global descriptor among the multiple above-mentioned global descriptors.

According to one embodiment, the main module is a learning module for ranking loss of image representation, and the auxiliary module is a learning module for classification loss of image representation. A learning module to train the above framework for image retrieval in an end-to-end manner and with a final loss that is the sum of the above ranking loss and the above classification loss.

According to yet another embodiment, the above-mentioned CNN acts as a backbone network that provides feature maps of a given image, and no down sampling is performed before the final stage of the above-mentioned backbone network.

According to yet another embodiment, the above-mentioned main module forms a final global descriptor by concatenating a plurality of the above-mentioned global descriptors after normalization, and can pass a ranking loss (ranking loss) to learn the final global descriptor above.

According to yet another embodiment, the above-mentioned main module includes a plurality of branches for outputting each image representation by using a plurality of the above-mentioned global descriptors, the number of which may vary according to the global description to be used.

According to yet another embodiment, the above-mentioned auxiliary module may use a classification loss to learn the above-mentioned specific global descriptor determined based on the learning performance among the plurality of above-mentioned global descriptors.

According to yet another embodiment, when the above-mentioned auxiliary module uses the classification loss for learning, at least one of label smoothing and temperature scaling techniques can be used.

A descriptor learning method is provided, executed on a computer system, wherein the computer system includes at least one processor, and the at least one processor executes a plurality of computer-readable instructions contained in a memory, and the descriptor learning method includes: mainly: A learning step of cascading a plurality of mutually different global descriptors extracted from the CNN to learn using a ranking loss; and an auxiliary learning step of further learning a specific global descriptor among the plurality of above-mentioned global descriptors using a classification loss.

A non-transitory computer-readable recording medium is provided, in which a computer program for executing the above-described descriptor learning method on the above-described computer system is stored. [Compared to the efficacy of the prior art]

According to an embodiment of the present invention, by applying a new framework for combining multiple global descriptors, ie a combination of multiple global descriptors that can be trained in an end-to-end manner of multiple global descriptors, CGD), which can achieve the equivalent effect of composition without using an explicit composition model or diversity control for global descriptors. It is flexible and scalable through global descriptors, CNN backbones, losses, and datasets. Since the method using combined descriptors can use other types of features, it not only has excellent performance relative to a single global descriptor, but also Improves image retrieval performance.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to the framework of deep learning models for image retrieval, and in particular, to techniques for performing multiple global descriptors for image retrieval.

Embodiments incorporating what is specifically disclosed in this specification propose a framework to achieve the equivalent effect of combining by applying multiple global descriptors that can be trained in an end-to-end manner, thereby reducing flexibility, scalability, time reduction, savings Significant advantages are achieved in terms of cost, retrieval performance, etc.

FIG. 1 is a block diagram for explaining an example of the internal structure of a computer system according to an embodiment of the present invention. For example, the descriptor learning system of the embodiment of the present invention can be implemented by the computer system 100 in FIG. 1 . As shown in FIG. 1, the computer system 100, as a structural element for executing the descriptor learning method, may include a processor 110, a memory 120, a persistent storage device 130, a bus 140, an input/output interface 150 and a network interface 160.

As a structural element for learning descriptors, the processor 110 may comprise, or be part of, any apparatus capable of processing a sequence of multiple instructions. The processor 110 may include, for example, a computer processor, a processor in a mobile device or other electronic device and/or a digital processor. The processor 110 may be included in, for example, a server computing device, a server computer, a series of server computers, a server farm, a cloud computer, a content platform, and the like. The processor 110 can be cascaded to the memory 120 through the bus bar 140 .

Memory 120 may include volatile memory, persistent, virtual or other memory for storing information used by or output by computer system 100 . The memory 120 may include, for example, random access memory (RAM) and/or dynamic RAM (DRAM). The memory 120 may be used to store any information such as status information of the computer system 100 . Memory 120 may also be used to store instructions of computer system 100 including instructions for learning descriptors, for example. Computer system 100 may include more than one processor 110 as needed or where appropriate.

The bus bar 140 may include a communication infrastructure that enables interaction between the various components of the computer system 100 . The bus bar 140 may transfer data between components such as the computer system 100 , such as between the processor 110 and the memory 120 . Bus bar 140 may include wireless and/or wired communication media between the various components of computer system 100 and may include parallel, serial, or other topological arrangements.

Persistent storage 130 (eg, as opposed to memory 120 ) may include a number of components such as memory or other persistent storage, such as those used by computer system 100 to store data in memory for extended periods of time. Persistent storage 130 may include non-volatile main memory, such as used by processor 110 within computer system 100 . Persistent storage 130 may include, for example, flash memory, hard disk, optical disk, or other computer-readable medium.

Input/output interface 150 may include a keyboard, mouse, voice command input, display, or interface to other input or output devices. Input for configuration instructions and/or learning descriptors may be received through input/output interface 150 .

Network interface 160 may include an interface to a network such as a local area network or the Internet. The network interface 160 may include wired or wireless cascaded interfaces. Input for configuration instructions and/or learning descriptors may be received through network interface 160 .

Furthermore, in other embodiments, the computer system 100 may also include more structural elements than those shown in FIG. 1 . However, most prior art structural elements need not be explicitly shown. For example, the computer system 100 may be implemented to include at least a part of a plurality of input/output devices cascaded with the above-mentioned input/output interface 150, or may also include devices such as a transceiver, a Global Positioning System (GPS) ) modules, cameras, various sensors, databases and other structural elements.

Embodiments of the present invention relate to a deep learning model framework that can be used with a single model learning different global descriptors at once.

In recent image retrieval research, deep CNN-based global descriptors have more complete features than existing techniques such as Scale Invariant Feature Transform (SIFT). SPoC is sum pooling in the last feature map of CNN. MAC is another powerful descriptor, and R-MAC finally adds in-region MAC descriptors after performing in-region max pooling. GeM uses pooling parameters to generalize max pooling and mean pooling. Other global descriptor methods include weighted sum pooling, weighted value GeM, Multiscale R-MAC, etc.

In some studies, an additional strategy or an attention mechanism has been used to maximize the activation of important features in the feature map, or a batch feature erasure (BFE) strategy has been proposed, which can force the network to optimize other Feature representation of the region. And, a model that simultaneously optimizes the feature representation and has smooth pixels and hard-to-focus regions is also applied. The disadvantage of the above technique is that not only is it possible to increase the network size and training time, but it also requires additional parameters for training.

In other words, recent research on image retrieval work is to combine different models and combine multiple global descriptors, however, training different models for this combination is not only difficult, but also inefficient in terms of time or storage.

In the present embodiment, a new framework is proposed that can obtain an effect equivalent to combining by applying multiple global descriptors within a period that can be trained in an end-to-end manner. The framework of the present invention is flexible and scalable through global descriptors, CNN backbones, losses and datasets. Also, the framework of the present invention only needs to possess a few additional parameters for training, and does not require additional policies or attention mechanisms.

Combining is a well-known technique for improving results by training multiple learners and obtaining combined results from the trained learners, which has been widely used in image retrieval over the past few decades. However, existing combinatorial techniques have the disadvantage of increasing computational cost as model complexity increases and requiring further control in order to compute the diversity among learners.

The framework of the present invention enables the idea of combining techniques to be applied when training in an end-to-end manner without controlling for diversity.

FIG. 2 shows a framework of a combination of multiple global descriptors (CGD) for image retrieval according to an embodiment of the present invention.

The CGD framework 200 of the present invention can be implemented by the computer system 100 described above, and can be included in the processor 110 as a structural element for learning descriptors.

Referring to FIG. 2 , the CGD framework 200 may be composed of a CNN backbone network 201 and a main module 210 and an auxiliary module 220 as two modules.

In this case, the main module 210 performs the function of learning an image representation, formed by the combination of multiple global descriptors for ranking loss. And, the auxiliary module 220 performs the function of fine-tuning the CNN through a classification loss.

The CGD framework 200 can be trained in an end-to-end manner with a final loss that is the sum of the ranking loss from the main module 210 and the classification loss from the auxiliary module 220 . 1. CNN Backbone 201

All CNN models can be used as CNN backbone 201. The CGD framework 200 can use CNN backbones such as BN-Inception, ShuffleNet-v2, ResNet, and other deformable models, for example, as shown in Figure 2, ResNet-50 can be used as the CNN backbone 201.

As an example, the CNN backbone network 201 can utilize a network formed by four stages. In this case, in order to save more information in the final feature map (feature map), it is possible to abandon stage 3 (stage 3) and stage 4 (stage4) downsampling operations to modify the corresponding network. Thereby, a feature map of size 14×14 is provided for an input size of 224×224, thus improving the performance of individual global descriptors. In other words, in order to improve the performance of global descriptors, no downsampling is performed after stage 3 (stage 3) and before the final stage ( stage 4) of ResNet-50 to include more information. 2. Main Module 210: Multiple Global Descriptors

The main module 210 extracts global descriptors from the last feature map of the CNN backbone 201 through multiple feature aggregation methods and normalizes them with the FC layers.

The global descriptors extracted from the main module 210 are concatenated and normalized to form a final global descriptor, in this case the final global descriptor is lost at the instance level by ordering ) are learned. Among them, the ranking loss can be replaced by a loss for metric learning, and typically a triplet loss can be used.

In detail, the main module 210 includes a plurality of branches for outputting each image representation using different global descriptors at the last convolutional layer. As an example, the main module 210 includes sum pooling of convolution (SPoC), maximum activation of convolution (MAC), generalized-mean pooling (GeM), and in The three most typical types of global descriptors used in each branch.

The number of branches included in the main module 210 can be increased or decreased, and the global descriptor to be used can be deformed and combined according to the user's needs.

，其中，C是特徵圖的數量。將

假設是特徵圖

的          H×W啟動集。則網路輸出由2D特徵圖的C通道構成。全局描述符將

用作輸入，作為池化過程的輸出來生成向量

。這種池化方法可以泛化成如數學公式1。 [數學公式1]

When given an image I, the output of the final convolutional layer is a 3D tensor of C×H×W dimensions

, where C is the number of feature maps. Will

Suppose it is a feature map

The H×W launch set. Then the network output consists of the C channel of the 2D feature map. The global descriptor will

Used as input, as output of the pooling process to generate a vector

. This pooling method can be generalized as Mathematical Equation 1. [Mathematical formula 1]

時，將SPoC定義為

，當

時，將SPoC定義為

，在剩餘情況下，將GeM定義為

。在GeM的情況下，可使用通過實驗固定的

參數3，根據實施例，可由用戶手動設置參數

，或者可以學習參數

本身。when

, the SPoC is defined as

,when

, the SPoC is defined as

, in the remaining cases, the GeM is defined as

. In the case of GeM, experimentally fixed

Parameter 3, according to the embodiment, the parameter can be manually set by the user

, or the parameters can be learned

itself.

-歸一化（normalization）層的歸一化來生成第i個分支的輸出特徵向量

。 [數學公式2]

Dimension reduction through the FC layer and through

- Normalization of the normalization layer to generate the output feature vector of the ith branch

. [Mathematical formula 2]

時，

可以是分支數，

可以是FC層的加權值，當

時，全局描述符

可以是SPoC，當

時，可以是MAC，當

時，可以是GeM。when

hour,

can be the number of branches,

can be the weighted value of the FC layer, when

, the global descriptor

can be SPoC, when

, can be a MAC, when

can be GeM.

的最終特徵向量通過級聯多種分支的輸出特徵向量來依次進行

-歸一化。 [數學公式3]

The CGD framework 200 of the present invention is called a composite descriptor

The final eigenvector of

-Normalized. [Mathematical formula 3]

時，

為級聯（concatenation）。when

hour,

For the cascade (concatenation).

Such combined descriptors can be trained in any type of ranking loss, typically using a batch-hard triplet loss as an example.

Combining multiple global descriptors in the CGD framework 200 has two advantages. First, the equivalent effect of the combination can be brought about by only a few additional parameters. As in the previously mentioned studies, the combined effect is obtained, but in order to be able to train it in an end-to-end manner, the CGD framework 200 extracts multiple global descriptors from a single CNN backbone 201. Second, additional properties are automatically provided on the output of each branch without diversity control. While recent studies have proposed losses specifically designed to encourage diversity among learners, the CGD framework 200 does not require losses specifically designed to control diversity among multiple branches.

Through experiments, descriptor combinations can be found by comparing the performance of multiple combinations of global descriptors. However, depending on the output feature dimension of each data, there are cases where there is little difference in performance. For example, if the performance difference between SPoC 1536-dimension and 768-dimension is not significant, the combination of SPoC 768-dimension + GeM 768-dimension (multiple global descriptors) can be used to obtain better performance than SPoC 1536-dimension (single global descriptor). 3. Auxiliary Module 220: Classification Loss

The auxiliary module 220 can utilize the classification loss to learn the image representation output from the first global descriptor of the main module 210 to learn at the categorical level of the embedding. In this case, label smoothing and temperature scaling techniques can be applied in order to improve performance when learning with classification loss.

In other words, the auxiliary module 220 utilizes the auxiliary classification loss to fine-tune the CNN backbone based on the first global descriptor of the main module 210 . The auxiliary module 220 may utilize the classification loss to learn the image representation presented by the first one of the global descriptors included by the main module 210 . This follows a two-step access method that fine-tunes the CNN backbone together with the classification loss to improve the convolutional filters, followed by fine-tuning the network to improve the performance of the global descriptor.

The CGD framework 200 modifies this access so that there is only one step for end-to-end training. Training with an auxiliary classification loss enables image representations with disjunctive properties between ranks, and helps train the network faster and more robustly than using only a ranking loss.

Temperature scaling and label smoothing in softmax cross-entropy loss (softmax loss) help classification loss training, and softmax loss is defined as Mathematical Equation 4. [Mathematical formula 4]

、

、

分別表示批量大小、類數及第i個輸入的ID標籤。

和

分別是可訓練的加權值和偏差（bias）。並且，

為第一分支的全局描述符，其中

為默認值（default value）1的溫度參數。in,

,

,

represent the batch size, the number of classes, and the ID label of the i-th input, respectively.

and

are trainable weights and biases, respectively. and,

is the global descriptor of the first branch, where

The temperature parameter with the default value of 1.

的溫度定標將更大的梯度（gradient）分配給更難的例子，對於類內的緊湊及類之間擴展嵌入很有用。標籤平滑通過加強模型來推定訓練中的標籤丟失的邊際效果，從而改善泛化性。因此，為了防止過度擬合併學習更好的嵌入方法，在輔助分類損失中追加標籤平滑和溫度定標。In Math Equation 4, use the temperature parameter

The temperature scaling of , assigns larger gradients to harder examples, useful for compact within-class and extended embeddings between classes. Label smoothing improves generalization by strengthening the model to infer the marginal effect of label loss in training. Therefore, to prevent overfitting and learn better embedding methods, label smoothing and temperature scaling are appended to the auxiliary classification loss.

The first global descriptor used to compute the classification loss can be determined by the performance of each global descriptor. As an example, after multiple global descriptors to be used for a group are used as a single branch for learning, the best performing global descriptor among them may be used as the first global descriptor for computing the classification loss. For example, if the performance of SPoC, MAC, and GeM is learned separately, and the performance is GeM>SPoC>MAC, the combination of GeM+MAC tends to show better performance than the combination of MAC+GeM. Compute the global descriptor for the classification loss. 4. Frame structure

The CGD framework 200 is extensible based on the number of global descriptor branches, allowing other types of networks based on the structure of the global descriptor. For example, using 3 global descriptors (SPoC, MAC, GeM) and using the original global descriptor alone for the auxiliary classification loss, 12 possible configurations can be formed.

For the convenience of description, SPoC is abbreviated as S, MAC is abbreviated as M, and GeM is abbreviated as G, and the first letter in the symbol represents the first global descriptor used to assist the classification loss. The CGD framework 200 can extract three global descriptors S, M, G from a CNN backbone 201, in this case, the following 12 configurations can be made based on the global descriptors S, M, G: S, M, G, SM , MS, SG, GS, MG, GM, SMG, MSG, GSM. All global descriptors are combined to learn at the ranking loss, and only the first global descriptor is additionally learned at the classification loss. For example, in the case of SMG, only the global descriptor S is additionally learned in the classification loss, and all S, M, and G are combined (SM, MS, SG, GS, MG, GM, SMG, MSG, GSM) while sorting loss learning.

Therefore, unlike existing methods that learn multiple models individually to combine multiple global descriptors, the present invention can achieve the same effect as combining by learning only one model in an end-to-end manner. Existing methods perform diversity control by making losses separately for combination, while this method can achieve the same effect as combination without diversity control. According to the present invention, the final global descriptor can be used for image retrieval, and multiple image representations before concatenation can be used as needed to use smaller dimensions. The method global descriptor can be used according to user needs, and the number of global descriptors can be adjusted to expand and shrink the model.

An example of the above-described CGD framework 200 is as follows.

As a dataset for image retrieval, use the document "C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011." The dataset (CUB200) and the literature "J. Krause, M. Stark, J. DenG, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision WorkshopS, pages 554–561, 2013.” used the dataset (CARS196) to evaluate the CGD framework 200 of the present invention. In the case of CUB200 and CARS196, clipped images with bounding box information are used.

為0.1，柔性最大值損失的溫度

為0.5。所有數據集使用128個批量大小，每個類的CARS196、CUB200使用64個實例，並且僅通過作為默認輸入大小的224×224來調整圖像大小。 1. 構架設計實驗 1）訓練排序和分類損失 [分類損失]All experiments were performed using MXNet on a Tesla P40 GPU with 24GB of memory. In addition, the predetermined weights of mageNet ILSVRC of MXNet GluonCV use BNInception, ShuffleNet-v2, ResNet-50, and SEResNet-50 together. All experiments use an input size of 224 × 224 and an embedding of 1536 dimensions. In the training step, the size of the input image is resized to 252×252, and after arbitrarily cropped to 224×224, it is randomly flipped in the horizontal direction. Use the Adam optimizer with a learning rate of 1e-4, and use stepwise decay when tuning the learning rate. In all experiments, the margin of triplet loss

is 0.1, the temperature at which the maximum flexibility is lost

is 0.5. All datasets use a batch size of 128, CARS196, CUB200 use 64 instances per class, and only resize images by 224×224 as the default input size. 1. Architecture Design Experiment 1) Training Ranking and Classification Loss [Classification Loss]

The CGD framework 200 is trained by using the first global descriptor's classification loss and ranking loss together. The table in Figure 3 compares the results for the case of using the ranking loss (ranking) and the case of using the auxiliary classification loss and the ranking loss (both) at CARS196. In this experiment, label smoothing and temperature scaling were not applied to the classification loss in all cases. This proves that using both losses provides higher performance than when using the ranking loss alone. The classification loss focuses on clustering each class into a closed embedding space at the category level. The ranking loss focuses on collecting samples at the same level and separating the distances between samples at different levels at the instance level. Therefore, if the ranking loss and the auxiliary classification loss are trained together, the optimization of feature embeddings for classification types and subdivisions is improved. [Label smoothing and temperature scaling]

The table in Figure 4 compares the cases where label smoothing and temperature scaling are not used at CARS196 (no trick) (not used), when label smoothing is used (label smoothing), when temperature scaling is used (temperature scaling), and when label smoothing and The results of temperature scaling (both tricks) (both) were compared. This is done on the ResNet-50 backbone using the global descriptor SM, showing that using label smoothing and temperature scaling improves performance over no tricks. In particular, it can be seen that if label smoothing and temperature scaling are applied together, each performance is improved and the best performance is obtained. 2) Multiple global descriptor combinations [Position of the combination]

Since the CGD framework 200 uses multiple global descriptors, in order to select the best framework, experiments were performed on different positions where multiple global descriptors were combined.

Figure 5 shows a first type of architecture for training multiple global descriptors, and Figure 6 shows a second type of architecture for training multiple global descriptors.

As shown in Figure 5, the first type of architecture combines in the inference step after training each global descriptor with a separate ranking loss, and uses the same global descriptor for each branch without using a classification loss.

On the other hand, the second type of architecture shown in Figure 6 educates with a single ranking loss by combining the raw outputs of global descriptors and does not use multiple global descriptors.

-歸一化。In contrast, as shown in FIG. 2, the CGD framework 200 of the present invention combines multiple global descriptors after the FC layer and

-Normalized.

The table of FIG. 7 compares the performance of the CGD framework with framework A of the first type and framework B of the second type as using the global descriptor SM in CUB200. It can be seen that the performance of the CGD framework is the highest.

The second type of framework B contains multiple branch features and a variety of output feature vectors. Contrary to the CGD framework, in the training step, the final embedding of the first type of architecture A is different from the inference step, and the final embedding of the second type of architecture B loses every property of the global descriptor due to the cascaded FC layers. [Combination method]

From a combined approach point of view, concatenation and summation of multiple global descriptors improves model results. Therefore, the CGD framework of the present invention can select the best method by comparing the two combined methods.

The table of Fig. 8 compares the results of the summation method (Sum) and the concatenation method (Concat) as combined methods as using the global descriptor SM in CUB200. The concatenation method of multiple global descriptors (Concat) provides better performance than the summation method (Sum). The sum method (Sum) may lose the properties of each global descriptor due to the intermixing of the initiation of global descriptors (mix), in contrast, the concatenated method (Concat) can preserve the properties of each global descriptor and maintain diversity . 2. Effects of Combining Descriptors (1) Quantitative analysis

The core of the CGD framework of the present invention is the application of multiple global descriptors. Experiments with 12 possible configurations are conducted on each image retrieval dataset where the CGD framework uses temperature scaling for the auxiliary classification loss.

Figure 9 compares the performance of various architectures relative to the CGD framework of CARS196, and Figure 10 compares the performance of various architectures relative to the CGD framework of CUB200. This experiment utilizes a test set that samples 100 instances per class. Due to the uncertainty of the deep learning model, the results are shown more than 10 times by using a boxplot.

9 and 10 , it can be seen that the combined descriptors (SG, GSM, SMG, SM, GM, GS, MS, MSG, MG) exhibit better performance than the single global descriptors (S, M, G). In the case of CUB200, the single global descriptors G and M exhibit relatively high performance, conversely, the best performing configuration is still the combined descriptor MG. The performance can vary according to the properties of the dataset, the features used for the classification loss, the size of the input, and the dimension of the output. The main essence is that if multiple global descriptors are used, the performance can be improved over a single global descriptor.

The table in Figure 11 compares the performance of CARS196's combined descriptors (SG, GSM, SMG, SM, GM, GS, MS, MSG, MG) with individual global descriptors (S, M, G). Individual descriptors represent the output feature vector of each branch. The combined descriptor is the final feature vector of the CGD framework.

Figure 11 shows the performance of individual global descriptors before combining and the computable improvement in performance after combining. All combined descriptors have 1536-dimensional embedding vectors, in contrast, each individual descriptor has 1536-dimensional embedding vectors for SM, MS, SG, GS, MG, GM and 1536-dimensional embedding vectors for SMG, MSG, GS, MG, GS, 512-dimensional embedding vector for GM. Larger embedding vectors generally provide better performance. However, if the performance difference between large and small embedding vectors is not significant, it may be preferable to use multiple small embedding vectors of other global descriptors. For example, the individual descriptor GeM of the 768-dimensional embedding vector SG has similar performance to the single descriptor G of the 1536-dimensional embedding vector, so SG achieves a significant performance improvement by combining different features of SPC and GeM. 3. Flexibility of the CGD Framework

Figure 12 shows that the CGD framework of the present invention can use various ranking losses (batch-hard triplet loss, HAP2S loss, weighted sampling margin loss, etc.). If the performance of a single global descriptor S and multiple global descriptors SM are compared, in all cases, the performance of multiple global descriptors SM is better than that of a single global descriptor S. From this point, the loss can be applied, so it can be seen that its flexible.

In addition to the ranking loss, the CGD framework of the present invention can be applied to various image retrieval datasets at the level of various CNN backbone networks. A CGD framework that applies multiple global descriptors provides higher performance than existing models in most backbones or datasets.

As such, according to an embodiment of the present invention, by applying a new framework for combining multiple global descriptors, ie, CGD composed of multiple global descriptors that can be trained in an end-to-end manner, it is possible to achieve Equivalent effect to composition without using an explicit composition model or diversity control for global descriptors. The CGD framework of the present invention is flexible and scalable through global descriptors, CNN backbones, losses, and datasets. Since the method using combined descriptors can use other types of features, it is not only superior to a single global descriptor. performance, and can improve image retrieval performance.

The above-described apparatus may be implemented as hardware structural elements, software structural elements, and/or a combination of hardware structural elements and software structural elements. For example, the devices and structural elements described in the embodiments may utilize devices such as processors, controllers, arithmetic logic units (ALUs), digital signal processors (digital signal processors), microcomputers, field programmable gate arrays ( Field programmable gate array, FPGA), programmable logic unit (programmable logic unit, PLU), microprocessor or any other device capable of executing and responding to instructions (instruction) and more than one general-purpose computer or special-purpose computer to implement. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Also, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, there are instances where one processing device is described, but one of ordinary skill in the art will appreciate that a processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. Also, other processing configurations such as parallel processors are also possible.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, and may configure the processing device to operate as desired, or to command independently or collectively processing device. Software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for interpretation by processing means or to provide instructions or data to processing equipment. Software can be distributed across networked computer systems so that it can be stored or run in a distributed fashion. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment can be implemented in the form of program instructions executable by various computer units and recorded on a computer readable medium. In this case, the media can continue to store programs that can be run by the computer or temporarily store them to run or download. Also, the medium may be a single or multiple hard-integrated recording units or storage units, which are not limited to media directly concatenated to a computer system, but may be distributed over a network. Examples of the media include: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks optical medium); and media including ROM, RAM, flash memory, etc. to store program instructions. In addition, as another example of the medium, a recording medium or a storage medium managed by an application store that distributes application programs, a site that provides or distributes various software, a server, and the like can be cited.

As described above, although the present invention has been described with reference to the limited embodiment and the accompanying drawings, those skilled in the art to which the present invention pertains can modify and modify the method from this description. For example, even if the illustrated techniques are performed in a different order than the illustrated methods, and/or structural elements of the illustrated systems, structures, devices, circuits, etc. are combined or combined in different ways than the illustrated methods, through other structural elements or equivalents Appropriate results can also be achieved by substitution or substitution of technical solutions.

Therefore, other examples, other embodiments, and technical solutions equivalent to the protection scope of the invention also fall within the protection scope of the invention described later.

100: Computer System 110: Processor 120: memory 130: Persistent Storage Device 140: Busbar 150: Input/Output Interface 160: Web Interface 200: CGD Framework 201: CNN Backbone 210: Main Module 220: Auxiliary module

FIG. 1 is a block diagram for explaining an example of the internal structure of a computer system according to an embodiment of the present invention.

FIG. 2 shows a framework of a combination of multiple global descriptors (CGD) for image retrieval according to an embodiment of the present invention.

FIG. 3 is a table for illustrating the performance of a CGD framework using both classification loss and ranking loss according to an embodiment of the present invention.

4 is a table for illustrating the performance of a CGD framework using label smoothing and temperature scaling according to an embodiment of the present invention.

5-6 illustrate other types of architecture examples for training multiple global descriptors.

FIG. 7 is a table showing comparison results comparing the performance of the CGD framework of the present invention with other types of frameworks.

FIG. 8 is a table for explaining the performance of a CGD framework for combining multiple global descriptors through concatenation according to an embodiment of the present invention.

9 to 12 are graphs and tables for explaining the performance of a structure composed of a combination of a plurality of global descriptors according to an embodiment of the present invention.

200: CGD Framework

201: CNN Backbone

210: Main Module

220: Auxiliary module

Claims (13)

A framework for image retrieval, implemented by a computer system, including: a main module, which is used for a plurality of different global descriptors (global descriptors) extracted from a convolution neural network (convolution neural network, CNN). ) for concatenation (concatenate) to learn; and an auxiliary module for further learning a specific global descriptor in a plurality of the above-mentioned global descriptors; wherein, the above-mentioned main module is used for image representation (image representation) The learning module of the ranking loss, the auxiliary module is a learning module for the classification loss of the above image representation, in an end-to-end manner and using as The final loss of the sum of the above ranking loss and the above classification loss to train the above framework for image retrieval. A framework for image retrieval as claimed in claim 1, wherein the above-mentioned CNN acts as a backbone network providing feature maps of a given image, and no downsampling is performed before the final stage of the above-mentioned backbone network ( down sampling). The framework for image retrieval of claim 1, wherein the main module forms a final global descriptor by cascading after normalizing a plurality of the global descriptors, And learn the above final global descriptor through ranking loss. A framework for image retrieval as claimed in claim 1, wherein said main module includes a plurality of branches for outputting each image representation by using a plurality of said global descriptors, the number of said branches is according to the number to be used changes to the global description. A framework for image retrieval as in claim 1, wherein, The above-mentioned auxiliary module uses the classification loss to learn the above-mentioned specific global descriptor determined based on the learning performance among the plurality of above-mentioned global descriptors. The framework for image retrieval according to claim 5, wherein the auxiliary module uses at least one of label smoothing and temperature scaling techniques when using classification loss for learning. A descriptor learning method, executed on a computer system, wherein the computer system includes at least one processor, and the at least one processor executes a plurality of computer-readable instructions contained in a memory, and the descriptor learning method includes: mainly: A learning step of cascading a plurality of mutually different global descriptors extracted from the CNN to learn using a ranking loss; and an auxiliary learning step of further learning a specific global descriptor among the plurality of above-mentioned global descriptors using a classification loss. The descriptor learning method of claim 7, wherein, in the above-mentioned descriptor learning method, a plurality of the above-mentioned global descriptors are trained in an end-to-end manner and with a final loss that is the sum of the above-mentioned ranking loss and the above-mentioned classification loss. The descriptor learning method of claim 7, wherein the above-mentioned CNN serves as a backbone network that provides feature maps of a given image, and no downsampling is performed before the final stage of the above-mentioned backbone network. The descriptor learning method of claim 7, wherein, in the above-mentioned main learning step, after a plurality of the above-mentioned global descriptors are normalized, they are formed into a final global descriptor by cascading, and the above-mentioned ordering loss to learn the final global descriptor above. The descriptor learning method of claim 7, wherein, In the above-mentioned auxiliary learning step, the above-mentioned specific global descriptor determined based on the learning performance among the plurality of above-mentioned global descriptors is learned by using the above-mentioned classification loss. The descriptor learning method of claim 11, wherein, in the auxiliary learning step, at least one of label smoothing and temperature scaling techniques is used when using the classification loss for learning. A non-transitory computer-readable recording medium storing a computer program for executing the descriptor learning method described in any one of Claims 7 to 12 on the above-mentioned computer system.

Images