WO2021013095A1 - Image classification method and apparatus, and method and apparatus for training image classification model - Google Patents

Abstract

The present application relates to image recognition technology in the field of artificial intelligence. Provided are an image classification method and apparatus, and a method and apparatus for training an image classification model, relating to the field of the field of artificial intelligence, and in particular to the field of computer vision. The image classification method comprises: acquiring an image to be processed; classifying said image according to a preset global class feature to obtain a classification result of said image, wherein the global class feature comprises multiple class features obtained by means of training according to multiple training images in a training set; the multiple class features in the global class feature are used for indicating visual features of all classes in the training set; all classes in the training set are classes to which all training images in the training set belong; and the training set comprises images in a base class and images in a new class. The image classification method in the embodiments of the present application can better classify images.

Description

Image classification method, image classification model training method and device

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on July 24, 2019, the application number is 201910672533.7, and the application name is "Image Classification Method, Image Classification Model Training Method and Device", all of which are approved The reference is incorporated in this application.

Technical field

This application relates to the field of artificial intelligence, and more specifically, to image classification methods, image classification model training methods and devices.

Background technique

Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. Vividly speaking, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data. Generally speaking, computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

The classification of images (or pictures) is the basis of various image processing applications, and computer vision often involves the problem of how to classify the acquired images. With the rapid development of machine learning (or deep learning), machine learning algorithms have been more and more widely used in image classification processing, and at the same time, they have also achieved very good results. However, machine learning algorithms are highly dependent on a large number of well-labeled The training data is very difficult to obtain in many cases.

Therefore, how to better classify images when the training data is insufficient is an urgent problem to be solved.

Summary of the invention

This application provides an image classification method, an image classification model training method and a device thereof, which can better classify images.

In a first aspect, an image classification method is provided, the method comprising: obtaining an image to be processed; and classifying the image to be processed according to preset global category characteristics to obtain a classification result of the image to be processed, wherein, The global category features include multiple category features trained according to multiple training images in the training set, and multiple category features in the global category features are used to indicate visual features of all categories in the training set. All categories in the set are categories to which all training images in the training set belong, and the training set includes images in the base class and images in the new class.

The above-mentioned base class can be understood as a large-scale training image set. The above-mentioned base class usually includes a large number of annotated images used for model training. The images in the base class can be annotated images. Here, annotated images can refer to those that have been annotated with the category of the image. image.

Correspondingly, with respect to the aforementioned base class, the aforementioned new class usually includes a small number of labeled samples, and the images in the new class may also be labeled images. That is to say, in this embodiment of the present application, the new class is a small sample, that is, the new class includes a small number of images that have been labeled with the category.

For example, the base category includes 100 categories, each category includes 1000 training images, the new category includes 5 categories, and each category includes 5 training images.

In this application, the global category features are obtained by training multiple training images in the training set. The global category features include multiple category features that can indicate the visual features corresponding to all categories in the training set. At the same time, because the global The training set used in the category training process includes the images in the base category and the images in the new category, which can prevent the global category features from being overfitted to the images in the base category, so that images in the new category can be identified more accurately.

Optionally, the image classification model in this application may be trained based on a multi-stage training (episodic training) strategy. For example, the model training process can be divided into multiple training stages (training episodes). In each training stage, several categories in the training set can be randomly selected to train the model. Finally, after multiple training stages, the model is completed training.

Specifically, in multiple training stages in the model training process, the global category features can be updated, so that the global category features obtained by training can have better consistency. At the same time, the training set includes the base class The images in the new class and the images in the new class, the effect of the new class (training image in) during the training process can continue to accumulate with the global category features. Therefore, it can avoid the image classification model obtained by training from overfitting to the base class. According to the image classification model to classify the image to be processed, a better image classification result can be obtained.

With reference to the first aspect, in some implementations of the first aspect, the classifying the to-be-processed image according to preset global category features to obtain the classification result of the to-be-processed image includes: extracting the The feature vector of the image to be processed; according to the feature vector of the image to be processed, the confidence that the image to be processed belongs to a candidate category is determined, the candidate category being one or more of the multiple categories indicated by the global category feature A; According to the confidence, the classification result of the image to be processed is determined from the candidate category.

With reference to the first aspect, in some implementations of the first aspect, before the determining the confidence that the image to be processed belongs to the candidate category according to the feature vector of the image to be processed, the method further includes: The support set of the image to be processed determines the local category feature of the image to be processed; the candidate category is determined according to the local category feature of the image to be processed and the global category feature; wherein, the image to be processed The support set of includes multiple images, and the category to which the multiple images belong is one or more of the multiple categories indicated by the global category feature.

With reference to the first aspect, in some implementations of the first aspect, the determining the confidence that the image to be processed belongs to the candidate category according to the feature vector of the image to be processed includes: Feature vector, determining the distance between the feature vector of the image to be processed and the feature vector corresponding to each of the candidate categories; determining the confidence that the image to be processed belongs to the candidate category according to the distance .

With reference to the first aspect, in some implementations of the first aspect, the determining the classification result of the image to be processed from the candidate category according to the confidence level includes: The category with the greatest confidence is determined as the classification result of the image to be processed.

With reference to the first aspect, in some implementations of the first aspect, the global category feature is obtained by training based on a classification error, and the classification error is based on the classification result of the training image in the query set and the query set The training image is determined by a pre-labeled label, the label is used to indicate the category to which the training image belongs, and the query set includes part of the training images in the partial categories in the training set.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. According to the classification error, the global category feature is updated to avoid the overfitting of the global category feature to the image in the base category, so that the image in the new category can be identified more accurately.

Further, in multiple training stages in the model training process, the classification error of the current training stage can be used to update the global category features. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class in the training process can be continuously accumulated with the global category features. Therefore, it can avoid the overfitting of the trained image classification model to the base class. According to the image classification model, the image to be processed can be classified, which can get better The result of image classification.

With reference to the first aspect, in some implementations of the first aspect, the global category feature is obtained by training based on classification error and registration error, and the classification error is based on the classification result and the result of the training image in the query set. The pre-labeled training image in the question set is determined, the label is used to indicate the category to which the training image belongs, the question set includes some training images in some categories in the training set, and the configuration The quasi-error is determined based on the local category feature of the training image and multiple category features in the global category feature. The local category feature of the training image includes multiple category features determined from multiple training images in the support set. The multiple category features in the local category features of the training image are used to indicate the visual features of all categories in the support set, and the support set includes part of the training images in the partial categories in the training set.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. The registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, and the local category feature of the training image includes the determination based on multiple training images in the support set Multiple category features, update the global category features according to the classification error and the registration error, which can prevent the global category features from being over-fitted to the images in the base class, so that the new class can be more accurately identified image.

Further, in multiple training stages in the model training process, the classification error and registration error of the current training stage can be used to update the global category features. At the same time, the training set includes the images in the base category and the new category. The effect of the new class in the training process can be continuously accumulated with the global category features. Therefore, it can prevent the image classification model obtained by training from overfitting to the base class, and classify the image to be processed according to the image classification model. Can get better image classification results.

With reference to the first aspect, in some implementations of the first aspect, the local category feature of the training image is determined by multiple training images in the support set that have undergone expansion processing, and the expansion processing includes performing an image Tailoring, flipping, and/or data transformation.

The multiple training images in the above support set are images in the new class.

In this application, multiple training images in the new class are expanded through cropping, flipping, and/or data illusion processing to increase the number of training images in the new class. Therefore, the training image classification can be avoided The model is overfitted to the base class, and the image to be processed is classified according to the image classification model, which can obtain better image classification results.

In a second aspect, a method for training an image classification model is provided. The method includes: acquiring a plurality of training images in a training set, wherein the training set includes a support set and a training set, and the plurality of training images includes the Support multiple training images in the set and multiple training images in the query set; according to the preset first neural network, extract feature vectors of the multiple training images in the query set, and the query set includes all Part of the images in the partial categories in the training set; according to the preset second neural network and preset global category features, the feature vectors of the multiple training images in the query set are processed to obtain the query set The classification result of the multiple training images of, wherein the global category feature includes multiple category features, and the multiple category features in the global category feature are used to indicate the visual features of all categories in the training set, and the training All categories in the set are the categories to which all the training images in the training set belong. The training set includes images in the base class and images in the new class; according to the classification results of multiple training images in the query set, update The global category characteristics.

The above-mentioned base class can be understood as a large-scale training image set. The above-mentioned base class usually includes a large number of annotated images used for model training. The images in the base class can be annotated images. Here, annotated images can refer to those that have been annotated with the category of the image. image.

Correspondingly, with respect to the aforementioned base class, the aforementioned new class usually includes a small number of labeled samples, and the images in the new class may also be labeled images. That is to say, in this embodiment of the present application, the new class is a small sample, that is, the new class includes a small number of images that have been labeled with the category.

For example, the base category includes 100 categories, each category includes 1000 training images, the new category includes 5 categories, and each category includes 5 training images.

In this application, the global category features are trained from the classification results of the training images in the training set, and the global category features include multiple category features that can indicate the visual features corresponding to all categories in the training set. The training set used in the global category training process includes images in the base category and images in the new category, which can prevent the global category features from being overfitted to the images in the base category, so that images in the new category can be identified more accurately.

Optionally, the training method of the image classification model in this application can train the model based on a multi-stage training (episodic training) strategy. For example, the model training process can be divided into multiple training stages (training episodes). In each training stage, several categories in the training set can be randomly selected to train the model. Finally, after multiple training stages, the model is completed training.

Specifically, in multiple training stages in the model training process, the global category features can be updated, so that the global category features obtained by training can have better consistency. At the same time, the training set includes the base class The images in the new class and the images in the new class, the effect of the new class (training image in) during the training process can continue to accumulate with the global category features. Therefore, it can avoid the image classification model obtained by training from overfitting to the base class. According to the image classification model to classify the image to be processed, a better image classification result can be obtained.

Optionally, feature vectors of multiple training images in the support set and feature vectors of multiple training images in the query set may be extracted.

With reference to the second aspect, in some implementations of the second aspect, the feature vectors of multiple training images in the query set are processed according to a preset second neural network and preset global category features , Obtaining classification results of the multiple training images in the query set includes: extracting feature vectors of the multiple training images in the support set, the support set including part of the training images in the partial categories in the training set; Determine the local category features of the training images according to the feature vectors of the multiple training images in the support set, where multiple category features in the local category features of the training images are used to indicate visual features of all categories in the support set, The support set includes part of the training images in the partial categories in the training set; according to the second neural network, the local category feature of the training image, and the global category feature, determine a plurality of the query set The classification result of the training image.

With reference to the second aspect, in some implementations of the second aspect, the updating the global category feature according to the classification results of the multiple training images in the query set includes: Update the global category feature, the first neural network and the second neural network for the classification results of the training images.

With reference to the second aspect, in some implementations of the second aspect, the updating the global category feature according to the classification results of the multiple training images in the query set includes: updating the global The category feature, the classification error is determined based on the classification results of the multiple training images in the query set and the pre-labeled tags of the multiple training images in the query set, and the tags are used to indicate the training images The category it belongs to.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. According to the classification error, the global category feature is updated to avoid the overfitting of the global category feature to the image in the base category, so that the image in the new category can be identified more accurately.

Further, in multiple training stages in the model training process, the classification error of the current training stage can be used to update the global category features. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class in the training process can be continuously accumulated with the global category features. Therefore, it can avoid the overfitting of the trained image classification model to the base class. According to the image classification model, the image to be processed can be classified, which can get better The result of image classification.

With reference to the second aspect, in some implementations of the second aspect, the updating the global category feature according to the classification results of the multiple training images includes: updating all the features according to the classification error and the registration error The global category feature, wherein the registration error is determined according to a local category feature of the training image and multiple category features in the global category feature.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. The registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, and the local category feature of the training image includes the determination based on multiple training images in the support set Multiple category features, update the global category features according to the classification error and the registration error, which can prevent the global category features from being over-fitted to the images in the base class, so that the new class can be more accurately identified image.

Further, in multiple training stages in the model training process, the classification error and registration error of the current training stage can be used to update the global category features. At the same time, the training set includes the images in the base category and the new category. The effect of the new class in the training process can be continuously accumulated with the global category features. Therefore, it can prevent the image classification model obtained by training from overfitting to the base class, and classify the image to be processed according to the image classification model. Can get better image classification results.

With reference to the second aspect, in some implementations of the second aspect, the local category feature of the training image is determined by a plurality of training images in the support set after expansion processing, and the expansion processing includes performing an image Tailoring, flipping, and/or data transformation.

The multiple training images in the above support set are images in the new class.

In this application, multiple training images in the new class are expanded through cropping, flipping, and/or data illusion processing to increase the number of training images in the new class. Therefore, the training image classification can be avoided The model is overfitted to the base class, and the image to be processed is classified according to the image classification model, which can obtain better image classification results.

In a third aspect, an image classification device is provided, including: an acquisition module for acquiring an image to be processed; a classification module for classifying the image to be processed according to preset global category features to obtain the image to be processed Process the classification result of the image, wherein the global category feature includes multiple category features trained according to multiple training images in the training set, and the multiple category features in the global category feature are used to indicate all the categories in the training set. Visual features of the category, all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in the base category and images in the new category.

In the embodiment of the present application, the global category feature in the image classification device is obtained by training multiple training images in the training set, and the global category feature includes multiple category features that can indicate visual features corresponding to all categories in the training set. At the same time, since the training set used in the global category training process includes the images in the base category and the images in the new category, it can prevent the global category features from being over-fitted to the images in the base category, thereby enabling more accurate recognition Images in the new category.

Specifically, in multiple training stages in the model training process, the global category features in the image classification device can be updated, so that the global category features obtained by training can have better consistency. At the same time, training The collection includes the images in the base class and the images in the new class. The effect of the new class (training image in) during the training process can be accumulated with the global category features, so it can avoid the training of the image classification device from over-fitting Combine it into the base class and classify the image to be processed according to the image classification device, which can obtain better image classification results.

With reference to the third aspect, in some implementations of the third aspect, the classification module is specifically configured to: extract a feature vector of the image to be processed; and determine the image to be processed according to the feature vector of the image to be processed Confidence that belongs to a candidate category, where the candidate category is one or more of the multiple categories indicated by the global category feature; according to the confidence, the category of the image to be processed is determined from the candidate category result.

With reference to the third aspect, in some implementations of the third aspect, the device further includes a determining module, configured to: determine the local category characteristics of the image to be processed according to the support set of the image to be processed; The local category feature of the image to be processed and the global category feature to determine the candidate category; wherein the support set of the image to be processed includes multiple images, and the category to which the multiple images belong is the global category feature One or more of the indicated categories.

With reference to the third aspect, in some implementations of the third aspect, the classification module is specifically configured to: determine the feature vector of the image to be processed and the candidate category according to the feature vector of the image to be processed The distance of the feature vector corresponding to each category; according to the distance, the confidence that the image to be processed belongs to the candidate category is determined.

With reference to the third aspect, in some implementation manners of the third aspect, the classification module is specifically configured to: determine the category with the greatest confidence among the candidate categories as the classification result of the image to be processed.

With reference to the third aspect, in some implementations of the third aspect, the global category feature is obtained by training based on a classification error, and the classification error is based on the classification result of the training image in the query set and the query set The training image is determined by a pre-labeled label, the label is used to indicate the category to which the training image belongs, and the query set includes part of the training images in the partial categories in the training set.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. According to the classification error, the global category feature is updated to avoid the overfitting of the global category feature to the image in the base category, so that the image in the new category can be identified more accurately.

With reference to the third aspect, in some implementations of the third aspect, the global category feature is obtained by training based on classification errors and registration errors, and the classification error is based on the classification results and the results of the training images in the query set. The pre-labeled training image in the question set is determined, the label is used to indicate the category to which the training image belongs, the question set includes some training images in some categories in the training set, and the configuration The quasi-error is determined based on the local category feature of the training image and multiple category features in the global category feature. The local category feature of the training image includes multiple category features determined from multiple training images in the support set. The multiple category features in the local category features of the training image are used to indicate the visual features of all categories in the support set, and the support set includes part of the training images in the partial categories in the training set.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. The registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, and the local category feature of the training image includes the determination based on multiple training images in the support set Multiple category features, update the global category features according to the classification error and the registration error, which can prevent the global category features from being over-fitted to the images in the base class, so that the new class can be more accurately identified image.

With reference to the third aspect, in some implementations of the third aspect, the local category feature of the training image is determined by multiple training images in the support set that have undergone expansion processing, and the expansion processing includes performing an image Tailoring, flipping, and/or data transformation.

The multiple training images in the above support set are images in the new class.

In this application, multiple training images in the new class are expanded through cropping, flipping, and/or data illusion processing to increase the number of training images in the new class. Therefore, the training image classification can be avoided The device is overfitted to the base class, and the image to be processed is classified according to the image classification device, which can obtain better image classification results.

In a fourth aspect, a training device for an image classification model is provided, which includes: an acquisition module for acquiring multiple training images in a training set, wherein the training set includes a support set and a training set, and the multiple training images Including multiple training images in the support set and multiple training images in the query set; a feature extraction module for extracting features of multiple training images in the query set according to a preset first neural network Vector, the question set includes part of images in the partial categories in the training set; the classification module is used to compare the plurality of images in the question set according to the preset second neural network and the preset global category features The feature vectors of the training images are processed to obtain classification results of multiple training images in the query set, wherein the global category features include multiple category features, and the multiple category features in the global category features are used to indicate Visual features of all categories in the training set, all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in the base class and images in the new class; update module , Used to update the global category feature according to the classification results of the multiple training images in the query set.

In this application, the global category features are trained from the classification results of multiple training images in the training set. The global category features include multiple category features that can indicate the visual features corresponding to all categories in the training set. At the same time, because The training set used in the global category training process includes images in the base category and images in the new category, which can prevent the global category features from being overfitted to the images in the base category, thereby enabling more accurate identification of the images in the new category. image.

Specifically, in multiple training stages in the model training process, the global category features can be updated, so that the global category features obtained by training can have better consistency. At the same time, the training set includes the base class The images in the new class and the images in the new class, the effect of the new class (training image in) during the training process can continue to accumulate with the global category features. Therefore, it can avoid the image classification model obtained by training from overfitting to the base class. According to the image classification model to classify the image to be processed, a better image classification result can be obtained.

With reference to the fourth aspect, in some implementations of the fourth aspect, the classification module is specifically configured to: extract feature vectors of multiple training images in the support set, and the support set includes partial categories in the training set Part of the training images in the support set; determine the local category features of the training images according to the feature vectors of the multiple training images in the support set, and the multiple category features in the local category features of the training images are used to indicate the support set Visual features of all categories, the support set includes part of the training images in the partial categories in the training set; according to the second neural network, the local category features of the training images, and the global category features, the Ask the classification results of multiple training images in the set.

With reference to the fourth aspect, in some implementation manners of the fourth aspect, the update module is specifically configured to: update the global category feature, the first A neural network and the second neural network.

With reference to the fourth aspect, in some implementations of the fourth aspect, the update module is specifically configured to: update the global category feature according to a classification error, and the classification error is based on multiple trainings in the query set The classification result of the image and the pre-labeled labels of the multiple training images in the query set are determined, and the labels are used to indicate the category to which the training image belongs.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. According to the classification error, the global category feature is updated to avoid the overfitting of the global category feature to the image in the base category, so that the image in the new category can be identified more accurately.

With reference to the fourth aspect, in some implementations of the fourth aspect, the update module is specifically configured to: update the global category feature according to the classification error and the registration error, wherein the registration error is based on The local category feature of the training image and the multiple category features of the global category feature are determined.

In this application, the classification error is determined based on the classification results of the training images in the query set and the pre-labeled labels of the training images. The training images in the query set include images in the base class and images in the new class. The registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, and the local category feature of the training image includes the determination based on multiple training images in the support set Multiple category features, update the global category features according to the classification error and the registration error, which can prevent the global category features from being over-fitted to the images in the base class, so that the new class can be more accurately identified image.

With reference to the fourth aspect, in some implementations of the fourth aspect, the local category feature of the training image is determined by multiple training images in the support set that have undergone expansion processing, and the expansion processing includes performing an image Tailoring, flipping, and/or data transformation.

The multiple training images in the above support set are images in the new class.

In this application, multiple training images in the new class are expanded through cropping, flipping, and/or data illusion processing to increase the number of training images in the new class. Therefore, the training image classification can be avoided The model is overfitted to the base class, and the image to be processed is classified according to the image classification model, which can obtain better image classification results.

In a fifth aspect, an image classification device is provided. The device includes: a memory for storing a program; a processor for executing the program stored in the memory. When the program stored in the memory is executed, the processing The device is used to execute the method in any one of the foregoing first aspects.

In a sixth aspect, an image classification model training device is provided. The device includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, The processor is configured to execute the method in any one of the foregoing second aspect.

The processors in the fifth and sixth aspects described above can be either a central processing unit (CPU), or a combination of a CPU and a neural network computing processor, where the neural network computing processor can include graphics processing GPU (graphics processing unit, GPU), neural-network processing unit (NPU), tensor processing unit (TPU), etc. Among them, TPU is an artificial intelligence accelerator application specific integrated circuit fully customized by Google for machine learning.

In a seventh aspect, a computer-readable medium is provided, and the computer-readable medium stores program code for device execution. The program code includes a method for executing any one of the first aspect or the second aspect. .

In an eighth aspect, a computer program product containing instructions is provided, when the computer program product runs on a computer, the computer executes the method in any one of the foregoing first aspect or second aspect.

In a ninth aspect, a chip is provided, the chip includes a processor and a data interface, the processor reads instructions stored in a memory through the data interface, and executes any one of the first aspect or the second aspect above The method in the implementation mode.

Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the The processor is configured to execute the method in any one of the implementation manners of the first aspect or the second aspect.

The aforementioned chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

In a tenth aspect, an electronic device is provided, the electronic device includes the image classification device in any one of the foregoing third aspects, or the electronic device includes the image classification model in any one of the foregoing fourth aspects Training device.

When the above electronic device includes the image classification apparatus in any one of the above third aspects, the electronic device may specifically be a terminal device.

When the above electronic device includes the image classification model training device in any one of the above fourth aspects, the electronic device may specifically be a server.

In this application, in multiple training stages in the model training process, the global category features can be updated, so that the global category features obtained by training can have better consistency. At the same time, the training set includes the base The images in the class and the images in the new class. The effect of the new class (training image in) during the training process can be accumulated with the global category features. Therefore, it can avoid the image classification model obtained by training from overfitting to the base Class, according to the image classification model to classify the image to be processed, can more accurately identify the image in the new class.

Description of the drawings

Fig. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.

Fig. 2 is a schematic block diagram of a convolutional neural network model provided by an embodiment of the present application.

Fig. 3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.

Fig. 4 is a schematic diagram of an application scenario provided by an embodiment of the present application.

Fig. 5 is a schematic flowchart of a method for training an image classification model provided by an embodiment of the present application.

Fig. 6 is a schematic block diagram of a method for training an image classification model provided by an embodiment of the present application.

FIG. 7 is a schematic flowchart of a method for training an image classification model provided by another embodiment of the present application.

FIG. 8 is a schematic flowchart of an image classification method provided by an embodiment of the present application.

FIG. 9 is a schematic diagram of the hardware structure of an image classification device according to an embodiment of the present application.

FIG. 10 is a schematic diagram of the hardware structure of an image classification model training device according to an embodiment of the present application.

Detailed ways

The technical solution in this application will be described below in conjunction with the drawings.

The image classification method provided by the embodiments of the present application can be applied to image retrieval, album management, safe city, human-computer interaction, and other scenes that require image classification or image recognition. It should be understood that the images in the embodiments of this application may be static images (or called static pictures) or dynamic images (or called dynamic pictures). For example, the images in this application may be videos or dynamic pictures, or The images in can also be static pictures or photos. For ease of description, in the following embodiments of the present application, static images or dynamic images are collectively referred to as images.

Specifically, the image classification method of the embodiment of the present application can be specifically applied to album classification and photo recognition scenes, and these two scenes are described in detail below.

Album category:

Users store a large number of pictures on mobile phones and cloud disks, and sorting and managing albums according to categories can improve user experience. Using the image classification method of the embodiment of the present application to classify pictures in an album, it is possible to obtain albums arranged or stored according to categories. The image classification method of the embodiment of the present application can facilitate the user to classify and manage different object categories, thereby facilitating the user's search, saving the user's management time, and improving the efficiency of album management.

Specifically, when the image classification method of the embodiment of the present application is used to classify albums, the picture features of the pictures in the album can be extracted first, and then the pictures in the album are classified according to the extracted picture characteristics to obtain the classification result of the pictures. Next, the pictures in the album are classified according to the classification results of the pictures, and the albums arranged according to the picture categories are obtained. Among them, when arranging pictures in the album according to picture categories, pictures belonging to the same category may be arranged in a row or a row. For example, in the final album, the pictures in the first row belong to airplanes, and the pictures in the second row belong to cars.

Take photos to identify things:

When taking a photo, the user can use the image classification method of the embodiment of the present application to process the captured photo, and can automatically recognize the category of the object being photographed, for example, it can automatically recognize that the object being photographed is a flower, an animal, etc. Further, the image classification method of the embodiment of the application can be used to identify the object obtained by taking a photo, and the category to which the object belongs can be identified. For example, the photo obtained by the user includes a shared bicycle, and the image classification method of the embodiment of the application is used The shared bicycle can be recognized, and the object is recognized as a bicycle, and further, bicycle related information can be displayed.

Since the present invention has an excellent learning effect on generalized small samples, it can be well recognized regardless of whether the photographed object comes from the base class or the new class.

In the prior art, for image categories with few training samples, it is often impossible to effectively perform image classification or image recognition on images belonging to the category. Using the image classification method of the embodiment of the present application, for a new class (small sample) with few training samples, image classification or image recognition can also be implemented well.

It should be understood that the album classification and photo identification described above are only two specific scenarios applied by the image classification method in the embodiment of this application, and the image classification method in the embodiment of this application is not limited to the above two scenarios when applied. The image classification method of the application embodiment can be applied to any scene that requires image classification or image recognition.

The embodiments of this application involve a large number of related applications of neural networks. In order to better understand the solutions of the embodiments of this application, the following first introduces related terms and other related concepts of neural networks that may be involved in the embodiments of this application.

(1) Neural network

A neural network can be composed of neural units, which can refer to an arithmetic unit that takes x _s and intercept 1 as inputs. The output of the arithmetic unit can be as shown in formula (1-1):

Among them, s=1, 2,...n, n is a natural number greater than 1, W _s is the weight of x _s , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be a region composed of several neural units.

(2) Deep neural network

Deep neural network (DNN), also called multi-layer neural network, can be understood as a neural network with multiple hidden layers. DNN is divided according to the positions of different layers. The neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the number of layers in the middle are all hidden layers. The layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.

其中，

是输入向量，

是输出向量，

是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量

经过如此简单的操作得到输出向量

由于DNN层数多，系数W和偏移向量

的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为

上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。 Although DNN looks complicated, it is not complicated in terms of the work of each layer. In simple terms, it is the following linear relationship expression:

among them,

Is the input vector,

Is the output vector,

Is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just the input vector

After such a simple operation, the output vector is obtained

Due to the large number of DNN layers, the coefficient W and the offset vector

The number is also relatively large. The definition of these parameters in the DNN is as follows: Take the coefficient W as an example: Suppose that in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as

The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

It should be noted that the input layer has no W parameter. In deep neural networks, more hidden layers make the network more capable of portraying complex situations in the real world. Theoretically speaking, a model with more parameters is more complex and has a greater "capacity", which means it can complete more complex learning tasks. Training a deep neural network is also a process of learning a weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W of many layers).

(3) Convolutional neural network

Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with convolutional structure. The convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer, which can be regarded as a filter. The convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can be connected to only part of the neighboring neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Sharing weight can be understood as the way to extract image information has nothing to do with location. The convolution kernel can be initialized in the form of a matrix of random size. During the training of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

(4) Loss function

In the process of training a deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value that you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then based on the difference between the two To update the weight vector of each layer of neural network (of course, there is usually an initialization process before the first update, that is, pre-configured parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make its prediction lower, and keep adjusting until the deep neural network can predict the really wanted target value or a value very close to the really wanted target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value". This is the loss function or objective function, which is used to measure the difference between the predicted value and the target value. Important equation. Among them, take the loss function as an example. The higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing this loss as much as possible.

(5) Back propagation algorithm

The neural network can use an error back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged. The backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal neural network model parameters, such as the weight matrix.

(6) Pixel value

The pixel value of the image can be a red-green-blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For grayscale images, the pixel values can be grayscale values.

(7) Base class

In the prior art, the base class includes a large number of labeled samples used to train the model, and the number of these labeled samples is sufficient to meet the requirements of model training. For example, the base category may include multiple images that have been labeled with categories, the multiple images may belong to one category, or the multiple images may also belong to multiple different categories. The base class can be used to train the image classification model in the embodiment of the present application.

(8) New class

In the prior art, a new class (novel class) is a concept opposite to the base class. For example, if a model is trained using multiple labeled samples, then for the (trained) model, the multiple The labeled sample is the base category, and the categories that are not included in the base category are the new categories. For example, we have trained a model through images of a large number of animals (except dogs). At this time, if we want the model to recognize dogs, the images of the large number of animals are the base class, and the images of the dog are the new class.

Usually, each category in the new category includes only a small number of labeled samples. In this application, the new category can refer to a small sample

(few-shot), that is, the new category includes a small number of images that have been labeled with their categories. These images can belong to one category, or these images can also be divided into multiple different categories.

(9) Small sample learning

Small sample learning (few-shot learning, FSL) refers to the use of large-scale training sets (including one or more base classes), after training the image classification model, for new classes that have never been seen before (new classes and base classes). Non-overlapping), with the help of a few training samples included in each new class, accurately identify the test samples of the new class (category to which it belongs).

Further, small sample learning may include standard small sample learning and generalized small sample learning. For example, if the test sample in small sample learning includes only new classes, then this type of problem can be called standard small sample learning; if the test sample includes not only new classes but also base classes, then this type of problem can be called Learning for generalized small samples.

The image classification method in the embodiment of the present application can be applied to standard small sample learning, and can also be applied to generalized small sample learning. The following describes the system architecture applicable to the embodiment of the present application with reference to FIG. 1.

As shown in FIG. 1, an embodiment of the present application provides a system architecture 100. In FIG. 1, a data collection device 160 is used to collect training data. For the image classification method of the embodiment of the present application, the training data may include training images and classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-labeled results.

After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 trains to obtain the target model/rule 101 based on the training data maintained in the database 130.

The following describes the target model/rule 101 obtained by the training device 120 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the output image of the training device 120 differs from the original image. The difference is less than a certain threshold, thereby completing the training of the target model/rule 101.

The above-mentioned target model/rule 101 can be used to implement the image classification method of the embodiment of the present application, that is, the image to be processed is input into the target model/rule 101 after relevant preprocessing to obtain the classification result of the image. The target model/rule 101 in the embodiment of the application may specifically be the image classification model in the embodiment of the application. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices. In addition, it should be noted that the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of Examples.

The target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 1. The execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, Notebook computers, augmented reality (AR)/virtual reality (VR), vehicle-mounted terminals, etc., can also be servers or cloud devices. In FIG. 1, the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices. The user can input data to the I/O interface 112 through the client device 140. The input data in this embodiment of the application may include: the image to be processed input by the client device.

The preprocessing module 113 and the preprocessing module 114 are used for preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112. In the embodiment of the present application, the preprocessing module 113 and the preprocessing module may not be provided. 114 (there may only be one preprocessing module), and the calculation module 111 is directly used to process the input data.

When the execution device 110 preprocesses input data, or when the calculation module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing , The data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.

Finally, the I/O interface 112 returns the processing result, such as the classification result of the to-be-processed image obtained as described above, to the client device 140 to provide it to the user.

It is worth noting that the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide the user with the desired result.

In the case shown in FIG. 1, the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112. In another case, the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140. The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action. The client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure. Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure. The data is stored in the database 130.

It is worth noting that Fig. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in Fig. 1, the data The storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

As shown in FIG. 1, the target model/rule 101 is obtained by training according to the training device 120. The target model/rule 101 may be the image classification model in the embodiment of the application. Specifically, the image provided in the embodiment of the application The classification model may include one or more neural networks. The one or more neural networks may include CNN, deep convolutional neural networks (DCNN), and/or recurrent neural networks (RNNS), etc. .

Since CNN is a very common neural network, the structure of CNN will be introduced in detail below in conjunction with Figure 2. As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to a machine learning algorithm. Multi-level learning is carried out on the abstract level of As a deep learning architecture, CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.

As shown in FIG. 2, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (the pooling layer is optional), and a neural network layer 230. The following is a detailed introduction to the relevant content of these layers.

Convolutional layer/pooling layer 220:

Convolutional layer:

The convolutional layer/pooling layer 220 as shown in FIG. 2 may include layers 221-226 as shown in Examples. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, and layer 223 is Convolutional layer, 224 is a pooling layer, 225 is a convolutional layer, and 226 is a pooling layer; in another implementation, 221 and 222 are convolutional layers, 223 is a pooling layer, and 224 and 225 are convolutions. The accumulation layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

The convolution layer 221 can include many convolution operators. The convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same. During the convolution operation, the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row × column) are applied. That is, multiple homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image. Perform fuzzification, etc. The multiple weight matrices have the same size (row×column), and the feature maps extracted by the multiple weight matrices of the same size have the same size, and then the multiple extracted feature maps of the same size are combined to form a convolution operation. Output.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; with the convolutional neural network As the network 200 deepens, the features extracted by the subsequent convolutional layers (for example, 226) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.

Pooling layer/pooling layer 220:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the 221-226 layers as illustrated by 220 in Figure 2, it can be a convolutional layer followed by a layer The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the size of the image space. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural network layer 230:

After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network 200 needs to use the neural network layer 230 to generate one or a group of required classes of output. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 2) and an output layer 240. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.

After the multiple hidden layers in the neural network layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (as shown in Figure 2 from 210 to 240 is the forward propagation) is completed, the back propagation (as shown in Figure 2 is the propagation from 240 to 210 is the back propagation) Start to update the weight values and deviations of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the output result of the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

In this application, the image classification model may include the convolutional neural network 200 shown in FIG. 2, and the image classification model may process the image to be processed to obtain the classification result of the image to be processed.

FIG. 3 is a hardware structure of a chip provided by an embodiment of the application, and the chip includes a neural network processor 30. The chip may be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111. The chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101. The algorithms of each layer in the convolutional neural network as shown in Figure 2 can be implemented in the chip as shown in Figure 3. Optionally, the convolutional neural network can be (one or more) included in the above-mentioned image classification model. A) one of the neural networks.

The neural network processor NPU 30 is mounted on a host CPU (host CPU) as a coprocessor, and the host CPU distributes tasks. The core part of the NPU is the arithmetic circuit 303. The controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform calculations.

In some implementation manners, the arithmetic circuit 303 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 303 fetches the data corresponding to the matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit 303. The arithmetic circuit 303 takes the matrix A data and the matrix B from the input memory 301 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 308.

The vector calculation unit 307 can perform further processing on the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on. For example, the vector calculation unit 307 can be used for network calculations in the non-convolution/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .

In some implementations, the vector calculation unit 307 can store the processed output vector to the unified buffer 306. For example, the vector calculation unit 307 may apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a combined value, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 303, for example for use in subsequent layers in a neural network.

The unified memory 306 is used to store input data and output data.

The weight data directly transfers the input data in the external memory to the input memory 301 and/or the unified memory 306 through the storage unit access controller 305 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 302, And the data in the unified memory 306 is stored in the external memory.

The bus interface unit (BIU) 310 is used to implement interaction between the main CPU, the DMAC, and the fetch memory 309 through the bus.

An instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304;

The controller 304 is used to call the instructions cached in the memory 309 to control the working process of the computing accelerator.

Generally, the unified memory 306, the input memory 301, the weight memory 302, and the instruction fetch memory 309 are all on-chip (On-Chip) memories. The external memory is a memory external to the NPU. The external memory can be a double data rate synchronous dynamic random access memory. Memory (double data rate synchronous dynamic random access memory, referred to as DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.

Among them, the operations of each layer in the convolutional neural network shown in FIG. 2 can be executed by the arithmetic circuit 303 or the vector calculation unit 307.

The execution device 110 in FIG. 1 introduced above can execute each step of the image classification method of the embodiment of the present application. Optionally, the execution device 110 in FIG. 1 may include the CNN model shown in FIG. 2 and the image classification method shown in FIG. Chip. The image classification method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

The image classification method provided in the embodiments of the present application can be executed on a server, can also be executed on the cloud, and can also be executed on a terminal device. Taking a terminal device as an example, as shown in FIG. 4, the technical solution of the embodiment of the present invention can be applied to a terminal device. The image classification method in the embodiment of the present application can classify an input image to obtain a classification result of the input image. The terminal device may be mobile or fixed. For example, the terminal device may be a mobile phone with image processing function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC). ), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, wearable device (WD) or self-driving vehicle, etc., embodiments of the present invention There is no restriction on this.

The classification of images (or pictures) is the basis of various image processing applications, and computer vision often involves the problem of how to classify the acquired images. However, the training of an image classification model requires a large amount of labeled training data. For image categories with few training samples (such as new categories), it is often impossible to effectively perform image classification or image recognition on images belonging to the category. In many cases, it is very difficult to obtain valid data, for example, in the medical field and security field.

Based on the foregoing problems, the embodiments of the present application propose an image classification method and an image classification model training method. For new classes (small samples) with few training samples, image classification or image recognition can also be implemented well.

FIG. 5 is a schematic flowchart of an image classification model training method 500 according to an embodiment of the present application. The method shown in FIG. 5 may be executed by a device with strong computing capability such as a computer device, a server device, or a computing device. For example, the method may be executed by the terminal device in FIG. 4. The method shown in Fig. 5 includes steps 510, 520, 530, and 540, which are respectively described in detail below.

S510: Acquire multiple training images in the training set.

Among them, the training set (training set) may be a set of all training images used during training.

Optionally, the training set includes images in the base class and images in the new class, and each class in the base class includes far more training images than each class in the new class.

Far more than it can be understood here is that the number of training images included in each category in the base class is at least an order of magnitude higher than the number of training images included in each category in the new class, that is, each category in the base class The number of training images included is at least ten times the number of training images included in each category in the new class.

For example, the base category includes 100 categories, each category includes 1000 training images, the new category includes 5 categories, and each category includes 5 training images.

Optionally, the image classification model in this application may be trained based on a multi-stage training (episodic training) strategy. For example, the model training process can be divided into multiple training episodes. In each training stage, several categories in the training set can be randomly selected to train the model. Finally, after multiple training stages, the image classification is completed Model training. For the description of multi-stage training, please refer to the prior art, which will not be repeated here.

Specifically, in multiple training stages in the model training process, the global category features can be updated, so that the global category features obtained by training can have better consistency. At the same time, the training set includes the base class The images in the new class and the images in the new class, the effect of the new class (training image in) during the training process can continue to accumulate with the global category features. Therefore, it can avoid the image classification model obtained by training from overfitting to the base class. According to the image classification model to classify the image to be processed, a better image classification result can be obtained.

The training method of the image classification model in the embodiment of the present application can better balance the difference between the number of samples in the base class and the number of samples in the new class, and the impact on the model training process. The detailed description of the training method of the image classification model involved in this application may be as described in the method 700 in FIG. 7.

Based on the above-mentioned multi-stage training strategy, the method 500 in this application can be divided into multiple training stages. Accordingly, the above-mentioned S510 can mean that in one training stage, multiple training images in the training set are randomly selected, and the extracted The multiple training images in the training set are divided into a support set (support set) and a query set (query set).

Therefore, it can also be said that the training set includes a support set and a query set, and the multiple training images in the training set include: multiple training images in the support set and multiple training images in the query set.

S520: Extract feature vectors of multiple training images in the query set according to the preset first neural network.

Wherein, the feature vector of the training image may be used to indicate the visual feature (or image feature) of the training image, and the query set includes partial images in partial categories in the training set.

Optionally, when extracting feature vectors of multiple training images in the query set, feature vectors of multiple training images in the support set may also be extracted.

Optionally, the first neural network may be trained by the system 100 shown in FIG. 1.

Optionally, the first neural network may be the convolutional neural network shown in FIG. 2.

S530: Process the feature vectors of the multiple training images in the query set according to the preset second neural network and the preset global category features to obtain classification results of the multiple training images in the query set.

It should be understood that the classification result here may refer to the label of the category to which the image belongs.

Optionally, the second neural network may be trained by the system 100 shown in FIG. 1.

Optionally, the second neural network may be the convolutional neural network shown in FIG. 2.

Wherein, the global category feature includes multiple category features, and multiple category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are in the training set The category to which all training images belong.

In this application, according to the preset second neural network and the preset global category features, the feature vectors of the multiple training images in the query set are processed to obtain multiple trainings in the query set The classification result of the image may include: extracting feature vectors of multiple training images in the support set; determining the local category features of the training images according to the feature vectors of the multiple training images in the support set; The network, the local category feature of the training image, and the global category feature determine the classification results of the multiple training images in the query set.

Wherein, multiple category features in the local category features of the training image are used to indicate the visual features of all categories in the support set, and the support set includes part of the training images in some categories in the training set, and The query set includes some training images in some categories in the training set.

Optionally, the category in the support set may be the same as the category in the inquiry set.

The steps of determining the classification result in the above S530 will be described in detail below with reference to FIG. 6.

As shown in FIG. 6, the multiple training images in the training set are divided into a support set and an inquiry set. The multiple training images in the support set may include images in the base class and images in the new class.

It should be noted that in a training phase, the support set and query set are determined by multiple training images randomly selected from the training set. The training set includes images in the base class and images in the new class. Therefore, the support set The multiple training images of can also include only images in the base class, or the multiple training images in the support set can also only include images in the new class.

As shown in Figure 6, the category features of multiple training images in the base class in the support set and the category features of multiple training images in the new class can be determined. For example, the feature vector of each training image in the base category can be determined, and the feature vectors of multiple training images belonging to the same category can be averaged, and the average can be used as the category feature of the category. The method of determining the category features of multiple training images in the new class will not be repeated here.

In particular, before determining the category features of the multiple training images in the new class, the multiple training images in the new class can also be cropped, flipped and/or data transformed ( hallucinator) and other image expansion processing to get more new types of images.

In this application, the above-mentioned (obtained) category features (including category features of multiple training images in the base category and/or category features of multiple training images in the new category) can be referred to as local category features of training images (local class representations).

It can be seen that the local category features of the training image are determined by multiple training images randomly selected (supported in the concentration) during a training phase, that is, these local category features only act on one of the model training processes. In the training phase, therefore, local category features can also be referred to as episodic class representations. Or the local category feature may also be other names, which is not limited in this application.

In contrast, global class representations can be considered as parameters of the image classification model, sharing the class features of all categories in the training set. Therefore, global class representations can be used in multiple training stages in the model training process.

Optionally, after the local category feature of the training image is obtained, the local category feature of the training image can be registered to the global category feature, so as to obtain the registration result, that is, the registered global category feature.

The registration here can also be understood as finding the category feature in the global category feature corresponding to each category feature in the local category feature. For example, find a category feature in the global category feature that has the highest similarity with each category feature in the local category feature.

Further, the registration error can be determined according to the similarity of each category feature in the local category feature and the category feature in the corresponding global category feature.

Specifically, the registration error may be determined based on the similarity between each category feature in the local category features and the category feature in the corresponding global category feature. That is, the registration error in the current training stage is determined according to the local category features of the training image and the multiple category features in the global category features.

Optionally, the second neural network may be used to perform dimensionality reduction processing on the registration result, so as to process feature vectors of multiple training images in the query set in a low-dimensional vector space.

In this application, the registration result can be used to predict each training image in the query set to obtain the classification result of each training image in the query set.

Specifically, the nearest neighbor method can be used to predict the classification result of the training image.

For example, the distance between the category feature in the registration result and the feature vector of each training image in the query set can be calculated, and each distance can be normalized to obtain each training image in the training image. The probability of belonging to the category indicated by the category feature in the registration result is the classification result of each training image in the query set.

Further, comparing the classification result of each training image with the pre-labeled label of the training image, the classification error can be obtained, wherein the pre-labeled label is used to indicate the true category to which the training image belongs. That is, the classification error of the current training stage is determined according to the classification results of the multiple training images and the pre-labeled labels of the multiple training images.

S540: Update the global category feature according to the classification results of the multiple training images in the query set.

In this application, the global category feature, the first neural network, and the second neural network may be updated according to the classification results of the multiple training images in the query set.

Optionally, the global category feature can be updated according to the classification error. Optionally, the classification error may be determined according to the classification results of the multiple training images by the method in S530.

Further, the global category feature, the first neural network and the second neural network may be updated according to the classification error.

Optionally, the global category feature can be updated according to the classification error and the registration error. Optionally, the classification error may be determined according to multiple category features in the local category feature of the training image and the global category feature by using the method in S530.

Further, the global category feature, the first neural network and the second neural network can be updated according to the classification error and the registration error.

In this application, in multiple training stages in the model training process, the global category features can be updated, so that the global category features obtained by training can have better consistency. At the same time, the training set includes the base The images in the class and the images in the new class. The effect of the new class (training image in) during the training process can be accumulated with the global category features. Therefore, it can avoid the image classification model obtained by training from overfitting to the base Class, according to the image classification model to classify the image to be processed, can more accurately identify the image in the new class.

FIG. 7 is a schematic flowchart of a method 700 for training an image classification model according to another embodiment of the present application. The method shown in FIG. 7 can be executed by a device with strong computing capabilities such as a computer device, a server device, or a computing device. For example, the method can be executed by the terminal device in FIG. 4. Each step in the method 700 will be described in detail below.

S710, feature extraction (module).

Extract feature vectors of multiple training images in the support set, the multiple training images including images in the base class and images in the new class.

Optionally, the above-mentioned first neural network may be used to extract feature vectors of multiple training images in the support set.

For example, suppose there is a set (multiple) training images, these training images can be used to train the above-mentioned image classification model, including a total of C _total = {c ₁ ,...,c _N } categories, where N represents the total number of categories , The c _jth category includes k _j training images, j is an integer. Based on the multiple training images, a training set D _train can be obtained, and the training set D _train includes a base class image and a new class image.

In the current training stage, C _train categories are randomly selected from the training set D _train , and n _s training images are extracted from each category to form a support set S={(x _i ,y _i ),i=1,... .,n _s ×C _train }, y _i is the label of the i-th training image x _i , C _train is the number of categories extracted; from the C _total categories in the training set D _train , C _train categories are extracted, Each category includes n _q training images, forming an inquiry set Q={(x _q ,y _q ),q=1,...,n _q ×C _train }, where n _s and n _q are positive Integer.

Optionally, F(·) can be used to represent feature extraction, and for the i-th training image x _i in the training set D _train , the feature vector is f _i =F(x _i ), where i is an integer.

S701, image expansion.

Optionally, for the new classes that support the concentration, image expansion processing such as cropping, flipping, and/or data transformation (hallucinator) can be performed on the images in the new class to obtain more new images. Class image.

It should be noted that in the training set D _train , there are only n _few training images for each new class, and n _few are often smaller than n _s +n _q .

Therefore, in S701, it is necessary to expand the n _few training images in the new class to n _s + n _q training images, and then put n _s training images into the support set, and put n _q training images into the support set. Enter the inquiry set.

S720, local category feature calculation.

Optionally, for multiple training images in the support set, the feature vectors of multiple training images belonging to the same category are calculated to average, and the average can be used as the category feature of the category.

其中，

为支持集中第c _j个类别的类别特征。 Calculate the category features of the categories to which all the training images in the support set belong, so that the local category features of the training images can be obtained, which is recorded as

among them,

To support the category characteristics of the c _jth category in the set.

S730, category feature registration (module).

Optionally, the local category feature of the training image is registered to the global category feature to obtain a registration result.

In other words, find the category feature in the global category feature corresponding to each category feature in the local category feature.

其中，

为训练集中第c _j个类别的类别特征。 For example, the global category feature is recorded as

among them,

Is the category feature of the c _jth category in the training set.

可以在低维的嵌入空间(embedding space)中计算局部类别特征中的类别特征f _i和全局类别特征中的类别特征

的相似度，得到向量

其中，第j个元素

表示f _i和

的相似度。 For the jth category feature f _i in the local category feature and the jth category feature in the global category feature

The category feature f _i in the local category feature and the category feature in the global category feature can be calculated in the low-dimensional embedding space (embedding space)

The similarity of the vector

Among them, the jth element

Denote f _i and

The similarity.

For example, the following formula may be used to calculate the similarity

Among them, θ(·) is the embedding function of the visual feature of the training image, and φ(·) is the embedding function of the global category feature.

At this time, a global category class characteristic features the highest similarity with the category feature f _i g _cj, wherein the global category corresponding to the category feature f _i.

Further, the registration error can be determined according to the similarity of each category feature in the local category feature and the category feature in the corresponding global category feature.

For example, in this application, the loss function L _{reg of the} training image x _i can be used as the registration error, so that the training image is closest to its global category feature in the low-dimensional embedding space. The formula for calculating the loss function L _reg is as follows.

L _reg ＝CE(y _i ,V _i )

Among them, CE(·) is the cross-entropy loss, and y _i is the label of the training image x _i .

的配准误差可以通过以下公式计算。 Therefore, the local category features of the c _jth category

The registration error of can be calculated by the following formula.

进行归一化处理，从而可以获得每个类别的概率分布

将概率分布P _i作为权重，估计所有类别的(全局类别特征中)类别特征的加权和，将该加权和作为当前训练阶段的C _train个类别中的第i类的类别表征，记为ξ _i，即ξ _i＝P _iG。ξ _i就是上述实施例中的配准结果。 Next, you can use the softmax function to

Perform normalization processing to obtain the probability distribution of each category

The probability distribution P _i as a weight, estimated for all categories (global category feature) class characteristic weighted sum, the category representation weighted sum as the current training phase C _train a category of class i, referred to as a [xi] _i , That is, ξ _i =P _i G. ξ _i is the registration result in the above embodiment.

S740, image classification.

Optionally, the registration result can be used to predict each training image in the query set, and obtain a classification result of each training image in the query set.

Further, comparing the classification result of each training image with the pre-labeled label (true label) of the training image, the classification error can be obtained.

For example, for a given category feature ξ _i , the loss function L _fsl can be used as the classification error of the training images (x _k , y _k ) in the query set, and the loss function L _fsl is shown in the following formula.

L _fsl =CE(y _k ,W _k )

c _i∈C _train

c _i ∈C _train

表示问询集中的训练图像x _k与估计的全局类别特征ξ _i的相似度。 among them,

Indicates the similarity between the training image x _{k in the} query set and the estimated global category feature ξ _i .

对应的类别作为训练图像x _k的类别，即训练图像x _k的分类结果。 At this time, the most similar

The corresponding category is taken as the category of the training image x _k , that is, the classification result of the training image x _k .

S750, the image classification model is updated.

Optionally, registration errors and/or classification errors can be used to update the image classification model.

For example, in this application, the registration error (L _reg ) and the classification error (L _fsl ) can be combined, and the total loss function L _total (·) of multiple training stages can be calculated by the following formula.

Optionally, S750 may include at least one of S751, S752, and S753.

S751: Update the feature extraction (module).

Optionally, registration error (L _reg ), classification error (L _fsl ), and/or total loss function (L _total ) can be used to update the feature extraction (module).

S752, update the category feature registration (module).

Optionally, the registration error (L _reg ), the classification error (L _fsl ) and/or the total loss function (L _total ) can be used to update the category feature registration (module).

S753: Update the global category characteristics.

Optionally, the registration error (L _reg ), the classification error (L _fsl ), and/or the total loss function (L _total ) can be used to update the global category features.

FIG. 8 shows a schematic flowchart of an image classification method 800 provided by an embodiment of the present application. The method shown in FIG. 8 may be executed by a device with strong computing capabilities such as a computer device, a server device, or a computing device. For example, the The method can be executed by the terminal device in FIG. 4. The method shown in FIG. 8 includes steps 810 and 820, which are respectively described in detail below.

S810: Acquire an image to be processed.

Optionally, when the method 800 shown in FIG. 8 is executed by the terminal device in FIG. 4, the image to be processed may be an image captured by the terminal device through a camera; or, the image to be processed may also be from the terminal device. The obtained image, for example, the image stored in the album of the terminal device, or the image obtained by the terminal device from the cloud.

S820: Classify the image to be processed according to preset global category features, to obtain a classification result of the image to be processed.

Wherein, the global category features include multiple category features obtained by training based on multiple training images in the training set, and multiple category features in the global category features are used to indicate visual features of all categories in the training set, so All categories in the training set are categories to which all training images in the training set belong, and the training set includes images in the base class and images in the new class.

Optionally, the classifying the image to be processed according to preset global category features to obtain the classification result of the image to be processed may include: extracting the feature vector of the image to be processed; Process the feature vector of the image, and determine the confidence that the image to be processed belongs to a candidate category, where the candidate category is one or more of the multiple categories indicated by the global category feature; according to the confidence, from the The classification result of the image to be processed is determined from the candidate category.

Optionally, before the determining the confidence that the image to be processed belongs to the candidate category according to the feature vector of the image to be processed, the method may further include: determining the Process the local category feature of the image; determine the candidate category according to the local category feature of the image to be processed and the global category feature.

Wherein, the support set of the image to be processed includes multiple images, and the category to which the multiple images belong is one or more of the multiple categories indicated by the global category feature.

Optionally, the candidate category may include the category to which all training images in the support set belong.

Optionally, the determining the confidence that the image to be processed belongs to the candidate category according to the feature vector of the image to be processed may include: determining the feature of the image to be processed according to the feature vector of the image to be processed The distance between the vector and the feature vector corresponding to each of the candidate categories; and according to the distance, the confidence that the image to be processed belongs to the candidate category is determined.

Optionally, the determining the classification result of the image to be processed from the candidate categories according to the confidence may include: determining the category with the highest confidence in the candidate categories as the The classification result of the image to be processed.

It should be understood that the confidence level here may be the probability that the image to be processed belongs to the candidate category. Therefore, the degree of confidence is the greatest. It can also be said that the image to be processed has the greatest probability of belonging to the candidate category.

Optionally, the global category feature is obtained by training according to a classification error, and the classification error is determined according to the classification result of the training image in the query set and the pre-labeled label of the training image in the query set. The label is used to indicate the category to which the training image belongs, and the query set includes part of the training images in the partial categories in the training set. For the specific process of determining the classification error, refer to the method 500 in FIG. 5.

Optionally, the global category feature is obtained by training based on registration error, the registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, the training image The local category features include multiple category features determined according to multiple training images in the support set, and multiple category features in the local category features of the training image are used to indicate visual features of all categories in the support set, the The support set includes some training images in some categories in the training set. For the specific process of determining the registration error, refer to the method 500 in FIG. 5.

Optionally, the global category feature is obtained by training based on classification error and registration error.

Optionally, the local category feature of the training image is determined by a plurality of training images in the support set that undergoes expansion processing, and the expansion processing includes clipping, flipping, and/or data transformation processing on the image.

In this application, the global category features are trained from the classification results of multiple training images in the training set. The global category features include multiple category features that can indicate the visual features corresponding to all categories in the training set. At the same time, because The training set used in the global category training process includes images in the base category and images in the new category, which can prevent the global category features from being overfitted to the images in the base category, thereby enabling more accurate identification of the images in the new category. image.

FIG. 9 is a schematic diagram of the hardware structure of an image classification device according to an embodiment of the present application. The image classification device 4000 shown in FIG. 9 includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. Among them, the memory 4001, the processor 4002, and the communication interface 4003 implement communication connections between each other through the bus 4004.

The memory 4001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image classification method of the embodiment of the present application.

The processor 4002 may adopt a general-purpose central processing unit (central processing unit, CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more The integrated circuit is used to execute related programs to realize the functions required by the units in the image classification device of the embodiment of the present application, or to execute the image classification method of the method embodiment of the present application.

The processor 4002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image classification method in the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 4002 or instructions in the form of software.

The above-mentioned processor 4002 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an ASIC, a ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic Devices, discrete hardware components. The aforementioned general-purpose processor may be a microprocessor or the processor may also be any conventional processor. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and combines its hardware to complete the functions required by the units included in the image classification apparatus of the embodiment of the application, or execute the image classification of the method embodiment of the application. method.

The communication interface 4003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the image to be processed can be acquired through the communication interface 4003.

The bus 4004 may include a path for transferring information between various components of the device 4000 (for example, the memory 4001, the processor 4002, and the communication interface 4003).

FIG. 10 is a schematic diagram of the hardware structure of an image classification model training device 5000 according to an embodiment of the present application. Similar to the above device 4000, the image classification model training device 5000 shown in FIG. 10 includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. Among them, the memory 5001, the processor 5002, and the communication interface 5003 implement communication connections between each other through the bus 5004.

The memory 5001 may store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 is configured to execute each step of the neural network training method of the embodiment of the present application.

The processor 5002 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits to execute related programs to implement the image classification model training method of the embodiment of the present application.

The processor 5002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the image classification model in the embodiment of the present application can be completed by the integrated logic circuit of hardware in the processor 5002 or instructions in the form of software.

It should be understood that the image classification model is trained by the image classification model training device 5000 shown in FIG. 10, and the image classification model obtained by training can be used to execute the image classification method of the embodiment of the present application. Specifically, training the image classification model by the device 5000 can obtain the image classification model in the methods shown in FIG. 5 and FIG. 8.

Specifically, the device shown in FIG. 10 can obtain training data and the image classification model to be trained from the outside through the communication interface 5003, and then the processor trains the image classification model to be trained according to the training data.

It should be noted that although the foregoing device 4000 and device 5000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 4000 and device 5000 may also include those necessary for normal operation. Other devices. At the same time, according to specific needs, those skilled in the art should understand that the device 4000 and the device 5000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 4000 and the device 5000 may also only include the components necessary to implement the embodiments of the present application, and not necessarily include all the components shown in FIG. 9 and FIG. 10.

It should be understood that the processor in this embodiment of the application may be a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), and application-specific integrated circuits. (application specific integrated circuit, ASIC), ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

It should also be understood that the memory in the embodiments of the present application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electronic Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of random access memory (RAM) are available, such as static random access memory (static RAM, SRAM), dynamic random access memory (DRAM), and synchronous dynamic random access memory (DRAM). Access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory Take memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

The foregoing embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above-mentioned embodiments may be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website, computer, server or data center via wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more sets of available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state drive.

It should be understood that the term "and/or" in this article is only an association relationship describing the associated objects, indicating that there can be three types of relationships, for example, A and/or B can mean: A alone exists, and both A and B exist. , There are three cases of B alone, where A and B can be singular or plural. In addition, the character "/" in this document generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and/or" relationship, which can be understood with reference to the context.

In this application, "at least one" refers to one or more, and "multiple" refers to two or more. "The following at least one item (a)" or similar expressions refers to any combination of these items, including any combination of a single item (a) or plural items (a). For example, at least one item (a) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple .

It should be understood that, in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, rather than corresponding to the embodiments of the present application. The implementation process constitutes any limitation.

A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described system, device, and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (32)

An image classification method, characterized in that it includes: Obtain the image to be processed; The image to be processed is classified according to the preset global category features to obtain the classification result of the image to be processed, wherein the global category features include multiple category features trained based on multiple training images in the training set The multiple category features in the global category feature are used to indicate the visual features of all categories in the training set, all categories in the training set are categories to which all training images in the training set belong, and the training set Including images in the base class and images in the new class. The method according to claim 1, wherein the classifying the image to be processed according to a preset global category feature to obtain a classification result of the image to be processed comprises: Extracting the feature vector of the image to be processed; Determine, according to the feature vector of the image to be processed, the confidence that the image to be processed belongs to a candidate category, where the candidate category is one or more of multiple categories indicated by the global category feature; According to the confidence level, the classification result of the image to be processed is determined from the candidate category. The method according to claim 2, wherein before the determining the confidence that the image to be processed belongs to the candidate category according to the feature vector of the image to be processed, the method further comprises: Determine the local category feature of the image to be processed according to the support set of the image to be processed; Determining the candidate category according to the local category feature and the global category feature of the image to be processed; Wherein, the support set of the image to be processed includes multiple images, and the category to which the multiple images belong is one or more of the multiple categories indicated by the global category feature. The method according to claim 2 or 3, wherein the determining the confidence that the image to be processed belongs to a candidate category according to the feature vector of the image to be processed comprises: Determining the distance between the feature vector of the image to be processed and the feature vector corresponding to each of the candidate categories according to the feature vector of the image to be processed; According to the distance, the confidence that the image to be processed belongs to the candidate category is determined. The method according to any one of claims 2 to 4, wherein the determining a classification result of the image to be processed from the candidate category according to the confidence level comprises: Determine the category with the greatest confidence among the candidate categories as the classification result of the image to be processed. The method according to any one of claims 1 to 5, wherein the global category feature is obtained by training based on classification error, and the classification error is based on the classification result of the training image in the query set and the The training image in the query set is determined by pre-labeled labels, the label is used to indicate the category to which the training image belongs, and the query set includes part of the training images in the partial categories in the training set. The method according to any one of claims 1 to 5, wherein the global category feature is obtained by training based on classification error and registration error, and the classification error is based on the classification of training images in the query set The result and the pre-labeled labels of the training images in the query set are determined, the labels are used to indicate the category to which the training image belongs, and the query set includes some training images in some categories in the training set, The registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, and the local category feature of the training image includes multiple categories determined according to multiple training images in the support set Features, multiple category features in the local category features of the training image are used to indicate visual features of all categories in the support set, and the support set includes part of the training images in the partial categories in the training set. The method according to claim 7, wherein the local category feature of the training image is determined by a plurality of training images in the support set after expansion processing, and the expansion processing includes clipping the image, Flip processing and/or data transformation processing. A method for training an image classification model, characterized in that it includes: Acquiring multiple training images in a training set, wherein the training set includes a support set and a training set, and the multiple training images include multiple training images in the support set and multiple training images in the query set; Extracting feature vectors of multiple training images in the query set according to a preset first neural network, where the query set includes partial images in partial categories in the training set; According to the preset second neural network and the preset global category features, the feature vectors of the multiple training images in the query set are processed to obtain the classification results of the multiple training images in the query set, where, The global category features include multiple category features, and multiple category features in the global category features are used to indicate visual features of all categories in the training set, and all categories in the training set are all categories in the training set. The category to which the training image belongs, the training set includes images in the base category and images in the new category; The global category feature is updated according to the classification results of the multiple training images in the query set. The method according to claim 9, wherein the feature vectors of the multiple training images in the query set are processed according to the preset second neural network and the preset global category features to obtain all Describe the classification results of multiple training images in the query set, including: Extracting feature vectors of multiple training images in the support set, where the support set includes some training images in some categories in the training set; Determine the local category features of the training image according to the feature vectors of the multiple training images in the support set, where multiple category features in the local category features of the training image are used to indicate visual features of all categories in the support set; According to the second neural network, the local category feature of the training image, and the global category feature, the classification results of the multiple training images in the query set are determined. The method according to claim 9 or 10, wherein the updating the global category feature according to the classification results of the multiple training images in the query set comprises: According to the classification results of the multiple training images in the query set, the global category feature, the first neural network and the second neural network are updated. The method according to any one of claims 9 to 11, wherein the updating the global category feature according to the classification results of the multiple training images in the query set comprises: Updating the global category feature according to the classification error, the classification error being determined according to the classification results of the multiple training images in the query set and the pre-labeled labels of the multiple training images in the query set, the The label is used to indicate the category to which the training image belongs. The method according to any one of claims 10 to 12, wherein the updating the global category feature according to the classification results of the multiple training images comprises: The global category feature is updated according to the classification error and the registration error, wherein the registration error is determined according to the local category feature of the training image and multiple category features in the global category feature. The method according to claim 13, wherein the local category feature of the training image is determined by a plurality of training images in the support set after expansion processing, and the expansion processing includes clipping the image, Flip processing and/or data transformation processing. An image classification device, characterized by comprising: The acquisition module is used to acquire the image to be processed; The classification module is used to classify the image to be processed according to preset global category features to obtain the classification result of the image to be processed, wherein the global category features include training obtained from multiple training images in the training set The multiple category features of the global category feature are used to indicate the visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong , The training set includes images in the base class and images in the new class. The device according to claim 15, wherein the classification module is specifically configured to: Extracting the feature vector of the image to be processed; Determine, according to the feature vector of the image to be processed, the confidence that the image to be processed belongs to a candidate category, where the candidate category is one or more of multiple categories indicated by the global category feature; According to the confidence level, the classification result of the image to be processed is determined from the candidate category. The device according to claim 16, characterized in that the device further comprises a determining module for: Determine the local category feature of the image to be processed according to the support set of the image to be processed; Determining the candidate category according to the local category feature and the global category feature of the image to be processed; Wherein, the support set of the image to be processed includes multiple images, and the category to which the multiple images belong is one or more of the multiple categories indicated by the global category feature. The device according to claim 16 or 17, wherein the classification module is specifically configured to: Determining the distance between the feature vector of the image to be processed and the feature vector corresponding to each of the candidate categories according to the feature vector of the image to be processed; According to the distance, the confidence that the image to be processed belongs to the candidate category is determined. The device according to any one of claims 16 to 18, wherein the classification module is specifically configured to: Determine the category with the greatest confidence among the candidate categories as the classification result of the image to be processed. The device according to any one of claims 15 to 19, wherein the global category feature is obtained by training based on classification error, and the classification error is based on the classification result of the training image in the query set and the The training image in the query set is determined by pre-labeled labels, the label is used to indicate the category to which the training image belongs, and the query set includes part of the training images in the partial categories in the training set. The device according to any one of claims 15 to 19, wherein the global category feature is obtained by training based on classification error and registration error, and the classification error is based on the classification of training images in the query set The result and the pre-labeled labels of the training images in the query set are determined, the labels are used to indicate the category to which the training image belongs, and the query set includes some training images in some categories in the training set, The registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, and the local category feature of the training image includes multiple categories determined according to multiple training images in the support set Features, multiple category features in the local category features of the training image are used to indicate visual features of all categories in the support set, and the support set includes part of the training images in the partial categories in the training set. The device according to claim 21, wherein the local category feature of the training image is determined by a plurality of training images in the support set after expansion processing, and the expansion processing includes clipping the image, Flip processing and/or data transformation processing. A training device for an image classification model, characterized in that it comprises: An acquisition module for acquiring multiple training images in a training set, where the training set includes a support set and a training set, and the multiple training images include multiple training images in the support set and multiple training images in the query set Multiple training images; A feature extraction module, configured to extract feature vectors of multiple training images in the query set according to a preset first neural network, where the query set includes some images in some categories in the training set; The classification module is used to process the feature vectors of the multiple training images in the query set according to the preset second neural network and the preset global category features to obtain the information of the multiple training images in the query set The classification result, wherein the global category features include multiple category features, and the multiple category features in the global category features are used to indicate the visual features of all categories in the training set, and all categories in the training set are all categories. The category to which all training images in the training set belong, and the training set includes images in the base category and images in the new category; The update module is used to update the global category features according to the classification results of the multiple training images in the query set. The device according to claim 23, wherein the feature extraction module is further configured to: Extracting feature vectors of multiple training images in the support set, where the support set includes some training images in some categories in the training set; The classification module is specifically used for: Determine the local category features of the training images according to the feature vectors of the multiple training images in the support set, where multiple category features in the local category features of the training images are used to indicate visual features of all categories in the support set, The support set includes some training images in some categories in the training set; According to the second neural network, the local category feature of the training image, and the global category feature, the classification results of the multiple training images in the query set are determined. The device according to claim 23 or 24, wherein the update module is specifically configured to: According to the classification results of the multiple training images in the query set, the global category feature, the first neural network and the second neural network are updated. The device according to any one of claims 23 to 25, wherein the update module is specifically configured to: Updating the global category feature according to the classification error, the classification error being determined according to the classification results of the multiple training images in the query set and the pre-labeled labels of the multiple training images in the query set, the The label is used to indicate the category to which the training image belongs. The device according to any one of claims 24 to 26, wherein the update module is specifically configured to: The global category feature is updated according to the classification error and the registration error, where the registration error is determined according to the local category feature of the training image and multiple category features in the global category feature. The device according to claim 27, wherein the local category feature of the training image is determined by a plurality of training images in the support set after expansion processing, and the expansion processing includes clipping the image, Flip processing and/or data transformation processing. An image classification device, characterized by comprising a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the program instructions to execute the method according to any one of claims 1 to 8 . A training device for an image classification model, characterized by comprising a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions to execute any one of claims 9 to 14 The method described. A computer-readable storage medium, characterized in that the computer-readable medium stores program code for device execution, and the program code includes a program code for executing any one of claims 1 to 8 or 9 to 14. Methods. A chip, characterized in that the chip comprises a processor and a data interface, and the processor reads instructions stored on a memory through the data interface to execute any one of claims 1 to 8 or 9 to 14 The method described in one item.

Images