WO2020001196A1 - Image processing method, electronic device, and computer readable storage medium - Google Patents

Abstract

An image processing method, comprising: inputting training images to a neural network, and obtaining sample vectors output from a neural network activation layer; clustering the sample vectors according to the number of classifications, and obtaining clustering centers and classification vectors corresponding to the classifications; detecting the similarity between the clustering centers and the classification vectors in the classifications, and using the training images corresponding to the classification vectors of which the similarity is greater than a first threshold value as a first class of training images of the classifications.

Description

Image processing method, electronic device, computer-readable storage medium

Cross-reference to related applications

This application claims the priority of a Chinese patent application filed on June 28, 2018 with the Chinese Patent Office, application number 2018106860743, and the invention name is "image processing method and device, electronic device, computer-readable storage medium", and its entire content Incorporated by reference in this application.

Technical field

The present application relates to the field of computer technology, and in particular, to an image method, an electronic device, and a computer-readable storage medium.

Background technique

With the continuous development of computers and deep learning technologies, the use of deep learning algorithms to identify and classify images has become an important part of image processing. Deep learning algorithms require a large number of training images. In traditional technology, engineers formulate screening criteria and screen a large number of images to obtain training images based on the screening criteria. However, there is a problem of low efficiency in screening training images.

Summary of the invention

According to various embodiments of the present application, an image processing method, an electronic device, and a computer-readable storage medium are provided.

An image processing method includes:

Input the training image to the neural network, and obtain the sample vector output by the activation layer of the neural network;

Clustering the sample vector according to the number of classifications to obtain the clustering center and classification vector corresponding to each classification; level

The similarity between the clustering center and the classification vector in each classification is detected, and the training image corresponding to the classification vector whose similarity is greater than the first threshold is used as the first type of training image for the classification.

An electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor causes the processor to perform the following operations:

Input the training image to the neural network, and obtain the sample vector output by the activation layer of the neural network;

Clustering the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification; and

The similarity between the clustering center and the classification vector in each classification is detected, and the training image corresponding to the classification vector whose similarity is greater than the first threshold is used as the first type of training image for the classification.

A computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, the following operations are implemented:

Input the training image to the neural network, and obtain the sample vector output by the activation layer of the neural network;

Clustering the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification; and

The similarity between the clustering center and the classification vector in each classification is detected, and the training image corresponding to the classification vector whose similarity is greater than the first threshold is used as the first type of training image for the classification.

The image processing method, electronic device, and computer-readable storage medium provided in the embodiments of the present application can filter training images of each category according to the sample vectors output by the activation layer of the neural network, which can improve the filtering efficiency of the training image.

Details of one or more embodiments of the present application are set forth in the accompanying drawings and description below. Other features, objects, and advantages of the invention will be apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present application or the prior art more clearly, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without paying creative work.

FIG. 1 is a flowchart of an image processing method in one or more embodiments.

FIG. 2 is a flowchart of performing clustering processing on sample vectors in one or more embodiments.

FIG. 3 is a flowchart of performing clustering processing on sample vectors in another or more embodiments.

FIG. 4 is a schematic diagram of a clustering process of sample vectors in one or more embodiments.

FIG. 5 is a flowchart of an image processing method in one or more embodiments.

FIG. 6 is a flowchart of an image processing method in another embodiment.

FIG. 7 is a structural block diagram of an image processing apparatus in one or more embodiments.

FIG. 8 is a schematic diagram of an internal structure of an electronic device in one or more embodiments.

FIG. 9 is a schematic diagram of an image processing circuit in one or more embodiments.

detailed description

In order to make the purpose, technical solution, and advantages of the present application clearer, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

It can be understood that the terms “first”, “second”, and the like used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first image may be referred to as a second image, and similarly, the second image may be referred to as a first image. Both the first image and the second image are images, but they are not the same image.

FIG. 1 is a flowchart of an image processing method according to an embodiment. As shown in FIG. 1, an image processing method includes operations 102 to 106. among them:

In operation 102, a training image is input to a neural network, and a sample vector output by an activation layer of the neural network is obtained.

The training image may be an image stored locally on the electronic device, or an image downloaded from the network by the electronic device. During the training of the neural network, a large number of training images are needed for training. Neural network refers to a computing model composed of a large number of nodes (neurons) connected to each other. Specifically, the neural network may be CNN (Convolutional Neural Network, Convolutional Neural Network), DNN (Deep Neural Network, Deep Neural Network), RNN (Recurrent Neural Network, Recurrent Neural Network), etc., and is not limited thereto. A neural network generally includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the input of the image; the hidden layer is used to process the received image; the output layer is used to output the processing result of the image processing. The hidden layers of the neural network may include a convolutional layer, an activation layer, a pooling layer, and a fully connected layer. The sample vector refers to a vector composed of feature values in a feature map output by a neural network activation layer according to a preset rule after a training image is input into a neural network.

The electronic device can input the training image into the neural network, and the electronic device can obtain a sample vector composed of the feature values in the feature map according to the rules according to the feature map output by the activation layer in the neural network.

Operation 104: Perform clustering processing on the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification.

The number of classifications refers to the number of scene classifications that the trained neural network can use for recognition. The number of classifications can also refer to the number of scene classifications in the training image. In one embodiment, the classification can be landscape, beach, blue sky, green grass, snow, night, dark, backlight, sunrise / sunset, fireworks, spotlight, indoor, macro, text document, portrait, baby, cat, dog , Food, etc. are not limited to this. Clustering refers to the process by which training images are divided into multiple classifications composed of similar scene classifications. Specifically, the electronic device can use a partitioning method such as K-MEANS (hard clustering) algorithm or K-MEDOIDS (center point) algorithm, and a hierarchical method such as BIRCH (Balanced Iterative Reducing and Clustering Using Hierarchies). (Clustering) algorithm and other algorithms, such as graph theory clustering method, perform clustering processing on training images. The cluster center refers to the center vector of each classification, and the distance between the cluster center in the classification and the classification vector in the classification is the smallest. A classification vector refers to a sample vector corresponding to each classification. For example, there are sample vectors A, B, C, and D. If the classification is M and N, the electronic device performs cluster processing on the sample vectors. Sample vectors A and B form classification M, and sample vectors C and D form classification N. , The classification vectors corresponding to classification M are A and B, and the classification vectors corresponding to classification N are C and D.

The electronic device performs cluster processing on the sample vector according to the required number of classifications, and can obtain a classification vector and a clustering center corresponding to each classification.

Operation 106: Detect the similarity between the clustering center and the classification vector in each classification, and use the training image corresponding to the classification vector whose similarity is greater than the first threshold as the first type of training image for classification.

There are various methods for the electronic device to detect the similarity between the cluster center and the classification vector. In one embodiment, the electronic device may determine the similarity between the cluster center and the classification vector by calculating the distance between the cluster center and the classification vector; the greater the distance, the more similar the classification vector is to the cluster center of the classification. The smaller the distance, the greater the similarity between the classification vector and the clustering center of the classification. The electronic device can set corresponding similarity for different distance values in advance, obtain the distance value between the cluster center and the classification vector through detection, and obtain the corresponding similarity according to the distance value as the similarity between the cluster center and the classification vector. The first threshold can be set according to the requirements of the actual application. For example, the first threshold may be 70%, 80%, 90%, etc., and is not limited thereto. The first type of training image refers to a training image with the same classification that can be used to train a neural network that can recognize the classification, that is, the first type of training image can be used as a positive sample in neural network training.

The electronic device can obtain the clustering center of each classification and the classification vector in the classification, detect the similarity between each classification vector in the classification and the clustering center of the classification, and obtain the training image corresponding to the classification vector whose similarity is greater than the first threshold. This training image is used as the first type of training image in this classification.

In the embodiment provided by the present application, by inputting a training image to a neural network, obtaining a sample vector output from an activation layer of the neural network, and performing cluster processing on the sample vector according to the number of classifications, to obtain a clustering center and a classification vector corresponding to each classification. Detecting the similarity between the clustering center and the classification vector in each classification, and using the training image corresponding to the classification vector whose similarity is greater than the first threshold value as the first type of training image for classification can improve the efficiency of filtering the training image.

In one embodiment, the process of obtaining a sample vector output by the activation layer of the neural network in the provided image processing method includes: obtaining a sample vector output by the penultimate activation layer in the neural network.

The activation layer of the neural network is a layer for performing a function change on the feature map obtained through the convolution layer according to the activation function. The electronic device can obtain a sample vector of the output of the activation layer in the neural network. In one embodiment, the electronic device can obtain a sample vector output by the penultimate activation layer in the neural network. For example, when the training image E is input into the neural network, the output of each activation layer in the neural network is out (1), out (2), ..., out (k), and k is the number of activation layers in the neural network. Then, the electronic device can obtain the vector output by the penultimate activation layer in the neural network, that is, out (k-1), as the sample vector corresponding to the training image E.

The electronic device obtains the sample vector output from the penultimate activation layer in the neural network, which can increase the dimension of the sample vector and facilitate the differentiation of different training images. The sample vector is clustered according to the number of classifications to obtain the clustering center corresponding to each classification. And classification vector, detecting the similarity between the clustering center and the classification vector in each classification, and using the training image corresponding to the classification vector whose similarity is greater than the first threshold as the first type of training image for classification, can improve the efficiency of training image screening.

As shown in FIG. 2, in one embodiment, a process of performing clustering processing on a sample vector according to the number of classifications in the provided image processing method includes operations 202 to 206. among them:

In operation 202, a classification number of center vectors are configured according to the classification number.

The electronic device configures the number of classification center vectors according to the number of classifications, and each classification corresponds to a center vector. Specifically, the electronic device may randomly select the classified number of center vectors according to the classified number. In one embodiment, the electronic device may also obtain a preset number of classified center vectors.

Operation 204: Adjust each center vector according to the distance between the sample vector and each center vector.

The distance between the sample vector and each center vector can be detected using the distance formula. Specifically, the distance formula may be a Manhattan distance formula, a Euclidean distance formula, a relative entropy formula, and the like, and is not limited thereto. The electronic device can detect the distance between the sample vector and each center vector according to the distance formula, use the classification corresponding to the center vector with the smallest sample vector distance as the classification of the sample vector, and center the center according to the distance between each sample vector and the center vector in the classification. The vector is adjusted.

Operation 206: Obtain the adjusted center vector as the classified cluster center.

The electronic device obtains the adjusted center vector as the clustering center of the classification, and among the sample vectors, the sample vector with the smallest distance from the clustering center of the classification is the classification vector in the classification.

The electronic device configures the number of classified center vectors according to the number of classifications, adjusts each center vector according to the distance between the sample vector and each center vector, and obtains the adjusted center vector as the clustering center of the classification, and can obtain the clustering center corresponding to each classification.

As shown in FIG. 3, in one embodiment, the process of adjusting each center vector according to the distance between the sample vector and each center vector includes operations 302 to 310. among them:

Operation 302: Configure a classified number of center vectors according to the classified number.

In operation 304, the classification corresponding to the center vector with the smallest distance between the sample vectors is used as the classification of the sample vectors.

After the electronic device detects the distance between the sample vector and each center vector, and classifies the sample vector with the smallest distance from the center vector into a classification, the distance between all the sample vectors and the center vector in the classification is the lowest.

Operation 306: Adjust the center vector according to the distance between each sample vector and the center vector in the classification.

The electronic device adjusts the center vector so that the distance between all the sample vectors and the center vector in the classification is the smallest. Specifically, the electronic device may use a square error to construct an objective function. The objective function is the sum of the squared differences of the distances of all sample vectors in the classification to the clustering centers of the classification. The distance of the vector is the smallest, so the objective function needs to be smaller. The electronic device can obtain the partial derivative of the objective function to obtain the update function of the center vector, and adjust the center vector according to the update function and the distance between each sample vector and the center vector in the classification.

Operation 308: Repeat the classification of the center vector with the smallest distance from the sample vector as the sample vector according to the adjusted center vector, and adjust the center vector according to the distance between each vector and the center vector in the classification.

The adjusted center vector changes, so the distance between each sample vector and the adjusted center vector also changes. The electronic device uses the adjusted center vector as the new center vector of each classification, re-detects the distance between each sample vector and the center vector, and uses the classification corresponding to the center vector with the smallest sample vector distance as the new classification of the sample vector, and Adjust the center vector according to the distance between each sample vector and the center vector in the classification.

In operation 310, when the number of adjustments exceeds a preset number of times, a final center vector is obtained as a clustering center for classification.

The preset number of times can be set according to the requirements of the actual application, which is not limited here. The electronic device can obtain the center vector obtained as the cluster center of each classification when the number of adjustments of the center vector exceeds a preset number of times. Among the sample vectors, the sample vector with the smallest distance from the cluster center is the classification in the classification. vector. The electronic device may also obtain the adjusted center vector as the clustering center when the distance between the adjusted center vector and the last adjusted center vector is smaller than the first distance value. In one embodiment, the electronic device may also obtain the center vector as the clustering center of the classification when the distance between all the sample vectors and the center vector in the classification is smaller than the second distance value.

FIG. 4 is a schematic diagram of a clustering process of sample vectors in an embodiment. As shown in (a) of FIG. 4, the vectors F, G, H, I, and J are sample vectors corresponding to the training images F, G, H, I, and J, respectively; as shown in (b) of FIG. 4, When the number of classifications is 2, the electronic device can randomly configure two center vectors X and Y; the electronic device can detect the distance between each sample vector F, G, H, I, J and the center vector X and Y, as shown in FIG. 4 As shown in (c), the distance between the sample vector F, G, H and the center vector X is less than the distance from the center vector Y, and the distance between the sample vector I, J and the center vector X is greater than the distance from the center vector Y, then The sample vectors F, G, and H are classified into the classification corresponding to the center vector X, and the sample vectors I and J are classified into the classification corresponding to the center vector Y. As shown in (d) of FIG. 4, after the classification is completed, the electronic device The vectors F, G, and H adjust the center vector X, and the adjusted center vector is X1. The center vector is adjusted according to the sample vectors I and J, and the adjusted center vector is Y1. The electronic device needs to re-test the sample. The distances between the vectors F, G, H, I, J and the center vectors X1, Y1 are shown in (e) in Figure 4. The sample vectors H and The distance of the heart vector Y1 is smaller than the distance from the center vector X1, and the sample vector H is re-classified into the classification corresponding to the center vector Y1. During the clustering process, the electronic device can continue to repeatedly calculate (c) and FIG. 4 in FIG. 4 The process of (d) in 4 adjusts the center vector until the number of adjustments exceeds a preset number. As shown in (f) in FIG. 4, the center vectors Xn and Yn are the center vectors obtained by adjusting n times. When n is greater than a preset number, the electronic device can obtain Xn and Yn as the clustering centers of classification, respectively.

The electronic device adjusts the center vector according to the classification corresponding to the center vector with the smallest distance from the sample vector as the sample vector, adjusts the center vector according to the distance between each sample vector and the center vector in the classification, and readjusts the center vector according to the adjusted center vector. When the number of adjustments exceeds a preset number, the final center vector is obtained as the clustering center of the classification, and the clustering center corresponding to each classification can be obtained.

As shown in FIG. 5, in one embodiment, the process of adjusting each center vector according to the distance between the sample vector and each center vector includes operations 502 to 504. among them:

In operation 502, the similarity between the cluster center of each classification and the sample vector is detected.

The electronic device can determine the similarity between the cluster center and the classification vector by calculating the distance between the cluster center and the classification vector; the larger the distance, the smaller the similarity between the classification vector and the cluster center of the classification, and the smaller the distance, The greater the similarity between the classification vector and the clustering center of the classification; the electronic device can set the similarity for different distance values in advance.

In operation 504, the training image corresponding to the sample vector whose similarity between the cluster centers is less than the second threshold is used as the classified second type training image.

The second threshold may be set according to the requirements of the actual application, and may be, for example, 10%, 20%, 30%, and the like, without being limited thereto. The electronic device detects the similarity between each classification vector in the classification and the clustering center of the classification, obtains a training image corresponding to the classification vector whose similarity is less than the second threshold, and uses the training image as the second type of training image in the classification. The second type of training image may be a training image used to reduce the error rate of the neural network that can be trained to recognize the classification, that is, the second type of training image may be used as a negative sample in the training of the neural network.

The electronic device detects the similarity between the cluster center of each classification and the sample vector, and uses the training image corresponding to the sample vector whose similarity to the cluster center is less than the second threshold as the second type of training image for classification, which can improve the training image screening efficiency. .

As shown in FIG. 6, in one embodiment, the process of adjusting each center vector according to the distance between the sample vector and each center vector includes operations 602 to 604. among them:

Operation 602: Detect the distance between the cluster center and the classification vector in each classification.

The distance calculation formula may be a European-style distance calculation formula, a standard European-style distance calculation formula, a Manhattan distance calculation formula, a cosine distance calculation formula, and the like are not limited thereto. The electronic device can detect the distance between the cluster center and the classification vector in each classification according to the distance calculation formula. The larger the distance, the smaller the similarity between the classification vector and the cluster center of the classification, and the smaller the distance, the greater the similarity between the classification vector and the cluster center of the classification.

In one embodiment, the process of detecting the distance between the clustering center and the classification vector in each classification in the provided image processing method further includes: detecting a distance between the clustering center and the classification vector in each classification by using a European distance calculation formula.

其中，d _ij表示向量i与向量j之间的距离。x _ik表示向量i中的第k个特征值，x _jk表示向量j中的第k个特征值，n表示向量中特征值的个数。电子设备可以获取分类中的聚类中心与该分类的分类向量，通过上述欧式距离计算公式将聚类中心与分类向量中的各个特征值对应的代入距离计算公式中，获取聚类中心与分类向量的距离。 The electronic device can obtain the European distance calculation formula as

Where d _ij represents the distance between the vector i and the vector j. x _ik represents the k-th eigenvalue in the vector i, x _jk represents the k-th eigenvalue in the vector j, and n represents the number of eigenvalues in the vector. The electronic device can obtain the clustering center in the classification and the classification vector of the classification, and use the European-style distance calculation formula to substitute the clustering center and each feature value in the classification vector into the distance calculation formula to obtain the clustering center and the classification vector distance.

Operation 604: Use the training image corresponding to the classification vector whose distance is less than the preset distance as the first type of training image in the classification.

The preset distance can be set according to actual application requirements, which is not limited here. The electronic device uses the clustering center and the classification vector in each classification to use the training image corresponding to the classification vector whose distance is less than the preset distance as the first type of training image in the classification.

By detecting the distance between the clustering center and the classification vector in each classification, and using the training image corresponding to the classification vector whose distance is less than the preset distance as the first type of training image in the classification, the electronic device can obtain the correspondence of each classification from a large number of training images. The first type of training images can improve the filtering efficiency of training images.

In one embodiment, an image processing method is provided, and specific operations for implementing the method are as follows:

First, the electronic device inputs the training image to the neural network and obtains the sample vector output by the activation layer of the neural network. Neural networks are convolutional neural networks, deep neural networks, recurrent neural networks, and so on. A neural network generally includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the input of the image; the hidden layer is used to process the received image; the output layer is used to output the processing result of the image processing. The hidden layers of the neural network may include a convolutional layer, an activation layer, a pooling layer, and a fully connected layer. The electronic device can input the training image into the neural network, and the electronic device can obtain a sample vector composed of the feature values in the feature map according to the rules according to the feature map output by the activation layer in the neural network.

Optionally, the electronic device obtains a sample vector output by the penultimate activation layer in the neural network. For example, when the training image E is input into the neural network, the output of each activation layer in the neural network is out (1), out (2), ..., out (k), and k is the number of activation layers in the neural network. Then, the electronic device can obtain the vector output by the penultimate activation layer in the neural network, that is, out (k-1), as the sample vector corresponding to the training image E. The electronic device obtains the sample vector output by the penultimate activation layer in the neural network, which can improve the dimension of the sample vector and facilitate the discrimination of different training images.

Next, the electronic device performs cluster processing on the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification. The number of classifications refers to the number of scene classifications that the trained neural network can use for recognition. Scene classification can be scenery, beach, blue sky, green grass, snow, night, dark, backlight, sunrise / sunset, fireworks, spotlight, indoor, macro, text document, portrait, baby, cat, dog, food, etc. this. The electronic device performs cluster processing on the sample vector according to the required number of classifications, and can obtain a classification vector and a clustering center corresponding to each classification.

Optionally, the electronic device configures the number of classified center vectors according to the number of classifications, adjusts each center vector according to the distance between the sample vector and each center vector, and obtains the adjusted center vector as the clustering center for classification. The electronic device configures the number of classification center vectors according to the number of classifications, and each classification corresponds to a center vector. Specifically, the electronic device may randomly select the classified number of center vectors according to the classified number. The distance between the sample vector and each center vector can be detected using the distance formula. Specifically, the distance formula may be a Manhattan distance formula, a Euclidean distance formula, a relative entropy formula, and the like, and is not limited thereto. The electronic device obtains the adjusted center vector as the clustering center of the classification, and among the sample vectors, the sample vector with the smallest distance from the clustering center of the classification is the classification vector in the classification.

Optionally, the electronic device configures the number of classification center vectors according to the number of classifications; the classification corresponding to the center vector with the smallest distance from the sample vector is used as the classification of the sample vector; and the center vector is adjusted according to the distance between each sample vector and the center vector in the classification; According to the adjusted center vector, the classification with the center vector having the smallest distance from the sample vector as the sample vector is repeatedly performed, and the operation of adjusting the center vector according to the distance between each vector and the center vector in the classification is performed; when the number of adjustments exceeds a preset number of times, Obtain the final center vector as the clustering center for classification. The electronic device may also obtain the adjusted center vector as the clustering center when the distance between the adjusted center vector and the last adjusted center vector is smaller than the first distance value. In one embodiment, the electronic device may also obtain the center vector as the clustering center of the classification when the distance between all the sample vectors and the center vector in the classification is smaller than the second distance value.

Next, the electronic device detects the similarity between the clustering center and the classification vector in each classification, and uses the training image corresponding to the classification vector whose similarity is greater than the first threshold as the first type of training image for classification. The electronic device can obtain the clustering center of each classification and the classification vector in the classification, detect the similarity between each classification vector in the classification and the clustering center of the classification, and obtain the training image corresponding to the classification vector whose similarity is greater than the first threshold. This training image is used as the first type of training image in this classification.

Optionally, the electronic device detects the distance between the cluster center and the classification vector in each classification, and uses the training image corresponding to the classification vector whose distance is less than a preset distance as the first type of training image in the classification. The distance calculation formula may be a European-style distance calculation formula, a standard European-style distance calculation formula, a Manhattan distance calculation formula, a cosine distance calculation formula, and the like are not limited thereto. The preset distance can be set according to actual application requirements. The electronic device uses the clustering center and the classification vector in each classification to use the training image corresponding to the classification vector whose distance is less than the preset distance as the first type of training image in the classification.

Optionally, the electronic device detects the similarity between the cluster center of each classification and the sample vector, and uses the training image corresponding to the sample vector whose similarity to the cluster center is less than the second threshold as the second type of training image for classification. The second threshold can be set according to the requirements of the actual application. The electronic device detects the similarity between each classification vector in the classification and the clustering center of the classification, obtains a training image corresponding to the classification vector whose similarity is less than the second threshold, and uses the training image as the second type of training image in the classification.

Optionally, the electronic device uses a European-style distance calculation formula to detect the distance between the cluster center and the classification vector in each classification. The electronic device can obtain the Euclidean distance calculation formula, and obtain the cluster center in the classification and the classification vector of the classification, and use the Euclidean distance calculation formula to substitute the cluster center and each feature value in the classification vector into the distance calculation formula to obtain The distance between the cluster center and the classification vector.

It should be understood that although the operations in the flowcharts of FIGS. 2, 3, 5, and 6 are sequentially displayed in accordance with the directions of the arrows, these operations are not necessarily performed in the order indicated by the arrows. Unless explicitly stated in this article, there is no strict order in which these operations can be performed, and these operations can be performed in other orders. Moreover, at least a part of the operations in Figs. 2, 3, 5, and 6 may include multiple sub-operations or multiple stages. These sub-operations or stages are not necessarily performed at the same time, but may be performed at different times. These The execution order of the sub-operations or stages is not necessarily performed sequentially, but may be performed in turn or alternately with at least a part of the sub-operations or stages of other operations or other operations.

FIG. 7 is a structural block diagram of an image processing apparatus according to an embodiment. As shown in FIG. 7, an image processing apparatus includes a vector acquisition module 720, a cluster processing module 740, and an image determination module 760. among them:

A vector acquisition module 720 is configured to input a training image to a neural network, and obtain a sample vector output from an activation layer of the neural network.

A clustering processing module 740 is configured to perform clustering processing on the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification.

The image determination module 760 is configured to detect the similarity between the cluster center and the classification vector in each classification, and use the training image corresponding to the classification vector whose similarity is greater than the first threshold as the first type of training image for classification.

In one embodiment, the vector acquisition module 720 may be further configured to acquire a sample vector output by the penultimate activation layer in the neural network.

In one embodiment, the cluster processing module 740 may be further configured to configure the number of classification center vectors according to the number of classifications, adjust each center vector according to the distance between the sample vector and each center vector, and obtain the adjusted center vector as the classified cluster. center.

In one embodiment, the distance processing module 740 may be further configured to use the classification corresponding to the center vector with the smallest distance to the sample vector as the classification of the sample vector, adjust the center vector according to the distance between each sample vector and the center vector in the classification, and adjust according to the adjustment. The subsequent center vector repeatedly performs the classification using the center vector with the smallest distance from the sample vector as the sample vector, and adjusts the center vector according to the distance between each sample vector and the center vector in the classification. When the number of adjustments exceeds a preset number, the operation is obtained. The final center vector is used as the classification cluster center.

In one embodiment, the image determination module 760 may be further configured to detect the similarity between the cluster center of each classification and the sample vector, and use the training image corresponding to the sample vector whose similarity with the cluster center is less than the second threshold as the classification. The second type of training images.

In one embodiment, the image determination module 760 may be further configured to detect the distance between the cluster center and the classification vector in each classification, and use the training image corresponding to the classification vector whose distance is less than a preset distance as the first type of training image in the classification.

In one embodiment, the image determination module 760 may be further configured to detect the distance between the cluster center and the classification vector in each classification by using a European distance calculation formula.

The image processing apparatus provided in the embodiment of the present application obtains a sample vector output by an activation layer of a neural network by inputting a training image to a neural network, and performs cluster processing on the sample vector according to the number of classifications to obtain a clustering center and classification corresponding to each classification. Vector, to detect the similarity between the clustering center and the classification vector in each classification, and use the training image corresponding to the classification vector whose similarity is greater than the first threshold as the first type of training image for classification, which can improve the filtering efficiency of the training image.

The division of each module in the above image processing apparatus is for illustration only. In other embodiments, the image processing apparatus may be divided into different modules as needed to complete all or part of the functions of the above image processing apparatus.

For the specific limitation of the image processing device, refer to the foregoing limitation on the image processing method, and details are not described herein again. Each module in the image processing apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the hardware in or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

FIG. 8 is a schematic diagram of the internal structure of the electronic device in one embodiment. As shown in FIG. 8, the electronic device includes a processor, a memory, and a network interface connected through a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory is used to store data, programs, and the like. At least one computer program is stored on the memory, and the computer program can be executed by a processor to implement the image processing method applicable to the electronic device provided in the embodiments of the present application. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor to implement an image processing method provided by each of the following embodiments. The internal memory provides a cached operating environment for operating system computer programs in a non-volatile storage medium. The network interface may be an Ethernet card or a wireless network card, and is used to communicate with external electronic devices. The electronic device may be a mobile phone, a computer, a tablet computer, or a personal digital assistant or a wearable device.

The implementation of each module in the image processing apparatus provided in the embodiments of the present application may be in the form of a computer program. The computer program can be run on a terminal or a server. The program module constituted by the computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, the operations of the method described in the embodiments of the present application are implemented.

An embodiment of the present application further provides a computer-readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the operations of the image processing method.

A computer program product containing instructions that, when run on a computer, causes the computer to perform an image processing method.

An embodiment of the present application further provides an electronic device. The above electronic device includes an image processing circuit, and the image processing circuit may be implemented by using hardware and / or software components, and may include various processing units defining an ISP (Image Signal Processing) pipeline. FIG. 9 is a schematic diagram of an image processing circuit in one embodiment. As shown in FIG. 9, for ease of description, only aspects of the image processing technology related to the embodiments of the present application are shown.

As shown in FIG. 9, the image processing circuit includes an ISP processor 940 and a control logic 950. The image data captured by the imaging device 910 is first processed by the ISP processor 940, which analyzes the image data to capture image statistical information that can be used to determine and / or one or more control parameters of the imaging device 910. The imaging device 910 may include a camera having one or more lenses 912 and an image sensor 914. The image sensor 914 may include a color filter array (such as a Bayer filter). The image sensor 914 may obtain light intensity and wavelength information captured with each imaging pixel of the image sensor 914, and provide a set of raw data that may be processed by the ISP processor 940. Image data. The sensor 920 (such as a gyroscope) may provide parameters (such as image stabilization parameters) of the acquired image processing to the ISP processor 940 based on the interface type of the sensor 920. The sensor 920 interface may use a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the foregoing interfaces.

In addition, the image sensor 914 may also send the original image data to the sensor 920, and the sensor 920 may provide the original image data to the ISP processor 940 based on the interface type of the sensor 920, or the sensor 920 stores the original image data in the image memory 930.

The ISP processor 940 processes the original image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 9, 10, 12, or 14 bits, and the ISP processor 940 may perform one or more image processing operations on the original image data and collect statistical information about the image data. The image processing operations may be performed with the same or different bit depth accuracy.

The ISP processor 940 may also receive image data from the image memory 930. For example, the sensor 920 interface sends the original image data to the image memory 930, and the original image data in the image memory 930 is then provided to the ISP processor 940 for processing. The image memory 930 may be a part of a memory device, a storage device, or a separate dedicated memory in an electronic device, and may include a DMA (Direct Memory Access) feature.

When receiving raw image data from the image sensor 914 interface or from the sensor 920 interface or from the image memory 930, the ISP processor 940 may perform one or more image processing operations, such as time-domain filtering. The processed image data may be sent to the image memory 930 for further processing before being displayed. The ISP processor 940 receives processing data from the image memory 930 and performs image data processing on the processing data in the original domain and in the RGB and YCbCr color spaces. The image data processed by the ISP processor 940 may be output to the display 970 for viewing by the user and / or further processed by a graphics engine or a GPU (Graphics Processing Unit). In addition, the output of the ISP processor 940 can also be sent to the image memory 930, and the display 970 can read image data from the image memory 930. In one embodiment, the image memory 930 may be configured to implement one or more frame buffers. In addition, the output of the ISP processor 940 may be sent to an encoder / decoder 960 to encode / decode image data. The encoded image data can be saved and decompressed before being displayed on the display 970 device. The encoder / decoder 960 may be implemented by a CPU or a GPU or a coprocessor.

The statistical data determined by the ISP processor 940 may be sent to the control logic 950 unit. For example, the statistical data may include image information of the image sensor 914 such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, and lens 912 shading correction. The control logic 950 may include a processor and / or a microcontroller that executes one or more routines (such as firmware). The one or more routines may determine the control parameters of the imaging device 910 and the ISP processing according to the received statistical data. Parameters of the controller 940. For example, the control parameters of the imaging device 910 may include sensor 920 control parameters (such as gain, integration time for exposure control, image stabilization parameters, etc.), camera flash control parameters, lens 912 control parameters (such as focus distance for focusing or zooming), or these A combination of parameters. ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), and lens 912 shading correction parameters.

In this embodiment, the image processing method in FIG. 9 can be used to implement the foregoing image processing method.

Any reference to memory, storage, database, or other media used in this application may include non-volatile and / or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), which is used as external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR, SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several implementation manners of the present application, and their descriptions are more specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of this application patent shall be subject to the appended claims.

Claims (20)

An image processing method includes: Input the training image to the neural network, and obtain the sample vector output by the activation layer of the neural network; Clustering the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification; and The similarity between the clustering center and the classification vector in each classification is detected, and the training image corresponding to the classification vector whose similarity is greater than the first threshold is used as the first type of training image for the classification. The method according to claim 1, wherein the acquiring a sample vector output by an activation layer of a neural network comprises: Get the sample vector of the output of the penultimate activation layer in the neural network. The method according to claim 1, wherein the clustering the sample vectors according to the number of classifications comprises: Configuring a classification number of center vectors according to the classification number; Adjusting each center vector according to the distance between the sample vector and each center vector; and Obtain the adjusted center vector as the clustering center of classification. The method according to claim 3, wherein the adjusting each center vector according to a distance between the sample vector and each center vector comprises: Use the classification corresponding to the center vector with the smallest distance to the sample vector as the classification of the sample vector; Adjusting a center vector according to a distance between each sample vector and the center vector in the classification; Repeatedly performing the classification that uses the center vector with the smallest distance from the sample vector as the sample vector according to the adjusted center vector, and adjusts the center vector according to the distance between each sample vector and the center vector in the classification Operation; and When the number of adjustments exceeds a preset number, the final center vector is obtained as the clustering center of the classification. The method according to claim 1, further comprising: Detecting the similarity between the cluster center of each classification and the sample vector; and A training image corresponding to a sample vector whose similarity with the clustering center is less than a second threshold is used as the classified second-type training image. The method according to claim 1, further comprising: Detecting a distance between a clustering center and a classification vector in each of the classifications; and A training image corresponding to a classification vector whose distance is less than a preset distance is used as a first type training image in the classification. The method according to claim 6, wherein the detecting a distance between a clustering center and a classification vector in each classification comprises: The Euclidean distance calculation formula is used to detect the distance between the cluster center and the classification vector in each classification. An electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor causes the processor to perform the following operations: Input the training image to the neural network, and obtain the sample vector output by the activation layer of the neural network; Clustering the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification; and The similarity between the clustering center and the classification vector in each classification is detected, and the training image corresponding to the classification vector whose similarity is greater than the first threshold is used as the first type of training image for the classification. The electronic device according to claim 8, wherein when the processor executes the obtaining the sample vector output by the activation layer of the neural network, the processor further performs the following operations: Get the sample vector of the output of the penultimate activation layer in the neural network. The electronic device according to claim 8, wherein when the processor executes the clustering processing on the sample vectors according to the number of classifications, the processor further performs the following operations: Configuring a classification number of center vectors according to the classification number; Adjusting each center vector according to the distance between the sample vector and each center vector; and Obtain the adjusted center vector as the clustering center of classification. The electronic device according to claim 10, wherein when the processor executes the adjusting each center vector according to a distance between the sample vector and each center vector, the processor further performs the following operations: Use the classification corresponding to the center vector with the smallest distance to the sample vector as the classification of the sample vector; Adjusting a center vector according to a distance between each sample vector and the center vector in the classification; Repeatedly performing the classification that uses the center vector with the smallest distance from the sample vector as the sample vector according to the adjusted center vector, and adjusts the center vector's Operation; and When the number of adjustments exceeds a preset number, the final center vector is obtained as the clustering center of the classification. The electronic device according to claim 8, wherein when the computer program is executed by the processor, the processor further causes the processor to perform the following operations: Detecting the similarity between the cluster center of each classification and the sample vector; and A training image corresponding to a sample vector whose similarity with the clustering center is less than a second threshold is used as the classified second-type training image. The electronic device according to claim 12, wherein when the computer program is executed by the processor, the processor causes the processor to further perform the following operations: Detecting a distance between a clustering center and a classification vector in each of the classifications; and A training image corresponding to a classification vector whose distance is less than a preset distance is used as a first type training image in the classification. The electronic device according to claim 13, wherein when the processor executes the detecting a distance between a clustering center and a classification vector in each classification, the processor further performs the following operations: The Euclidean distance calculation formula is used to detect the distance between the cluster center and the classification vector in each classification. A computer-readable storage medium on which a computer program is stored is characterized in that, when the computer program is executed by a processor, the following operations are performed: Input the training image to the neural network, and obtain the sample vector output by the activation layer of the neural network; Clustering the sample vector according to the number of classifications to obtain a clustering center and a classification vector corresponding to each classification; and The similarity between the clustering center and the classification vector in each classification is detected, and the training image corresponding to the classification vector whose similarity is greater than the first threshold is used as the first type of training image for the classification. The computer-readable storage medium according to claim 15, wherein when the processor executes the obtaining the sample vector output by the activation layer of the neural network, it further executes: Get the sample vector of the output of the penultimate activation layer in the neural network. The computer-readable storage medium according to claim 15, wherein when the processor performs the clustering processing on the sample vectors according to the number of classifications, the processor further performs the following operations: Configuring a classification number of center vectors according to the classification number; Adjusting each center vector according to the distance between the sample vector and each center vector; and Obtain the adjusted center vector as the clustering center of classification. The computer-readable storage medium according to claim 15, wherein when the processor executes the adjusting each center vector according to the distance between the sample vector and each center vector, the processor further performs the following operations: Use the classification corresponding to the center vector with the smallest distance to the sample vector as the classification of the sample vector; Adjusting a center vector according to a distance between each sample vector and the center vector in the classification; Repeatedly performing the classification that uses the center vector with the smallest distance from the sample vector as the sample vector according to the adjusted center vector, and adjusts the center vector according to the distance between each sample vector and the center vector in the classification Operation; and When the number of adjustments exceeds a preset number, the final center vector is obtained as the clustering center of the classification. The computer-readable storage medium of claim 15, wherein the processor further performs the following operations: Detecting the similarity between the cluster center of each classification and the sample vector; and A training image corresponding to a sample vector whose similarity with the clustering center is less than a second threshold is used as the classified second-type training image. The computer-readable storage medium of claim 15, wherein the processor further performs the following operations: Detecting a distance between a clustering center and a classification vector in each of the classifications; and A training image corresponding to a classification vector whose distance is less than a preset distance is used as a first type training image in the classification.

Images