CN111476100A - Data processing method, device and storage medium based on principal component analysis - Google Patents

Abstract

The embodiments of the present invention relate to the field of software defect prediction, and disclose a data processing method, device and computer-readable storage medium based on principal component analysis. The method includes: performing dimension reduction processing on initial sample data to obtain a preset dimension sample data; obtain multiple features of the sample data, and calculate the correlation between each feature and a preset category, where the preset category is one of the multiple categories of the sample data; remove Among the plurality of features, the correlation degree is less than the preset correlation degree, and the remaining features are used as the identification features of the sample data. The data processing method, device and computer-readable storage medium based on principal component analysis provided by the present invention can remove redundant features in sample data to obtain sample data with high discrimination, thereby improving prediction efficiency.

Description

Data processing method, device and storage medium based on principal component analysis

technical field

Embodiments of the present invention relate to the field of data processing, and in particular, to a principal component analysis-based data processing method, device, and computer-readable storage medium.

Background technique

Information entropy is a measure of the amount of information required to eliminate uncertainty, that is, the amount of information an unknown event may contain. An event or a system, to be precise, is a random variable, which has a certain uncertainty. The uncertainty of some random variables is very high. To eliminate this uncertainty, it is necessary to introduce a lot of information. The measurement of this lot of information is expressed by "information entropy". The more information that needs to be introduced to eliminate uncertainty, the higher the information entropy, and vice versa. If a situation is very deterministic, almost no information needs to be introduced, so the information entropy is very low. According to the information entropy formula given by Shannon, for any random variable X, its information entropy is defined as follows, the unit is bit (bit): H(X)=-∑xεX[P(x)logP(x)]. The more equal the probability of various randomness in the system, the greater the information entropy, and vice versa.

The inventors found that there are at least the following problems in the prior art: analyzing the features of the sample data according to the above formula, many redundant features are obtained, resulting in a low prediction efficiency of the model trained by using the sample data.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a data processing method, device and computer-readable storage medium based on principal component analysis, which can remove redundant features in sample data, obtain sample data with high discrimination, thereby improving prediction efficiency.

In order to solve the above technical problems, embodiments of the present invention provide a data processing method based on principal component analysis, including:

Perform dimensionality reduction processing on the initial sample data to obtain sample data of preset dimensions; obtain multiple features of the sample data, and calculate the correlation between each feature and a preset category, wherein the preset category is the One of multiple categories that the sample data has; remove the features whose correlation is less than the preset correlation among the multiple features, and use the remaining features as the identification features of the sample data.

Embodiments of the present invention also provide a principal component analysis-based data processing device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores data that can be The instructions executed by the at least one processor are executed by the at least one processor, so that the at least one processor can execute the above-mentioned principal component analysis-based data processing method.

Embodiments of the present invention further provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned data processing method based on principal component analysis is implemented.

Compared with the prior art, the embodiment of the present invention obtains sample data with a preset dimension by performing dimension reduction processing on the initial sample data, so as to facilitate the calculation of the subsequent steps, reduce the calculation amount of the subsequent steps, and thereby improve the The efficiency of the data processing method; by acquiring multiple features of the sample data, and calculating the correlation between each feature and the preset category, since the preset category is one of the multiple categories of the sample data, through this In this way, it is possible to know which features of the multiple features of the sample data are redundant features according to the similarity; by removing the features whose correlation is less than the preset correlation among the multiple features, the remaining features are used as the sample data. The discriminative feature can obtain sample data with high discriminant, which makes the budget speed of the training model using the sample data faster, thereby improving the prediction efficiency.

In addition, after removing the features whose correlation degree is less than the preset correlation degree among the plurality of features, the method further includes: sorting the remaining features according to the order of the correlation degree from high to low; sorting the sorted remaining features Divided into N feature segments, wherein each feature segment includes M features, and N and M are both integers greater than 1; judging whether there are feature segments with M features greater than a preset threshold, when judging that there are, The feature with the smallest similarity in the feature segment is removed.

In addition, the performing dimensionality reduction processing on the initial sample data specifically includes: converting the initial sample data into a data matrix; calculating a covariance matrix of the data matrix, and performing eigendecomposition on the covariance matrix, Obtain the eigenvalues of the covariance matrix and the eigenvectors corresponding to the eigenvalues; obtain a projection matrix according to the eigenvalues and the eigenvectors, and reduce the dimension of the initial sample data to the projection matrix the corresponding dimension.

In addition, the obtaining the projection matrix according to the eigenvalues and the eigenvectors specifically includes: arranging the eigenvectors in rows from top to bottom into a matrix, wherein, the larger the eigenvalue corresponding to the eigenvector, the The eigenvector is located in the forward row of the matrix; the first k rows are taken to form the projection matrix, where k is an integer greater than 1.

In addition, before calculating the covariance matrix of the data matrix, the method further includes: performing zero-average processing on each row of the data matrix; and the calculating the covariance matrix of the data matrix specifically includes: calculating zero-average The covariance matrix of the processed data matrix.

In addition, the correlation between the feature and the preset category is calculated by the following formula: Si=[X ^T ×Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`]+[2×(IG (X|L))-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))]; where Si is the said similarity; X and Y are two different characteristics of the sample data; X` is the representation of X in different dimensions, Y` is the representation of Y in different dimensions; L is the preset category; [X ^T ×Y+ X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`] represents the discriminative correlation between the representation of X in different dimensions and the representation of Y in different dimensions; [2×(IG(X|L))- (H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))] represents the correlation between X, Y and the preset category respectively Spend.

In addition, the correlation between the feature and the preset category is calculated by the following formula: Si=[X ^T ×Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`]+λ×[2× (IG(X|L))-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))]; where Si is the similarity; X and Y are two different characteristics of the sample data; X` is the representation of X in different dimensions, Y` is the representation of Y in different dimensions; L is the preset category; [X ^T × Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`] represents the discriminative correlation between the representation of X in different dimensions and the representation of Y in different dimensions; [2×(IG(X|L) )-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))] represents the difference between X, Y and the preset category The correlation degree; λ is the equilibrium constant.

In addition, the initial sample data is image sample data.

Description of drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplifications do not constitute limitations of the embodiments, and elements with the same reference numerals in the drawings are denoted as similar elements, Unless otherwise stated, the figures in the accompanying drawings do not constitute a scale limitation.

1 is a flowchart of a data processing method based on principal component analysis provided according to a first embodiment of the present invention;

2 is a flowchart of a data processing method based on principal component analysis provided according to a second embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a data processing apparatus based on principal component analysis provided according to a third embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can appreciate that, in the various embodiments of the present invention, many technical details are set forth for the reader to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present invention can be realized.

The first embodiment of the present invention relates to a data processing method based on principal component analysis. The specific process is shown in Figure 1, including:

S101: Perform dimension reduction processing on the initial sample data to obtain sample data of a preset dimension.

Specifically, before the dimensionality reduction processing is performed on the initial sample data, the initial sample data to be processed will be acquired in advance. The dimensionality reduction processing on the initial sample data in this embodiment specifically includes: converting the initial sample data into a data matrix; calculating a covariance matrix of the data matrix, and characterizing the covariance matrix Decompose to obtain the eigenvalues of the covariance matrix and the eigenvectors corresponding to the eigenvalues; obtain a projection matrix according to the eigenvalues and the eigenvectors, and reduce the dimension of the initial sample data to the The dimension corresponding to the projection matrix. The obtaining of the projection matrix according to the eigenvalues and the eigenvectors specifically includes: arranging the eigenvectors into a matrix from top to bottom in rows, wherein the larger the eigenvalue corresponding to the eigenvectors, the greater the eigenvalues. The vector is located in the first row of the matrix; the first k rows are taken to form the projection matrix, where k is an integer greater than 1. For example, the matrix in which the eigenvectors are arranged in rows from top to bottom has 8 rows, indicating that there are 8 eigenvectors. If the eigenvalue corresponding to a certain eigenvector is the largest among the eigenvalues corresponding to the 8 eigenvectors, then the The vector is in the first row of the matrix, and so on.

It is worth mentioning that, in order to reduce the error of the data matrix and avoid the influence of the noise data in the data matrix on the final analysis result, before calculating the covariance matrix of the data matrix, it also includes: Each row is subjected to zero-average processing; the calculating the covariance matrix of the data matrix specifically includes: calculating the covariance matrix of the zero-averaged data matrix.

It can be understood that, in this embodiment, the dimension of the initial sample data is reduced by the PCA method. It should be noted that, in the prior art, multidimensional scaling analysis (MDS) is usually used to reduce the dimension of data samples. MDS is a dimensionality reduction method that mines the hidden structural information in the data by analyzing similar data. Usually, the similarity measure is represented by the Euclidean distance measure. Therefore, the purpose of the MDS algorithm is to map the data samples to a low-dimensional space while preserving the distance between the data samples as much as possible, thereby reducing the dimension of the samples. MDS is a classic method of theoretically maintaining Euclidean distance. MDS was first mainly used for data visualization. Since the low-dimensional representation obtained by MDS is centered at the origin, it can be said that the inner product is maintained. That is, the distance in the high-dimensional space is approximated by the inner product in the low-dimensional space. In the classic MDS method, Euclidean distance is generally used for the distance in high-dimensional space. Both Multidimensional Scaling Analysis (MDS) and Principal Component Analysis (PCA) are data dimensionality reduction techniques, but differ in the direction of optimization. The input of PCA is the original vector of the n-dimensional space, and the data is projected to the projection direction with the largest covariance, so the characteristics of the data are basically preserved during the dimensionality reduction process. The input to MDS is the pairwise distances between points, and the output of MDS is the projection of the distance-preserved points in 2D or 3D.

In short, PCA minimizes the sample dimension, which is the covariance of the data that can be preserved. MDS minimizes the sample dimension, which is the distance between data points that can be preserved. If the data covariance and the Euclidean distance between high-dimensional data points agree, they are the same; if the distance measure is different, then the two methods are different. Obviously, MDS has its limitations, and PCA can be used as an alternative method to make up for it, and the application range is wider, and the input of PCA is the original vector of n-dimensional space, so it simplifies the algorithm in terms of input compared to MDS, and reduces the complexity of the algorithm The most important thing is that the PCA method is widely used in data dimensionality reduction and preprocessing in terms of software defects, and the effect is better than MDS.

In order to facilitate understanding, the algorithm process of the PCA method is explained in detail below:

Suppose there are N image training samples, simply denoted as x _k ∈ X(k=1,...,N), X is the training sample data set, there are c classes of training samples, and each class has N _i training samples , the dimension of the column vector obtained by expanding the image matrix of each data is n. The average sample of all image training samples is expressed as:

The average sample representation of the i-th (i=1,...,c) class of training samples is as follows:

The specific process of the principal component analysis method is: first, read the database, expand each two-dimensional data image data read into a one-dimensional vector, and each type of image sample can select a certain number of images according to the generated random matrix. The images constitute the training sample set, and the rest constitute the test sample set. The next step is to calculate the generator matrix of the KL orthogonal transformation. The generator matrix can be represented by the overall divergence matrix S _T of the training samples, or by the inter-class divergence matrix S _B of the training samples. The divergence matrix is represented by the training set generated, represented here by the population divergence matrix S _T , defined as:

The generator matrix Σ can be expressed as: Σ=S _T S _T ^T

Then perform eigenvalue decomposition, calculate the eigenvalues and eigenvectors of the generated matrix Σ, sort the eigenvalues in descending order, retain the first m largest eigenvalues, and the eigenvectors corresponding to these m eigenvalues, Thus, the projection matrix from the high-dimensional space to the low-dimensional space is obtained, and the feature subspace is constructed. That is to say, the PCA method using the KL transform aims to find a set of optimal projection vectors that satisfy the criterion function:

又能够满足标准正交条件的最佳投影向量组w₁,w₂,...,w_m。而最佳投影矩阵就是通过最佳投影向量组表示的，即P＝[w₁,w₂,...,w_m]。The next step is to find the best projection vector, which is to maximize the unit vector w of the above criterion function. Its physical meaning is: in the direction represented by the projection vector w, the overall dispersion of the feature vector obtained after the image vector projection is the largest, That is, the distance between each sample of the image data and the average sample of the overall training samples is the largest. Because the optimal projection vector calculated above is the unit eigenvector corresponding to the largest eigenvalue of the overall divergence matrix _ST . In the case of a large number of sample categories, only a single optimal projection direction is not enough to fully represent the characteristics of all image samples. Therefore, it is necessary to find a set of functions that can both maximize the criterion

The best projection vector groups w ₁ , w ₂ , . . . , w _m can also satisfy the standard orthogonal conditions. The optimal projection matrix is represented by the optimal projection vector group, namely P=[w ₁ ,w ₂ ,...,w _m ].

Next, the training samples and the test samples are respectively projected into the feature subspace obtained above. After each data image is projected into the feature subspace, it will correspond to a point in the subspace. Similarly, any point in the feature subspace can also find a corresponding data image, and the points obtained by the projection of the data image in these feature subspaces are called "eigenfaces". As the name suggests, the "eigenface" method refers to a method of data recognition through K-L orthogonal transformation.

Finally, compare all the test image samples transformed into the feature subspace through the above vector projection with the training image samples to determine the category to which the data image samples to be identified belong. This is the classification of the test samples, and it is necessary to select the appropriate The classifier and dissimilarity test formula.

S102: Acquire multiple features of the sample data, and calculate the correlation between each feature and a preset category.

Specifically, in this embodiment, the correlation between the feature and the preset category can be calculated by the following formula: Si=[X ^T ×Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`] +[2×(IG(X|L))-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))] Wherein, Si is the degree of similarity; X, Y are two different characteristics of the sample data; X' is the representation of X in different dimensions, Y' is the representation of Y in different dimensions; L is the preset category; [X ^T ×Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`] represents the discriminative correlation between the representation of X in different dimensions and the representation of Y in different dimensions; [2×(IG( X|L))-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))] means that X and Y are Set the correlation between categories.

It is worth mentioning that, in order to balance the calculation items of the first part and the second part, a balance parameter λ needs to be added. Therefore, in this embodiment, the correlation between the feature and the preset category can also be calculated by the following formula:

Si=[X ^T ×Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`]+λ×[2×(IG(X|L))-(H(X)+H( L))]+[2×(IG(Y|L))-(H(Y)+H(L))]; wherein, Si is the similarity; X and Y are two different features of the sample data ; X` is the representation of X in different dimensions, Y` is the representation of Y in different dimensions; L is the preset category; [X ^T ×Y+X` <sup>T</sup> ×Y+X <sup>T</sup> ×Y`+X` <sup>T</sup> ×Y`] represents the discriminative correlation between the representation of X in different dimensions and the representation of Y in different dimensions; [2×(IG(X|L))-(H(X)+H(L))]+[2 ×(IG(Y|L))-(H(Y)+H(L))] represents the correlation between X, Y and the preset category respectively; λ is the equilibrium constant.

It can be understood that, compared with the information entropy formula in the prior art, the formula in this embodiment expands the calculation method of only the characteristics of the sample in the original method, so that the discrimination between any two different characteristics of the same sample is made. Feature selection category correlation is high. The first square brackets indicate the discriminative correlation between any two different features of the same sample and their representations in different dimensions, and through the constraint of the calculation, a more intuitive and higher correlation can be obtained. The original method cannot intuitively calculate the features with high correlation. It only considers the correlation of sample features, but ignores the more important feature correlation between sample features. Make full use of the relationship between different features of the same sample, including related features and redundant features. Obviously, if the value calculated by the first square bracket is larger, it means that the correlation between the sample features is higher, and it is possible to obtain Its related features, and vice versa, if the value calculated by the first square bracket is smaller, it means that the redundancy between the sample features is higher, and its redundant features can be effectively removed, making the sample features more discriminative. . The calculation of the following two square brackets indicates the similarity between the two different features of the same sample and the categorical variable. Similarly, the larger the value, the higher the similarity between the feature and the category, that is, the higher the correlation, and vice versa, The smaller the value, the lower the similarity between the feature and the category, that is, the lower the correlation.

S103: Remove features with a correlation degree less than a preset correlation degree among the plurality of features, and use the remaining features as the identification features of the sample data.

Specifically, the size of the preset correlation degree may be set according to actual requirements, which is not specifically limited in this embodiment. This embodiment is based on principal component analysis of sample data. The basic idea of principal component analysis is to extract the main features of the high-dimensional data space and keep most of the information of the original high-dimensional data, so that the high-dimensional data can be stored in a lower-dimensional data space. processed in feature space. The K-L transform is the basis of principal component analysis. It is an optimal orthogonal transformation based on the statistical characteristics of the target, and its purpose is to find a linear projection transformation through which the new feature components are orthogonal or irrelevant, and in order to make the energy of the data more concentrated , it is required that the error between the feature component after projection reconstruction and the original input sample in the sense of least mean square is the smallest. Thus, a low-dimensional approximate representation of the original sample is obtained, which can better compress the original data. Using K-L transform for data recognition, a classic eigenfaces method is proposed, which forms the basis of the subspace learning method. In short, it is to obtain a set of eigenface images from the input data training image through principal component analysis, and then given any data image, so that each data image can be linearly represented by this set of eigenface images, that is, A weighted linear combination of eigenface images obtained by computational principal component analysis.

The essence of PCA is to compute and diagonalize the covariance matrix. It can be assumed that all data images are in a linear low-dimensional space, and all data images in this low-dimensional space are linearly separable, and then the principal component analysis method is used for data feature recognition. The specific method is to Perform K-L transformation, transform the high-dimensional image input space to obtain a new set of orthonormal bases, screen the transformed orthonormal bases according to certain conditions, remove some redundant vectors, and retain those feature identification A powerful vector is used to generate a low-dimensional data subspace, that is, the eigenface subspace of the data. The key to using the principal component analysis method to reduce the spatial dimension of the input data is to find the projection method that can best represent the original data, in order to "reduce noise" and eliminate the "redundant" dimension, so that the original input data can be guaranteed while reducing the dimension. The most important features are not lost. In the covariance matrix, only those dimensions with relatively large energy (eigenvalues) need to be selected, and the remaining relatively low dimensions are discarded, so that those important feature information in the input image data can be retained, and those that are not conducive to data identification are discarded. other parts.

In order to facilitate understanding, a specific example of how to process sample data in this embodiment is described below:

Input: training sample set: X=[X1, X2,..., Xc], where Xi=(F1, F2,..., Fm, L), k<m, i=1...m.

The dimension of PCA data dimensionality reduction: k

Correlation threshold (preset correlation): β

1) The original data is formed into a data matrix by columns, and each two-dimensional data image data read in is expanded into a one-dimensional vector.

2) Zero-mean each row of the data matrix (representing an attribute field), that is, subtract the mean of this row.

3) Find the covariance matrix.

4) Carry out eigenvalue decomposition to obtain the eigenvalues of the covariance matrix and the corresponding eigenvectors.

5) Arrange the eigenvectors into a matrix from top to bottom according to the size of the corresponding eigenvalues, and take the first k rows to form a sample projection matrix.

6) Reduce the dimension of the data to the dimension corresponding to the projection matrix, that is, k, X'=PX is the data after dimension reduction to k dimension. The sample set after dimensionality reduction is obtained as X'=[X'1, X'2, ..., X'c], where X'i=(F1, F2, ..., Fk, L), k<m, i =1...m.

7) Let i=1 to k, j=1 to k (i≠j) loop, and calculate Si=ISU(Fi, Fi', Fj, Fj', L).

8) Sort Si in descending order.

9) Take the first g features in the sequence as the features of the new sample, and obtain the sample set X"=[X"1, X"2,...,X"c], where X"i=(F1, F2,... , Fg, L), g<k, i=1...m.

10) Perform correlation analysis on each pair of features from back to front, remove the specified features greater than β, and obtain the final sample Y.

Output: sample set Y.

Compared with the prior art, the embodiment of the present invention obtains sample data with a preset dimension by performing dimension reduction processing on the initial sample data, so as to facilitate the calculation of the subsequent steps, reduce the calculation amount of the subsequent steps, and thereby improve the The efficiency of the data processing method; by acquiring multiple features of the sample data, and calculating the correlation between each feature and the preset category, since the preset category is one of the multiple categories of the sample data, through this In this way, it is possible to know which features of the multiple features of the sample data are redundant features according to the similarity; by removing the features whose correlation is less than the preset correlation among the multiple features, the remaining features are used as the sample data. The discriminative feature can obtain sample data with high discriminant, which makes the budget speed of the training model using the sample data faster, thereby improving the prediction efficiency.

The second embodiment of the present invention relates to a data processing method based on principal component analysis. The second embodiment is further improved on the basis of the first embodiment. The specific improvement is that: in the second embodiment , after removing the features whose correlation degree is less than the preset correlation degree among the plurality of features, further comprising: sorting the remaining features according to the order of the correlation degree from high to low; dividing the sorted remaining features into is N feature segments, wherein each feature segment includes M features, and N and M are both integers greater than 1; it is judged whether there are feature segments with M features that are all greater than the preset threshold, and when it is judged to exist, remove the The feature with the smallest similarity in the feature segment. In this way, redundant features in the sample data can be further reduced, so that the prediction efficiency is further improved.

The specific process of this embodiment is shown in Figure 2, including:

S201: Perform dimensionality reduction processing on the initial sample data to obtain sample data of preset dimensions.

S202: Acquire multiple features of the sample data, and calculate the correlation between each feature and a preset category.

S203: Remove features whose correlation degree is less than a preset correlation degree among the multiple features.

S204 : Sort the multiple features after removing the features whose correlation is less than the preset correlation in descending order of the correlation, and divide the sorted remaining features into N feature segments.

S205: Determine whether there are feature segments with M features that are all greater than a preset threshold, and when it is determined that there are feature segments with the smallest similarity in the feature segments.

For the above steps S204 to S205, specifically, a threshold correlation method is used to remove redundant features of the sample data. The threshold correlation method uses the correlation between features to identify redundant features. In actual software metrics, there is a nonlinear relationship, so ISU is still selected to calculate the correlation between a pair of features. The threshold correlation method uses pre- The set β (that is, the preset threshold) is used as the critical value of the correlation. After removing the features whose correlation is less than the preset correlation among the multiple features, the correlation analysis is performed on the remaining features from the back to the front. A pair of features removes the later features from the sample set, and so on. The reason why the correlation analysis is performed from the back to the front is that the discrimination of the features whose correlation degree is less than the preset correlation degree among the multiple features is removed. When two features with a correlation greater than the β value are encountered, the less discriminative features can be preferentially removed, thereby retaining the more discriminative features.

S206: Use the remaining features as the discriminating features of the sample data.

Steps S201 to S203 and S206 of the present embodiment are similar to steps S101 to S103 of the first embodiment, and are not repeated here to avoid repetition.

Compared with the prior art, the embodiment of the present invention obtains sample data with a preset dimension by performing dimension reduction processing on the initial sample data, so as to facilitate the calculation of the subsequent steps, reduce the calculation amount of the subsequent steps, and thereby improve the The efficiency of the data processing method; by acquiring multiple features of the sample data, and calculating the correlation between each feature and the preset category, since the preset category is one of the multiple categories of the sample data, through this In this way, it is possible to know which features of the multiple features of the sample data are redundant features according to the similarity; by removing the features whose correlation is less than the preset correlation among the multiple features, the remaining features are used as the sample data. The discriminative feature can obtain sample data with high discriminant, which makes the budget speed of the training model using the sample data faster, thereby improving the prediction efficiency.

The third embodiment of the present invention relates to a data processing device based on principal component analysis, as shown in FIG. 3 , including:

at least one processor 301; and,

a memory 302 in communication with the at least one processor 301; wherein,

The memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301, so that the at least one processor 301 can execute the above-mentioned principal component analysis-based data processing method.

The memory 302 and the processor 301 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 301 and various circuits of the memory 302 together. The bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The data processed by the processor 301 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor 301 .

Processor 301 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory 302 may be used to store data used by the processor 301 when performing operations.

A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The above method embodiments are implemented when the computer program is executed by the processor.

That is, those skilled in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device ( It may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present invention. scope.

Claims (10)

1. A data processing method based on principal component analysis is characterized by comprising the following steps:

performing dimensionality reduction on the initial sample data to obtain sample data with preset dimensionality;

obtaining a plurality of characteristics of the sample data, and calculating the correlation degree of each characteristic and a preset category, wherein the preset category is one of a plurality of categories of the sample data;

and removing the features of which the correlation degree is smaller than the preset correlation degree from the plurality of features, and taking the residual features as the identification features of the sample data.

2. The principal component analysis-based data processing method according to claim 1, further comprising, after removing features of the plurality of features for which the degree of correlation is less than a preset degree of correlation:

sorting the residual features according to the sequence of the relevance from high to low;

dividing the sorted residual features into N feature segments, wherein each feature segment comprises M features, and N, M are integers greater than 1;

and judging whether the M characteristic sections with the characteristics larger than a preset threshold exist or not, and removing the characteristic with the minimum similarity in the characteristic sections when judging that the M characteristic sections exist.

3. The principal component analysis-based data processing method according to claim 1, wherein the performing dimensionality reduction processing on the initial sample data specifically includes:

converting the initial sample data into a data matrix;

calculating a covariance matrix of the data matrix, and performing characteristic decomposition on the covariance matrix to obtain an eigenvalue of the covariance matrix and an eigenvector corresponding to the eigenvalue;

and obtaining a projection matrix according to the eigenvalue and the eigenvector, and reducing the dimensionality of the initial sample data to the dimensionality corresponding to the projection matrix.

4. The principal component analysis-based data processing method according to claim 3, wherein obtaining a projection matrix from the eigenvalues and the eigenvectors specifically comprises:

arranging the eigenvectors into a matrix from top to bottom, wherein the larger the eigenvalue corresponding to the eigenvector is, the more the eigenvector is located in the front row of the matrix;

and taking the first k rows to form the projection matrix, wherein k is an integer larger than 1.

5. The principal component analysis-based data processing method according to claim 3 or 4, further comprising, before calculating the covariance matrix of the data matrix:

carrying out zero equalization processing on each line of the data matrix;

the calculating the covariance matrix of the data matrix specifically includes:

and calculating a covariance matrix of the data matrix after zero averaging processing.

6. The principal component analysis-based data processing method according to claim 1, wherein the degree of correlation of a feature with the preset category is calculated by the following formula:

Si＝[X^T×Y+X`<sup>T</sup>×Y+X<sup>T</sup>×Y`+X`<sup>T</sup>×Y`]+[2×(IG(X|L))-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))]；

wherein Si is the similarity, X, Y is two different characteristics of sample data, X 'is the representation of X in different dimensions, Y' is the representation of Y in different dimensions, L is the preset category^T×Y+X`<sup>T</sup>×Y+X<sup>T</sup>×Y`+X`<sup>T</sup>×Y`]Representing the discriminatory relevance of X in different dimensions to Y in different dimensions, [2 × (IG (X | L)) - (H (X) + H (L) ]]+[2×(IG(Y|L))-(H(Y)+H(L))]The correlation between the representations X, Y and the preset categories, respectively.

7. The principal component analysis-based data processing method according to claim 1, wherein the degree of correlation of a feature with the preset category is calculated by the following formula:

Si＝[X^T×Y+X`<sup>T</sup>×Y+X<sup>T</sup>×Y`+X`<sup>T</sup>×Y`]+λ×[2×(IG(X|L))-(H(X)+H(L))]+[2×(IG(Y|L))-(H(Y)+H(L))]；

wherein Si is the similarity, X, Y is two different characteristics of sample data, X 'is the representation of X in different dimensions, Y' is the representation of Y in different dimensions, L is the preset category^T×Y+X`<sup>T</sup>×Y+X<sup>T</sup>×Y`+X`<sup>T</sup>×Y`]Representing the discriminatory relevance of X in different dimensions to Y in different dimensions, [2 × (IG (X | L)) - (H (X) + H (L) ]]+[2×(IG(Y|L))-(H(Y)+H(L))]Representing X, Y a degree of correlation between each and a preset category; λ is the equilibrium constant.

8. The principal component analysis-based data processing method according to claim 1, wherein the initial sample data is image sample data.

9. A data processing apparatus based on principal component analysis, comprising: at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a principal component analysis-based data processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the principal component analysis-based data processing method according to any one of claims 1 to 8.

Images