JP6840627B2 - Hyperparameter evaluation method, computer and program - Google Patents

Description

The present invention relates to an apparatus that performs data analysis using machine learning.

Machine learning technology is attracting attention as a technology that creates new social value for the analysis of data accumulated in various fields such as healthcare, finance, and industry. Machine learning has the types of algorithms such as support vector machine and deep learning, and the parameters required to determine the model in each algorithm, and is called hyperparameters.

There are more than tens of thousands of combinations of hyperparameters. When applying machine learning to the problem you want to analyze, you need to try a number of hyperparameter combinations and find the one with the highest accuracy. In addition, since the optimum combination of hyperparameters differs depending on the problem to be analyzed, it is necessary to perform it every time the problem or data changes.

In addition, since it is possible to shorten the search time by estimating the search area from the effect of each hyperparameter, it is realistic for an expert to search for hyperparameters. However, even if an expert searches, it may take about half a year to find the optimum hyperparameters.

Therefore, if the target accuracy cannot be achieved, six months after the start of the search, the factors that failed to achieve the target accuracy are considered, and the conclusion is obtained that the problem setting and data quality are inappropriate. It will be.

In addition, even if it is desired to add data to the data each time the data accumulated daily increases, the model corresponding to the data to be added will be completed in half a year. Thus, when applying machine learning to the real world, high expertise and complexity of hyperparameter search are one of the problems.

As a prior art, a cheat sheet is known in which an expert systematizes the know-how of hyperparameter search as a flow (Non-Patent Document 1). It is thought that the type of algorithm can be determined by using the cheat sheet. In addition, as another prior art, a service has been commercialized that automatically and comprehensively tries a combination of hyperparameters prepared in advance and presents the hyperparameters with the highest prediction accuracy (non-patented). Document 2).

“Scikit-learn cheat sheet”, [online], [Search on May 8, 2017], Internet <URL: http://scikit-learn.org/stable/tutorial/machine_learning_map/> “Sap predictive analytics whitepaper”, [online], [Search on May 8, 2017], Internet <https://www.sap.com/japan/documents/2016/02/c4d70a6c-617c-0010-82c7- eda71af511fa.html>

When applying machine learning to the problem you want to analyze, you need to try a number of hyperparameter combinations and find the one with the highest accuracy. The above work requires about half a year and specialists.

When trying to combine hyperparameters automatically and comprehensively, it takes time as well, but no specialist is required. In addition, if the data to be analyzed cannot be taken out, it is necessary to bring the analysis computer to the facility that stores the data to be analyzed. Conventionally, the use of a large-scale analysis computer has been a prerequisite for machine learning hyperparameter search, but there is a possibility that the analysis computer cannot be brought into the facility. Moreover, the time required when using an existing computer may be even longer.

An object of the present invention is to estimate the predicted performance expected for each combination of hyperparameters from the characteristics of data without the need for an expert and a high-performance computer for analysis in the process of hyperparameter search. The purpose is to provide a computer that can quickly present a combination of predictive performance and hyperparameters that are expected to be achievable.

The present invention is a hyperparameter evaluation method in which a computer having a processor and a memory evaluates hyperparameters applicable to the data to be analyzed, and the computer uses preset preset data and a plurality of hyperparameters. The first step of generating the next feature quantity generation unit and the accuracy prediction unit, the second step of the computer accepting the analysis target data, and the secondary feature quantity generation unit include the analysis target data and a plurality. It includes a third step of calculating a secondary feature amount by inputting a hyperparameter, and a fourth step of calculating an expected prediction accuracy for each hyperparameter by the accuracy prediction unit using the secondary feature amount as an input. ..

According to the present invention, it is possible to quickly present a combination of hyperparameters predicted to have the highest accuracy and the highest prediction accuracy for the data to be analyzed. As a result, it can be expected that the hyperparameter search of machine learning does not require a long period of time such as half a year, and does not require a specialist and a high-performance computer for analysis.

It is a block diagram which shows Example 1 of this invention and shows an example of the structure of an abnormality detection apparatus. It is a block diagram which shows Example 1 of this invention and shows an example of the structure of software. FIG. 5 is a flowchart showing Example 1 of the present invention and showing an example of processing of an expected prediction accuracy predictor. It is explanatory drawing which shows Example 1 of this invention and shows an example of processing in a secondary feature quantity generator and accuracy predictor. FIG. 5 is a flowchart showing Example 1 of the present invention and showing an example of processing performed by the learner. It is a flowchart which shows Example 1 of this invention and shows an example of the process which saves a result. It is a figure which shows Example 1 of this invention and shows an example of screen display. It is explanatory drawing which shows Example 1 of this invention and shows an example of the generation of a secondary feature quantity generator and an accuracy predictor. It is explanatory drawing which shows Example 2 of this invention and shows an example of the function of the abnormality detection apparatus. It is a flowchart which shows Example 3 of this invention and shows an example of processing. FIG. 3 is a flowchart showing Example 3 of the present invention and showing an example of processing of a data analysis method. It is explanatory drawing which shows Example 4 of this invention and shows an example of the function of the abnormality detection apparatus.

The first embodiment of the present invention will be described with reference to the drawings. Hereinafter, in all the drawings for explaining the embodiment of the present invention, those having basically the same function are designated by the same reference numerals, and the repeated description thereof will be omitted.

First, the abnormality detection device according to the first embodiment of the present invention will be described. The abnormality detection device 100 of the first embodiment shows an example in which a model for data analysis is generated by machine learning, and data having an abnormality is detected by the generated model.

FIG. 1 is a block diagram showing an example of the hardware configuration of the abnormality detection device 100 of the present embodiment. The abnormality detection device 100 of the first embodiment includes a CPU 102, a memory 103, an input / output interface 104, a communication device 105, and a drive device 106.

These are interconnected by a data bus 101. The CPU 102 includes a control device and an arithmetic unit, and is a central processing unit that controls the abnormality detection device 100, performs arithmetic operations, and transfers information. The memory 103 is a storage device capable of holding digital data to be processed by the abnormality detection device 100 for a certain period of time.

The input / output interface 104 is an interface used for inputting / outputting information to / from a device connected to the outside of the device. An input device 107 such as a keyboard or mouse and an output device 108 such as a display can be connected to the input / output interface 104.

The communication device 105 is a device that enables connection to a network such as the Internet by connecting a cable that connects to a network such as the Internet to the communication device 105. The drive device 106 is a device that includes a blank disk medium or a storage medium 109 such as an HDD on which information is written or writable, and reads the written information and writes the information.

The memory 103 stores in advance a program 200 for arithmetic processing of the CPU 102 and various data necessary for realizing the function of the abnormality detection device 100 of the first embodiment. By executing the program 200 stored in the memory 103, the CPU 102 performs various processes for realizing the functions of the abnormality detection device 100.

The program 200 executed by the CPU 102 may be stored in the storage medium 109 connected to the drive device 106, and the program may be read and stored in the memory 103.

FIG. 2 is a block diagram showing an example of a function implemented by executing the operation processing program 200 stored in the memory 103 or the storage medium 109. The expected prediction accuracy predictor 210 includes a secondary feature amount generator 211 and an accuracy predictor 212. The secondary feature amount generator 211 may be used as a secondary feature amount generation module. Further, the accuracy predictor 212 may be an accuracy prediction module.

The secondary feature amount generator 211 includes a function of generating a secondary feature amount using a primary feature amount composed of a combination of analysis target data and hyperparameters. The accuracy predictor 212 has a function of estimating the expected prediction accuracy of hyperparameters by inputting the secondary feature amount generated by the secondary feature amount generator 211. As described above, the hyperparameters are information including the types of machine learning algorithms and the parameters required to determine the model in each algorithm.

Expected prediction accuracy can be expressed by the probability of being able to correctly distinguish between patients and non-patients when the hyperparameters and data to be analyzed are applied to machine learning and the accuracy of the generated model is evaluated. .. As the machine learning model, a preset model is used.

The learner 220 includes a function of performing learning by a combination of arbitrary hyperparameters and generating a model for data analysis. The result display 230 includes a function of displaying the operation by the user interface, the result of the expected prediction accuracy predictor 210, and the result of the learner 220.

Each functional unit of the secondary feature amount generator 211, the accuracy predictor 212, the learner 220, and the result display 230 is loaded into the memory 103 as a program 200. In the first embodiment, the learning device 220 is realized by software, but it may be realized by hardware.

The CPU 102 operates as a functional unit that provides a predetermined function by processing according to the program of each functional unit. For example, the CPU 102 functions as a learner 220 by processing according to a learning program. The same applies to other programs. Further, the CPU 102 also operates as a functional unit that provides each function of a plurality of processes executed by each program. A computer and a computer system are devices and systems including these functional parts.

Information such as programs and tables that realize each function of the abnormality detection device 100 can be obtained from a drive device 106, a non-volatile semiconductor memory, a hard disk drive, a storage device such as an SSD (Solid State Drive), or an IC card, SD card, DVD. It can be stored in a non-temporary data storage medium that can be read by a computer such as.

Next, the flow of processing when the program 200 of the first embodiment is executed will be described. FIG. 3 is a flowchart showing an example of processing performed by the abnormality detection device 100 of the first embodiment. The abnormality detection device 100 instantly (or quickly) presents the combination of hyperparameters predicted to be the highest accuracy and the highest prediction accuracy to the data to be analyzed by executing the program.

The abnormality detection device 100 makes the expected prediction accuracy predictor 210 function and reads the analysis target data into the memory 103 from the storage medium 109 connected to the drive device 106 (step S301). The abnormality detection device 100 receives the designation of the analysis target data from the input device 107 or the like, and reads the data from the storage medium 109 into the memory 103.

Subsequently, the expected prediction accuracy predictor 210 of the abnormality detection device 100 generates a secondary feature amount by the secondary feature amount generator 211 using the primary feature amount composed of the analysis target data and the hyperparameter set (step). S302).

Here, the hyperparameter set means a combination of hyperparameters. For example, the type of algorithm for analysis is deep learning, the number of layers is 10, the number of units (neurons) is 500, and so on.

The hyperparameter set for evaluating the prediction accuracy of hyperparameters applicable to the data to be analyzed is composed of a plurality of hyperparameters prepared in advance. Alternatively, a plurality of hyperparameters specified by an operator or the like via the input device 107 may be accepted to form a hyperparameter set.

As for the primary feature quantity, a plurality of predetermined hyperparameter sets such as the analysis target data and the hyperparameter set A and the analysis target data and the hyperparameter set B are automatically combined with the analysis target data. It is a data set. In the first embodiment, it is assumed that the primary feature amount is generated in advance.

Next, the expected prediction accuracy predictor 210 predicts the expected prediction accuracy by the accuracy predictor 212 using the secondary features generated as described later (step S303). Finally, the result display 230 outputs the hyperparameter set and the expected prediction accuracy of the top several (preset preset rank threshold values or more) of the predicted expected prediction accuracy having the highest expected prediction accuracy. Displayed on 108, the process ends (step S304).

FIG. 4 is a schematic diagram for explaining the process of generating the secondary feature amount in step S302 and predicting the expected prediction accuracy in step S303. As for the primary feature quantity, as described above, a plurality of samples are generated as a combination of the data to be analyzed and one hyperparameter set.

The following description will be given when there are 10 hyperparameter sets. For the primary feature amount of 10 samples, the secondary feature amount is generated by the secondary feature amount generation in step S302, the expected prediction accuracy is predicted by the expected prediction accuracy prediction in step S303, and for each of the 10 samples. The expected prediction accuracy is presented one by one.

As shown in step S304, the ten presented expected prediction accuracy are rearranged in descending order, and the hyperparameter set and the expected prediction accuracy are displayed in descending order of the expected prediction accuracy, and the process ends.

In the first embodiment, the hyperparameter set used for the feature quantity is defined in advance, but the hyperparameter set may be added and determined by the operator. Further, among the expected prediction accuracy, the hyperparameter set and the prediction accuracy of the top several of the highest prediction accuracy are displayed to end the processing, but the processing is not limited to this.

For example, the result display 230 displays the result by the expected prediction accuracy predictor 210 on the output device 108, and the operator selects one hyperparameter set from the displayed ones, and the result is based on the selected hyperparameter set. There may be a function of training and generating a model for data analysis.

Further, the operator may not select an arbitrary parameter set, and may automatically perform learning on only the top one or all the top five parameters. Furthermore, the expected prediction accuracy for all primary features and their hyperparameter sets may be specified or stored in a specific memory space, or only specific data specified by the operator may be specified or stored in a specific memory space. You may save it.

Subsequently, among the hyperparameter sets output by the expected prediction accuracy predictor 210, learning by one or a plurality of hyperparameter sets specified by the operator is performed by the learner 220 to generate a model for data analysis.

FIG. 5 is a flowchart showing an example of processing performed by the learner 220. The learner 220 performs learning using the designated hyperparameters and generates a model for data analysis. This process is started based on an instruction from the operator or the like. A well-known or known technique can be applied to the model generation method.

First, the learner 220 reads out the analysis target data designated by the input device 107 or the like from the memory 103 (step S501). Subsequently, the learner 220 determines whether or not to read the hyperparameters stored in the memory 103 in advance (step S502).

The hyperparameters stored in the memory 103 in advance are, for example, the hyperparameters whose expected prediction accuracy is displayed in step S304 of FIG.

The determination in step S502 is performed based on the response received from the input device 107 by displaying a button on the output device 108 whether or not the learner 220 uses the hyperparameter set in the memory 103. When the hyperparameter set in the memory 103 is used, the process proceeds to step S504, and if not, the process proceeds to step S503.

In step S503, the learner 220 outputs the user interface to the output device 108, and receives the hyperparameter set to be learned from the input device 107. On the other hand, when reading from the memory 103, the learner 220 reads the hyperparameter set stored in the memory 103 in order to select the hyperparameter set to be learned (step S504).

The learner 220 selects the hyperparameter set to be learned from the hyperparameter set read in step S504 (step S505). At this time, only one hyperparameter set may be selected, or a plurality of hyperparameter sets may be selected. For the selection of the hyperparameter set, the learner 220 may accept the operator's command from the input device 107.

Subsequently, the learner 220 performs predetermined learning using the selected hyperparameter set and the analysis target data read in step S501 to generate an analysis model (step S506).

The learner 220 uses, for example, only 50% of the data to be analyzed as the data used for learning, and uses the remaining 50% for evaluation of the generated model to evaluate the accuracy by 10-fold cross-validation or the like. .. By performing cross-validation using the remaining 50% or the like of the data to be analyzed, it is possible to suppress overfitting of the learning result.

Finally, the learner 220 displays the prediction accuracy of the generated model on the output device 108 as a result, and ends (step S507). In step S507, the model or hyperparameter set generated by learning in descending order of prediction accuracy as described above can be displayed on the output device 108.

By executing the program in the first embodiment, it is possible to instantly present the hyperparameter set predicted to be the highest accuracy and the highest accuracy to the data to be analyzed.

Next, saving of the processing result and saving of the result in the model generation process for data analysis by learning with one or a plurality of hyperparameter sets specified by the operator will be described.

FIG. 6 is a flowchart showing an example of the process of saving the result. The saving process of the first embodiment is started by the accuracy predictor 212 and the learning device 220 by transmitting a signal such as pressing the save button by the operator through the user interface or the like to the CPU 102. The case of the learner 220 will be described.

First, the learner 220 selects a storage target and a memory space (step S601). Subsequently, the learner 220 saves the selected target in the selected memory space or a specific memory space (step S602). Finally, the learner 220 displays a message to the effect that the saving is successful on the output device 108, and ends the process (step S603).

FIG. 7 is a diagram showing an example of a screen displayed on a display or the like which is an output device 108 in the above processing. By executing the result display 230 in the first embodiment, the display screen 700 is displayed on the output device 108.

The operator of the abnormality detection device 100 can select the data to be analyzed by operating the input device 107 and pressing the Import file button 701 to select a file from the file selection screen (not shown).

The name of the selected analysis target data is output to the Import file name field 705. After that, when the operator presses the Branch prediction button 702, the analysis target data is input to the expected prediction accuracy predictor 210, processing as represented by the flowchart of FIG. 3 is performed, and the result display 230 displays the result. The result is displayed in area 708.

The result of the expected prediction accuracy predictor 210 displayed in the result display area 708 is the hyper parameter set and the expected prediction accuracy in the top few of the predicted expected prediction accuracy with the highest prediction accuracy, and the expected prediction accuracy. It is conceivable that the items with the highest value are displayed in a table in order.

The operator can select any hyperparameter set by the radio button 709 (or check box or the like) displayed in the result display area 708. When the operator presses the Save button 704 after selecting the hyperparameter set, a screen on which the memory space to be saved can be specified is displayed.

When the operator specifies a memory space via the input device 107 and presses the execute button, the selected expected prediction accuracy and its hyperparameter set are saved in the designated memory space. The saved destination memory space is output to an Export file name 706 or the like.

Subsequently, when the operator wants to select the hyperparameter set to be learned, the hyperparameter set can be selected by pressing the Import file button 701 and selecting from the file selection screen (not shown).

When the Import file button 701 is pressed, there may be a function of first selecting whether the file to be selected is the data to be analyzed or the hyperparameter set. The name of the selected hyperparameter set is output to Import hyper-parameter name 707.

It is conceivable that the hyperparameter set is displayed in the result display area 708 in a tabular format or the like, and the hyperparameter set to be trained can be selected by the radio button 709 or the check box displayed accompanying the table. Be done.

Subsequently, when the operator presses the Training button 703 via the input device 107, the selected hyperparameter set and the analysis target data described in the Import file name column 705 are used in the flowchart of FIG. Processing as represented is performed, and the evaluation result of the model is displayed in the result display area 708.

When it is desired to save the evaluation result of the model, pressing the Save button 704 via the input device 107 displays a screen on which the memory space to be saved can be specified. When the operator specifies a memory space via the input device 107 and presses the execute button, the model evaluation result and a parameter set including the hyperparameters thereof are saved in the designated memory space. The saved destination memory space is output to an Export file name 706 or the like.

In this way, the expected prediction accuracy predictor 210 composed of the secondary feature generator 211 and the accuracy predictor 212 does not require a long period of time such as half a year, and is an expert and high-performance computer for analysis. Is unnecessary, and the model generated by the learner 220 can be applied.

By applying the above model to the abnormality detection device 100, it is possible to detect an abnormality with high prediction accuracy using the latest data. Examples of abnormalities detected by the abnormality detection device 100 include detection of diseases that can be read from images acquired by a medical diagnostic imaging device, detection of diseases using a gene expression data sequence acquired by a genetic testing device, and detection of diseases. , Detection of diseases using component analysis results obtained by biochemical testing equipment, detection of drug candidate components using drug activity values obtained by experiments conducted in drug development, etc. can be considered.

In areas other than healthcare-related areas, abnormal behavior is detected using data acquired from sensors attached to devices and people, and annual income and debt submitted by card applicants during credit card examinations. Exception detection using data such as presence / absence can be considered.

FIG. 8 is an explanatory diagram of an example of generation of the secondary feature amount generator 211 and the accuracy predictor 212. In the following, the generation of the secondary feature amount generator 211 that generates the secondary feature amount and the accuracy predictor 212 that predicts the expected prediction accuracy will be described with reference to FIG. The secondary feature quantity generator 211 and the accuracy predictor 212 are generated in advance before the expected prediction accuracy is presented using the analysis target data.

First, the generation of the secondary feature amount generator 211 will be described. The input of the secondary feature generator 211 is the primary feature, and the output is the secondary feature. The primary feature quantity consists of preset preset data and a hyperparameter set.

Here, the prior data is, for example, data such as annual income and the presence or absence of debt used for credit card examination, and the result of a blood test including a blood glucose level in a hospital. It is conceivable to collect a large amount of advance data by utilizing open data.

It is desirable that the prior data to be collected is rich in various variations such as data including missing values and data in which non-continuous values such as gender are included in the features.

In the first embodiment, the result of performing accuracy prediction in advance with a plurality of hyperparameter sets for one pre-data prepared in advance is stored in a storage medium 109 or the like as a pre-accuracy prediction result (prediction accuracy). ..

That is, when there are 10 prior data and 5 hyperparameter sets, it is possible to prepare a primary feature amount of 50 samples. Pre-accuracy prediction results will exist for each of the primary features of 50 samples.

The pre-accuracy prediction result can be expressed by the probability that the two can be correctly distinguished when the hyperparameter set and the pre-data are applied to machine learning, for example, in the case of distinguishing between a patient and a non-patient. As the pre-accuracy prediction result, the algorithm described in the hyperparameter set and machine learning using the hyperparameters may be performed, and the accuracy evaluation result using 10 fold validation or the like may be used.

The secondary feature amount generator 211 is designed so that the secondary feature amount can be generated so that the pre-accuracy prediction result can be predicted by the accuracy predictor 212.

Next, the generation of the accuracy predictor 212 will be described. The input of the accuracy predictor 212 is the secondary feature amount output by the secondary feature amount generator 211, and the output of the accuracy predictor 212 is the pre-accuracy prediction result. The accuracy predictor 212 is designed so that the pre-accuracy prediction result can be correctly predicted from the secondary features.

The design of the secondary feature amount generator 211 and the accuracy predictor 212 may be performed by using the learner 220 of the abnormality detection device 100. The expected prediction accuracy predictor 210 may use the learner 220 to generate the secondary feature amount generator 211 and the accuracy predictor 212.

Even if it takes time to design the secondary feature generator 211 and the accuracy predictor 212, it takes a long period of time such as half a year as described above because learning is not performed when applying the data to the analysis target data. It does not affect the effectiveness of the present invention, which does not require specialists and high-performance analytical computers.

Further, the method for realizing the design of the secondary feature amount generator 211 and the accuracy predictor 212 is not limited to machine learning. Further, as the secondary feature amount, the secondary feature amount may be explicitly given without using the secondary feature amount generator 211. As the secondary features to be explicitly given, the number of samples, the number of dimensions of the features, the proportion of missing values, the proportion of continuous features such as age, and the discrete values such as gender in the preliminary data and the data to be analyzed. The ratio of the feature amount of is considered.

As described above, the abnormality detection device 100 predicts the accuracy of each of the plurality of hyperparameter sets for the pre-data prepared in advance, and calculates the pre-accuracy prediction result. Then, the abnormality detection device 100 generates an accuracy predictor 212 that inputs the secondary feature amount as an output (target value) of the pre-accuracy prediction result for each hyperparameter set, and inputs the pre-data and the hyperparameter set. , Generates a secondary feature amount generator 211 that outputs a secondary feature amount as an input of the accuracy predictor 212. The secondary feature amount generator 211 and the accuracy predictor 212 are not limited to being generated by the abnormality detection device 100. It may be generated in advance using a high-performance computer for analysis or the like.

In the first embodiment, the pre-accuracy prediction result is calculated for each hyperparameter from the pre-data for a plurality of hyperparameter sets prepared in advance, and when the pre-data and the hyper-parameter set are input, the pre-accuracy prediction result is output. The expected prediction accuracy predictor 210 including the feature quantity generator 211 and the accuracy predictor 212 is generated.

Then, the abnormality detection device 100 calculates the expected prediction accuracy by the expected prediction accuracy predictor 210 by inputting the combination of the analysis target data and the plurality of hyperparameter sets, and predicts that the highest accuracy will be obtained with respect to the analysis target data. It is possible to quickly evaluate the combination of hyperparameters and the value with the highest accuracy.

As a result, it can be expected that the hyperparameter search of machine learning does not require a long period of time such as half a year, and does not require a specialist and a high-performance computer for analysis. The expected prediction accuracy is the prediction accuracy that is expected to be achievable when a model is generated using the hyperparameters.

Next, the abnormality detection device 100 of the second embodiment will be described. The device of the first embodiment is a device that predicts the expected prediction accuracy. On the other hand, the abnormality detection device 100 of the second embodiment shows an example having a function of specifying a data component necessary for improving the prediction accuracy of the data to be analyzed by using the predicted expected prediction accuracy.

FIG. 9 is a conceptual diagram illustrating the function realized in the second embodiment. In the second embodiment, in order to give a specific explanation, the secondary feature amount will be described as if the secondary feature amount is explicitly given without using the secondary feature amount generator 211.

First, in the prior data, the data group having a prediction accuracy of 85% or more in any of the hyperparameter sets is referred to as group A, and the data group having a prediction accuracy of 70% or less is referred to as group B.

Within each group, the average value of each item of the secondary feature amount is calculated. When the data to be analyzed is applied to the expected prediction accuracy predictor 210 and the obtained expected prediction accuracy is lower than the desired value such as 70%, the difference is compared with the difference between the secondary features of Group A. Select a large item.

The anomaly detection device 100 presents a proposal on how to change the analysis target data in order to bring the selected item closer to the value of group A. For example, when the average value of the number of samples in group A is 10,000 and the value of the data to be analyzed is 100, an instruction to increase the number of samples is presented.

As a result, the abnormality detection device 100 can present to the operator how to specifically change the data in order to improve the prediction accuracy. The number of items to be presented may be set in advance, or a threshold value of difference may be set so as to be presented for each item of the secondary feature amount.

Further, not only two groups according to the prediction accuracy but also groups A and B may be provided for each hyperparameter set from the viewpoint of the difference in prediction accuracy depending on the hyperparameter set. Alternatively, the hyperparameter set may be included in the secondary features.

Furthermore, the data group whose prediction accuracy was 85% or more in any of the hyperparameter sets was group A, and the data group whose prediction accuracy was 70% or less was group B. 70% may be changed to any value such as 60%, and the number of groups may be increased to 3 or 4 instead of 2 groups.

Further, in the above description, the case where the secondary feature amount is explicitly given has been described, but the secondary feature amount generated by using the secondary feature amount generator 211 may be used, or the primary feature amount may be used. You may. Further, when the data to be analyzed is applied to the expected prediction accuracy predictor 210 and the obtained expected prediction accuracy is lower than a desired value such as 70%, it is compared with the difference in each secondary feature amount of Group A. Although the item with a large difference is selected, the item considered to be the cause of the low prediction accuracy may be selected by a different method.

Further, instead of comparing the difference in the secondary features with the groups A and B, a machine that identifies whether the feature is in groups A or B from the features in the prior data, the primary features, or the secondary features. A learning model may be generated, a feature amount or a primary feature amount or a secondary feature amount of the analysis target data may be input, and the analysis target data may be evaluated from the identified group. Further, as a method for identifying which of the groups A and B is, a well-known or known statistical method such as a Markov chain Monte Carlo method may be used instead of machine learning.

As described above, in the abnormality detection device 100 of the second embodiment, the analysis target data whose expected prediction accuracy is equal to or less than the threshold is compared with the difference in each secondary feature amount of the analysis target data whose expected prediction accuracy exceeds the threshold. It is possible to select an item with a large difference and output an instruction to improve the data to be analyzed. This makes it possible to identify the components of the data required to improve the prediction accuracy.

Next, the abnormality detection device 100 of the third embodiment will be described. The abnormality detection device 100 of the first embodiment is a device that predicts the expected prediction accuracy. The abnormality detection device 100 of the third embodiment shows an example having a function of determining whether or not the analysis target data is worthy of analysis before performing data analysis of the analysis target data.

The data to be analyzed, which cannot be predicted with high accuracy by machine learning, contains a small proportion of important features, and there is a possibility that no information can be obtained by analysis. For example, when it is desired to clarify by medical research that the early morning fasting blood glucose level and the occasional blood glucose level, which are currently necessary items for diagnosing diabetes, are the values necessary for the diagnosis, the early morning fasting blood glucose level. If the value or occasional blood glucose level is not included in the analysis target data, the items necessary for diagnosing diabetes that you want to clarify are not included, so even if you analyze the analysis target data, you can diagnose diabetes. It is impossible to find the item you need.

Here, the analysis of the data to be analyzed is an analysis method that has been performed as a gold standard in each field, such as logistic regression that is well known or known in medical research, and an analysis method that has been developed to meet a specific purpose. It is an analysis using. As described above, the data to be analyzed needs to include important features that sufficiently contribute to the analysis target.

However, when analyzing an unknown event, it is not possible to know that the important feature amount is included in the analysis target data. Therefore, before performing the analysis, it is conceivable to apply the expected prediction accuracy predictor 210 as realized by the flowchart of FIG. 3 of the first embodiment.

Even in the prediction accuracy by machine learning, the presence or absence of the important feature amount affects the accuracy. When the expected prediction accuracy is higher than the predetermined threshold, the analysis target data contains the important feature amount, and when the expected prediction accuracy is lower than the predetermined threshold, the analysis target data is said to be important. It is possible to judge that the characteristic amount is not included. When the expected prediction accuracy is equal to or less than the threshold value, the abnormality detection device 100 can determine that no significant result can be obtained even if the analysis is performed, and can stop the analysis of the data or present it to the operator.

FIG. 10 is a flowchart showing an example of processing for realizing the third embodiment. First, the abnormality detection device 100 reads the analysis target data into the memory 103 (step S1001). Subsequently, the abnormality detection device 100 refers to the primary feature amount composed of the analysis target data and the hyperparameter set, and generates the secondary feature amount by the secondary feature amount generator 211 (step S1002).

Next, the abnormality detection device 100 predicts the expected prediction accuracy by the accuracy predictor 212 using the generated secondary feature amount (step S1003). Subsequently, the abnormality detection device 100 determines whether or not to perform data analysis based on the predicted expected prediction accuracy (step S1004).

If the expected prediction accuracy is equal to or higher than a predetermined threshold value, the abnormality detection device 100 proceeds to step S1005, and if it determines that the data analysis is not performed, the abnormality detection device 100 ends.

When it is determined that the data analysis is to be performed, the abnormality detection device 100 performs the data analysis described later and then ends the process (step S1005).

The threshold value of the expected prediction accuracy in step S1004 may be arbitrarily set or may be set to a specific value. Further, as one of the data analysis methods, the following methods may be used.

FIG. 11 is a flowchart showing an example of a data analysis method. This process is performed in step S1005 of FIG.

First, the abnormality detection device 100 reads the analysis target data into the memory 103 (step S1101). The anomaly detection device 100 calculates by non-linear correlation whether or not there is a correlation between each feature amount of the read analysis target data and the label described for each sample described later, and the correlation coefficient is a predetermined threshold value. Among those exceeding the above, the higher feature amount is selected (step S1102).

Here, each feature amount of the data to be analyzed is data such as annual income and the presence or absence of debt used for credit card examination, and the result of a blood test including a blood glucose level in a hospital. Further, the label described for each sample is, for example, a label added in advance for each record of the data to be analyzed, and the data is data on whether or not the data is patient data or data in a state where the device has failed. This is information that distinguishes between a state in which an abnormality such as presence / absence is detected and a state in which an abnormality is not detected (no abnormality is detected).

Further, in the above, the non-linear correlation is used for the calculation of the correlation coefficient, but the linear correlation may be used, or the relationship existing between the label and each feature amount of the data to be analyzed by a method other than the correlation coefficient. The height may be evaluated.

Subsequently, the abnormality detection device 100 predicts the accuracy of each feature amount of the analysis target data selected in step S1102 by machine learning (step S1103). The accuracy prediction by machine learning can be performed quickly by adopting the hyperparameter set with high accuracy predicted in the first embodiment.

Further, the abnormality detection device 100 generates a combination of two pairs in each feature amount of the analysis target data selected in step S1102 (step S1104).

The abnormality detection device 100 predicts the accuracy of the combination generated in step 1104S by machine learning (step S1105). This accuracy prediction can also be performed quickly by adopting the hyperparameter set with high accuracy predicted in the first embodiment.

The abnormality detection device 100 selects the important pair feature amount according to the selection rule (described later) of the important pair feature amount using the two types of prediction accuracy calculated and the 10-fold cross-validation result in step S1003 and step S1005 (step). S1106).

The abnormality detection device 100 selects the relationship between the plurality of important pair feature quantities selected in step S1106 and ends the process (step S1107).

Here, the important pair feature amount is a feature amount that is a combination that improves the expected prediction accuracy by pairing the feature amount of the analysis target data, although the prediction accuracy does not increase with the feature amount of one analysis target data. Point to.

The rules for selecting important pair features will be described. The anomaly detection device 100 primary sets a pair of feature quantities of analysis target data such that the difference between the single expected prediction accuracy calculated in step S1103 and the expected prediction accuracy of the pair calculated in step S1105 exceeds a predetermined threshold value. Select as an important feature.

Here, there are two types of individual prediction accuracy for the feature amount of the data to be analyzed as a pair, but the prediction accuracy is improved by making a pair by calculating the difference between the higher value and the pair prediction accuracy. It is thought that it will be easier to select such a combination.

Next, the abnormality detection device 100 is important only for the pair in which the average value of 10 fold cross validation calculated in step S1103 and step S1105 is equal to or more than the threshold value (for example, 70%) among the pairs that have become the primary important features. Select as a feature quantity.

The cross-validation may be a numerical value such as 5 instead of 10. Further, the threshold value of the average value of cross validation may be 90% instead of 70%, and in the case of regression, it may be an actual numerical value or a coefficient of determination.

The abnormality detection device 100 can determine the certainty of the prediction accuracy by using the cross-validation value. It may be executed without using the cross-validation value, but overfitting selects a pair as an important pair feature quantity, which has high single prediction accuracy and pair prediction accuracy and is not actually easy to predict. It is expected that the difficulty of avoiding this will increase.

The relationship between the plurality of selected important pair feature quantities in step S1106 will be described. For example, consider the case where the selected important pair feature amounts are feature amount A and feature amount B, feature amount B and feature amount C, and feature amount C and feature amount A.

Since the feature amount A, the feature amount B, and the feature amount C are important feature amounts to each other, it is considered that the prediction accuracy is improved when the accuracy is predicted by these three feature amounts as compared with the case where the prediction is made independently. Furthermore, when the accuracy is predicted by these three features, the expected prediction accuracy may be improved as compared with the case where the prediction is made in pairs.

By extracting the important pair features in this way, it is possible to discover not only two features but also features having a relationship of improving prediction accuracy by increasing the features in three or more features. It will be possible. In addition, a non-linear correlation may be performed on the two types of features that are the combination generated in step 1104S, and the pair with a low correlation may be indicated as a more important important pair feature. Although there is no correlation between the features, the features that are considered to be important pair features by the flow shown in FIG. 11 can be considered to be difficult to find by the analysis using the linear method. .. By using such an analysis method, it can be expected that relationships that could not be found by the linear method often used in medical research can be discovered. In addition to applying the data analysis method to the data to be analyzed, the data analysis method may be applied to data generated by a technique for generating similar data by referring to the data to be analyzed.

As described above, according to the third embodiment, the analysis target data includes the important feature amount, and when the expected prediction accuracy is low, the analysis target data includes the important feature amount. It is possible to judge that there is no such thing. In this case, it can be determined that no significant result can be obtained even if the analysis is performed, and the analysis of the data can be stopped.

Further, according to the third embodiment, it is possible to extract as an important pair feature amount a combination in which the prediction accuracy is improved by pairing the secondary feature amount.

Next, the abnormality detection device 100 of the fourth embodiment will be described. In the device of the first embodiment, an example of functioning as a device for predicting the expected prediction accuracy is shown. The abnormality detection device 100 of the fourth embodiment shows an example having a function of determining the quality of the purification method of the data to be analyzed based on an index (expected prediction accuracy).

As the data to be analyzed, the acquired data can be used as it is, but the data obtained by extracting the truly important (or useful) features by data purification (cleansing) may be used.

For example, if the number of features exceeds 100,000, not all 100,000 are meaningful features, and there may be many features that are completely unrelated to the problem to be analyzed. high.

Therefore, if a large number of features irrelevant to the analysis are included, the expected prediction accuracy may be adversely affected. In addition, when characters are included in the features, or when different types of features such as images and inspection values are included, it is necessary to perform preprocessing so that they can be treated as the same feature vector. is there.

Since there may be various methods and methods for determining important features from various viewpoints in these preprocessing, different types of preprocessing are applied not only to one pattern of analysis target data but also to one analysis target data. It is necessary to generate a plurality of patterns of the generated data and test which pattern has the highest expected prediction accuracy.

Therefore, by using the fourth embodiment, a plurality of types of analysis target data are applied to the expected prediction accuracy predictor 210 as represented by the flowchart of FIG. 2, and the input multiple types of analysis target data are selected. By presenting in descending order of expected prediction accuracy, it is possible to instantly determine which analysis target data generation method was good.

FIG. 12 is a diagram illustrating the fourth embodiment. From one original analysis target data, for example, using three types of data purification (or cleansing) methods, method A, method B, and method C, analysis target data A and analysis target data B are used as three types of analysis target data. , The data C to be analyzed is purified.

The expected prediction accuracy predictor 210 shown in the flowchart shown in FIG. 2 of the first embodiment is applied to each of the analysis target data A to C, and the accuracy with the highest expected prediction accuracy is presented. It is considered that the method of comparing the expected prediction accuracy of the three types of analysis target data (data A to C) and purifying the analysis target data with the highest expected prediction accuracy is the best method among the three data purification methods. Be done.

The abnormality detection device 100 of the fourth embodiment can adopt the purification method (data cleansing) having the highest expected prediction accuracy as the preprocessing of the data to be analyzed.

Utilizing this knowledge, a new data purification method D or the like can be implemented and compared with other data purification methods by the same procedure. By using the Expected Prediction Accuracy Predictor 210, even if you want to select the best data purification method from a plurality of data purification methods and perform data analysis, it does not require a long period of time such as half a year and is specialized. It makes it possible to eliminate the need for homes and high-performance analytical calculators.

<Summary>
The present invention is not limited to the above-described examples, and includes various modifications. For example, the above-described embodiment is described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the configurations described. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, for a part of the configurations of each embodiment, any of addition, deletion, or replacement of other configurations can be applied alone or in combination.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

100 Anomaly detection device 101 Data bus 102 CPU
103 Memory 104 Input / output interface 105 Communication device 106 Drive device 107 Input device 108 Output device 109 Storage medium 210 Expected prediction accuracy predictor 211 Secondary feature quantity generator 212 Accuracy predictor 220 Learner 230 Result display

Claims (15)

A computer with a processor and memory is a hyperparameter evaluation method that evaluates hyperparameters applicable to the data to be analyzed.
A first step in which the computer generates a secondary feature amount generation unit and an accuracy prediction unit from preset preset data and a plurality of hyperparameters.
The second step in which the computer receives the data to be analyzed, and
A third step in which the secondary feature amount generation unit calculates the secondary feature amount by inputting the analysis target data and a plurality of hyperparameters.
The fourth step in which the accuracy prediction unit calculates the expected prediction accuracy for each hyperparameter by inputting the secondary feature amount, and
A method for evaluating hyperparameters, which comprises. The hyperparameter evaluation method according to claim 1.
The third step is
A method for evaluating hyperparameters, which comprises inputting the analysis target data and a plurality of hyperparameters as primary feature quantities and calculating secondary feature quantities. The hyperparameter evaluation method according to claim 1.
The fourth step is
A method for evaluating hyperparameters, which comprises calculating the expected prediction accuracy expected to be achievable for each hyperparameter by inputting the secondary feature amount. The hyperparameter evaluation method according to claim 1.
A fifth step in which the computer outputs the calculated expected prediction accuracy for each hyperparameter, and
The sixth step in which the computer accepts the selected hyperparameters among the output hyperparameters, and
A seventh step in which the computer generates a model for analyzing the analysis target data based on the selected hyperparameters.
A method for evaluating hyperparameters, which further comprises. The hyperparameter evaluation method according to claim 1.
The computer acquires the secondary features and the expected prediction accuracy of the plurality of analysis target data, and for the analysis target data whose expected prediction accuracy is equal to or less than the threshold, each of the analysis target data whose expected prediction accuracy exceeds the threshold. The eighth step of selecting an item with a large difference compared to the difference in the secondary feature amount, and
A method for evaluating hyperparameters, which further comprises. The hyperparameter evaluation method according to claim 1.
A ninth step in which the computer determines that when the expected prediction accuracy is equal to or less than a predetermined threshold value, no significant result can be obtained even if data analysis is performed on the analysis target data.
A method for evaluating hyperparameters, which further comprises. The hyperparameter evaluation method according to claim 1.
The second step is
Including the step of performing data cleansing on the received data to be analyzed.
The tenth step of acquiring the expected prediction accuracy after the computer performs a plurality of data cleansing on the data to be analyzed and selecting the data cleansing with the highest expected prediction accuracy.
A method for evaluating hyperparameters, which further comprises. The hyperparameter evaluation method according to claim 1.
The first step includes a step of predicting the accuracy of each of the plurality of hyperparameters when applied to the analysis of the prior data and calculating the preliminary accuracy prediction result.
A step of generating an accuracy prediction unit that outputs the pre-accuracy prediction result for each hyperparameter and inputs a secondary feature amount, and
A step of generating a secondary feature amount generation unit that inputs the prior data and a plurality of hyperparameters and outputs the secondary feature amount input of the accuracy prediction unit, and
A method for evaluating hyperparameters, which comprises. The hyperparameter evaluation method according to claim 4.
The eleventh step in which the computer outputs the hyperparameters and the generated model in the order of the calculated expected prediction accuracy.
A method for evaluating hyperparameters, which further comprises. A computer that evaluates hyperparameters applicable to the data to be analyzed, including the processor and memory.
Expected prediction accuracy prediction unit including a secondary feature amount generation unit generated from preset preset data and a plurality of hyperparameters, and an accuracy prediction unit generated from preset preset data and a plurality of hyperparameters. Have and
The expected prediction accuracy prediction unit receives the analysis target data, and receives the analysis target data.
The secondary feature amount generation unit calculates the secondary feature amount by inputting the analysis target data and a plurality of hyperparameters.
A computer characterized in that the accuracy prediction unit calculates the expected prediction accuracy for each hyperparameter by inputting the secondary feature amount. The calculator according to claim 10.
A computer characterized in that the secondary feature amount generation unit calculates the secondary feature amount by inputting the analysis target data and a plurality of hyperparameters as the primary feature amount. The calculator according to claim 10.
A computer characterized in that the accuracy prediction unit calculates an expected prediction accuracy that is expected to be achievable for each hyperparameter by inputting the secondary feature amount. The calculator according to claim 10.
The accuracy prediction unit outputs the calculated expected prediction accuracy for each hyperparameter.
It is characterized by further having a learning unit that accepts the selected hyperparameters among the output hyperparameters and generates a model for analyzing the analysis target data based on the selected hyperparameters. Computer to be. The calculator according to claim 10.
The expected prediction accuracy prediction unit acquires the secondary feature amount and the expected prediction accuracy of a plurality of analysis target data, and analyzes the analysis target data whose expected prediction accuracy is equal to or less than the threshold, and the expected prediction accuracy exceeds the threshold. A computer characterized in that an item having a large difference is selected by comparing with the difference of each secondary feature amount of the target data. A program for controlling a computer with a processor and memory.
The first step of generating a secondary feature amount generation unit and an accuracy prediction unit from preset preset data and a plurality of hyperparameters, and
The second step of accepting the data to be analyzed and
A third step of inputting the analysis target data and a plurality of hyperparameters to the secondary feature amount generation unit to calculate the secondary feature amount, and
The fourth step of inputting the secondary feature amount into the accuracy prediction unit and calculating the expected prediction accuracy for each hyperparameter, and
A program for causing the computer to execute.

Images