KR20030071939A - Apparatus and method for construction model of data mining using ensemble machines - Google Patents

Abstract

The present invention relates to an ensemble construction apparatus and method having faster analysis time and stable model construction, superior analysis and predictive power, and a method for analyzing and predicting multidimensional large data.

According to the present invention, in a data mining model construction apparatus using an ensemble model, the apparatus includes M ensemble model construction means, and the first ensemble model construction means includes a first learner from the input multidimensional training material. Constructing the first learner itself as a first ensemble model; The k th ensemble model construction means receives a k-1 ensemble model that is the result of the k-1 ensemble model construction means, constructs an orthogonal k th learner in a multidimensional space, and constructs the k th learner and the k-1 th Provided is a data mining model building apparatus using the ensemble model, wherein the ensemble model is constructed by combining the ensemble model.

Description

Apparatus and method for construction model of data mining using ensemble machines}

The present invention relates to an apparatus and method for constructing a data mining model using an ensemble model, and more specifically, to a faster calculation time and a more stable construction of a model than a conventional method for analyzing and predicting multidimensional large data. The present invention relates to an apparatus and method for constructing a data mining model having excellent analysis and predictive power.

1. Introduction

Ensemble Algorithm in DataMining has been studied in recent years with 'Breiman''s bagging technique as the first.

In other words, a number of studies are being conducted to further improve the above-described bagging technique of Breiman. As a result of the conventional research, the boosting algorithm proposed by Freund and Schapire, Breiman, is proposed. The Arching algorithm and the Random Forest algorithm proposed by Breiman.

Among these ensemble techniques, the boosting algorithm proposed by 'Freund and Schapire' has recently appeared in various improved algorithms based on its superior predictive power. These improved algorithms include Real Boosting algorithm proposed by 'Schapire and Singer' and Gradient Boosting algorithm proposed by 'Friedman'.

That is, the ensemble algorithm used in the conventional data mining is mainly based on the boosting algorithm.

The following briefly introduces a number of conventional boosting algorithms applied to classification problems. In particular, the most widely used real boosting algorithm, logit boosting algorithm, and gradient boosting algorithm will be described.

2. Conventional Boosting Algorithms for Two-Classes

The boosting algorithm is the main method used for classification problems. The basic model of the classification problem is as follows.

Suppose that _n training materials (x ₁ , y ₁ ), ..., (x _n , y _n ) are given. Here, x _i is an explanatory variable of p dimension, that is, x _i = (x _1i , ..., x _pi ), and the response variable y _i represents a group to which the data belongs. That is, when there are J groups, y _i has an integer value from 1 to J.

The purpose of the classification problem is to find relationships that best describe the response variables with explanatory variables using n training data. In other words, we use the training material to create the optimal function H: R ^p- > {1, 2, ..., J}. And given a new explanatory variable x, we assign this data to group H (x).

First, consider the case of two groups, that is, J = 2. If there are multiple groups, they will be described later. The following are the various boosting algorithms.

2-1. Real Boosting Algorithm (Schapire and Singer, 1999)

Real boosting algorithm is the most representative algorithm for ensemble construction algorithm for data mining.

The algorithm is described in detail in US Pat. No. 5,819,247, Apparatus and methods for machine learning hypotheses, which will be described in detail as follows.

1 is a flowchart illustrating a conventional real boosting method applied to two groups in a classification problem. This will be described in detail as follows.

(1) Step S101: The response variable y _{i is set} to 1 if the data belongs to group 2, and to -1 if it belongs to group 1.

(2) Step S102: Initialize by setting n weights w ₁ , ..., w _n to w _i = 1 / n.

(3) Step S103: Respond to the explanatory variable x given using _n training materials (x ₁ , y ₁ ), ..., (x _n , y _n ) and weights w ₁ , ..., w _n The probability that the variable is 1 is estimated using the basic learner. At this time, the probability that the response variable is 1 is determined by Equation 1 below.

(4) Step S104: The above-mentioned P _m is converted to find f _m . This is determined by Equation 2 below.

(5) Step S105: After the new weight is obtained by the following Equation 3, it is obtained. Normalize to

(6) Step S106: By repeating steps S103 to S105 up to m = 1, ..., M times, M basic learners are generated.

(7) Step S107: The final ensemble model is determined by the following [Equation 4]. At this time, if H (x)> 0 for the new response variable x, it is assigned to Group 2, if H (x) <0, it is assigned to Group 1.

Through the above process, the ensemble model is finally constructed.

2-2. Logit Boosting Algorithm (Friedman et al., 2000)

2 is a flowchart illustrating a conventional logit boosting method applied to two groups in a classification problem. This will be described in detail as follows.

(1) Step S201: The reaction variable y _{i is set} to 1 if the data belongs to group 2 and to 0 if the data belongs to group 1.

(2) Step S202: n weights w ₁ , ..., w _n are initialized to w _i = 1 / n, F (x) = 0, p (x) = 1/2.

(3) Step S203: The new response variable z _i and the weight w _i are obtained for the given explanatory variable by the following [Equation 5].

(4) Step S204: Using the response variable z _i , the explanatory variable x _i, and the weight w _i , a regression model f _m (x) is constructed with reference to the basic learner.

(5) Step S205: The probability is updated using f _m (x) obtained in the above step S204. This is shown in Equation 6 below.

(6) Step S206: By repeating Step S203 to Step S205 M times, probability update is performed M times.

(7) Step S207: The final ensemble model is constructed with H (x) = F (x). At this time, if H (x)> 0 for the new response variable x, it is assigned to Group 2, if H (x) <0, it is assigned to Group 1.

2-3. Gradient Boosting Algorithm (Friedman, 2001)

3 is a flowchart illustrating a conventional gradient boosting method applied to two groups in a classification problem. This will be described in detail as follows.

(1) Step S301: The response variable y _{i is set} to 1 if the data belongs to group 2, and to -1 if it belongs to group 1.

(2) Step S302: Initialize various variables as shown in Equation 7 below.

(3) Step S303: The new response variable z _i is calculated for the given explanatory variable by the following [Equation 8].

(4) Step S304: f _m (x) is estimated using the response variable z _i , the explanatory variable x _i, and the basic learner (Regression Decision Tree Model).

(5) Step S305: Predicted value of the l th last node of the f _m (x) Is estimated using Equation 9 below.

Where R _i is a set of data belonging to the i th last node.

(6) Step S306: Update to F (x) = F (x) + f _m (x).

(7) Step S307: By repeating the steps S303 to S306 M times, F (x) is updated M times.

(8) Step S308: The final ensemble model is constructed as H (x) = F (x). At this time, if H (x)> 0 for the new response variable x, it is assigned to Group 2, if H (x) <0, it is assigned to Group 1.

Meanwhile, the above-described real boosting algorithm or logit boosting algorithm may use various basic learners, but the gradient boosting algorithm must use a decision tree as the basic learner.

In addition, when constructing the basic learner in the above algorithm, the complexity of the learner should be determined in advance. If the basic learner is a decision tree, the complexity of the learner can be controlled by the number of final nodes. Meanwhile, in the boosting algorithm, since the basic learners are weak learners, the complexity of the learners is minimized. In general, decision trees having 2 to 8 final nodes are commonly used as basic learners.

3. Conventional Boosting Algorithms for Multi Classes

The three boosting algorithms described above are algorithms applied when there are two groups, and when the groups are extended to two or more groups, the algorithms must be expanded and modified.

Hereinafter, the real boosting algorithm, the logit boosting algorithm, and the gradient boosting algorithm applied to the multi-class will be described.

3-1. Real Boosting Algorithm Applies to Multiclasses

4 is a flowchart illustrating a conventional real boosting method applied to a multi-class in a classification problem, which will be described in detail as follows.

(1) Step S401: J new materials ((x _i , 1), y _i1 ), ..., ((x _i , J), y _{iJ for a} given learning material (x _i , y _i ) ) At this time, y _ik is 1 if y _i is k, and -1 if k is not.

(2) Step S402: By applying a two-class real boosting algorithm to n X J new materials (i.e., converting multi-class to two-class), a final ensemble model is constructed by Equation 10 below.

(3) Step S403: The new explanation variable x is assigned to the argmax _j H (x, j) group.

On the other hand, the real boosting algorithm applied to the above-described multi-class has a problem in that it takes too long since the number of executions, that is, the number of calculations corresponds to the number of boosting times.

3-2. Logit Boosting Algorithm Applies to Multiclasses

5 is a flowchart illustrating a conventional logit boosting method applied to a multi-class in a classification problem, which will be described in detail as follows.

(1) Step S501: Initialize various variables. That is, let F _j (x) = 0, p _j (x) = 1 / J, j = 1, 2, ..., J.

(2) Step S502: If the i th data is included in the j group, y _i ^* = 1, and if not, y _i ^* = 0.

(3) Step S503: The new response variable z _i and the new weight w _i are determined by Equation 11 below.

(4) Step S504: The regression model f _mj (x) is calculated by referring to the basic learner using the response variable z _i , the explanatory variable x _i , and the weight w _i .

(5) Step S505: The above steps S502 to S504 are repeated J times.

(6) Step S506: The j th F _j (x) is updated by Equation 12 below.

(7) Step S507: The j th probability p _j (x) is updated by Equation 13 below.

(8) Step S508: The above steps S502 to S507 are repeated M times.

(9) Step S509: Assign the new explanatory variable x to the argmax _j F _j (x) group.

3-3. Gradient Boosting Algorithm Applies to Multiclasses

6 is a flowchart illustrating a conventional gradient boosting method applied to a multi-class in a classification problem, which will be described in detail as follows.

(1) Step S601: Initialize various variables. That is, let F _j (x) = 0, p _j (x) = 1 / J, j = 1, 2, ..., J.

(2) Step S602: If the i th data is included in the j group, y _i ^* = 1, and if not, y _i ^* = 0.

(3) Step S603: The new reaction variable z _i is determined by Equation 14 below.

(4) Step S604: Using the response variable z _i and the explanatory variable x _i , a regression model f _mj (x) is constructed by using a basic learner (Regression Decision Tree).

(5) step S605: predicted value of the l-th final node of the regression model f _mj (x) It is estimated by Equation 15 below.

(6) Step S606: F _j (x) is determined by the following [Equation 16].

(7) Step S607: Steps S602 to S606 are repeated j = 1, ..., J times.

(8) Step S608: The steps S602 to S607 are repeated m = 1, ..., M times.

(9) Step S609: Assign the new explanatory variable x to the argmax _j F _j (x) group.

4. Problem solving of prior arts

The basic idea of the boosting technique as described above is to build several weak learners and combine them to create a new strong learner. However, this idea has a number of problems, which can be summarized as follows.

First, it is difficult to interpret the constructed model.

Second, there are several tuning parameters, that is, the size of the decision tree, the number of decision trees, and so on.

Third, data tends to be overfitting (Ridgeway, 2000).

Fourth, the use of the algorithm is not easy when the type of response variable is continuous.

Fifth, it is not easy to estimate the probability in the classification problem. That is, when new data is inputted, it is not possible to estimate the classification probability that the new data inputted will take.

Sixth, the general (statistical) theory of boosting techniques is insufficient, making it difficult to verify its stability.

An object of the present invention for solving the problems of the prior art as described above is to build an ensemble with faster analysis time and stable model construction and excellent analysis and predictive power than conventional methods in analyzing and predicting multidimensional large data An apparatus and a method thereof are provided.

1 is a flowchart illustrating a conventional real boosting method applied to two groups in a classification problem.

2 is a flowchart illustrating a conventional logit boosting method applied to two groups in a classification problem.

3 is a flowchart illustrating a conventional gradient boosting method applied to two groups in a classification problem.

4 is a flowchart illustrating a conventional real boosting method applied to a multi-class in a classification problem.

5 is a flowchart illustrating a conventional logit boosting method applied to a multi-class in a classification problem.

6 is a flowchart illustrating a conventional gradient boosting method applied to a multi-class in a classification problem.

7 is a flowchart schematically illustrating a method for constructing an ensemble model according to an embodiment of the present invention.

8 is a flowchart schematically illustrating a cross-check process for selecting an optimal decision tree,

9 is a flowchart showing an overall overview of a method of selecting an optimal decision tree using TIC,

10 is a flowchart illustrating a process of generating boost strap data according to an embodiment of the present invention.

11 is a basic conceptual view showing that there are a plurality of learners having a small distance from a true learner included in a set of conventional basic learners.

12a is a diagram conceptually illustrating a method of constructing a first learner according to a cam algorithm proposed by the present invention;

12B is a diagram conceptually illustrating a method of building a second learner according to the cam algorithm proposed by the present invention.

12c conceptually illustrates a method of constructing a third learner according to the cam algorithm proposed by the present invention;

12D is a diagram conceptually showing a method of building an m-th learner according to the cam algorithm proposed by the present invention.

13 is a flowchart showing an outline of a cam algorithm according to an embodiment of the present invention;

14 is a flowchart schematically illustrating an ensemble algorithm for continuous variables according to an embodiment of the present invention;

15 is a flowchart showing an overview of an association rule generation algorithm;

16A to 16D are graphs showing the results of a virtual experiment for examining the performance of the conventional ensemble technique and the ensemble technique proposed by the present invention.

17A to 17I are graphs showing the results of actual data analysis to determine the performance of the conventional ensemble technique and the ensemble technique proposed by the present invention.

In order to achieve the above object, according to the present invention, in a data mining model building apparatus using an ensemble model, comprising M ensemble model building means, the first ensemble model building means is input Constructing a first learner from multidimensional learning materials, and constructing the constructed first learner itself as a first ensemble model; The k th ensemble model construction means receives a k-1 ensemble model that is the result of the k-1 ensemble model construction means, constructs an orthogonal k th learner in a multidimensional space, and constructs the k th learner and the k-1 th Provided is a data mining model building apparatus using the ensemble model, wherein the ensemble model is constructed by combining the ensemble model.

In addition, in a data mining model construction method using an ensemble model, the method includes constructing an M ensemble model, and the first ensemble model construction step includes a first learner from the input multidimensional training material. Constructing the constructed first learner itself as a first ensemble model; In the k-th ensemble model construction step, the k-1 th learner, which is the result of the k-1 th ensemble model construction step, is input, constructs an orthogonal k th learner in a multidimensional space, and constructs the k th learner and k. By combining the -1 th ensemble model, a method for constructing a data mining model using the ensemble model, characterized in that the k th ensemble model is constructed.

In addition, an apparatus for constructing a data mining model in two group classifications using an ensemble model, the apparatus comprising: input means for receiving training materials that are multidimensional data; Weight calculation means for calculating a weight using a residual of the ensemble model; Probability estimating means for estimating a probability that the response variable becomes 1 with respect to the explanatory variable x given from the data input by the input means using a basic learner and the weight; Correction parameter calculation means for calculating a correction parameter that minimizes a given loss function with respect to the probability estimated by the probability estimation means; Learner updating means for updating and rebuilding a learner using the calculated correction parameter; And an ensemble model building means for finally building an ensemble model based on the updated learner.

A method of constructing a data mining model in two group classifications using an ensemble model, the method comprising: a first step of receiving training materials that are multidimensional data; Calculating a weight using a residual of the ensemble model; A third step of estimating a probability that the response variable becomes 1 with respect to the explanatory variable x given from the data input in the first step using a basic learner and the weight; A fourth step of calculating a correction parameter that minimizes a given loss function with respect to the probability estimated in the third step; And a fifth step of updating and reconstructing the learner by using the correction parameters calculated in the fourth step.

An apparatus for constructing a data mining model in a multi-class classification using an ensemble model, the apparatus comprising: input means for receiving training materials that are multidimensional data; Weight calculation means for calculating a weight using a residual of the ensemble model; Probability estimating means for estimating a probability that the response variable belongs to the j group with respect to the explanatory variable x given from the data input by the input means using a basic learner and the weight; Correction parameter calculation means for calculating a correction parameter that minimizes a given loss function for the probability estimated by the probability estimation means; Learner updating means for updating and rebuilding a learner using the calculated correction parameter; And an ensemble model building means for finally building an ensemble model based on the updated learner.

A method of constructing a data mining model in a multi-class classification using an ensemble model, the method comprising: a first step of receiving training materials which are multidimensional data; Calculating a weight using a residual of the ensemble model; A third step of estimating, using the basic learner and the weight, a probability that the response variable belongs to the j group with respect to the explanatory variable x given from the data input in the first step; A fourth step of calculating a correction parameter that minimizes a given loss function with respect to the probability estimated in the third step; A fifth step of updating and rebuilding a learner using the correction parameter calculated in the fourth step; And a sixth step of finally constructing an ensemble model based on the updated learner.

In addition, a data mining model building apparatus using an ensemble model when the response variable is a regression, the apparatus comprising: input means for receiving training materials that are multidimensional data; Residual calculation means for calculating a residual of the ensemble model; Regression model building means for constructing a regression model using a basic learner and the residual from data input by the input means; Correction parameter calculation means for calculating a correction parameter that minimizes a given loss function for the regression model constructed by the regression model construction means; Learner updating means for rebuilding a learner by updating a response variable using the calculated correction parameter; And an ensemble model building means for finally constructing an ensemble model based on the reconstructed learner, and provides a data mining model building apparatus using the ensemble model when the response variable is continuous.

In addition, a data mining model construction method using an ensemble model when the response variable is a regression, the method comprising: a first step of receiving training materials that are multidimensional data; Calculating a residual of the ensemble model; A third step of constructing a regression model using a basic learner and the residual from data input in the first step; A fourth step of calculating a correction parameter that minimizes a given loss function for the regression model constructed in the third step; A fifth step of rebuilding the learner by updating the response variable using the correction parameter calculated in the fourth step; And a sixth step of finally constructing the ensemble model based on the learner reconstructed in the fifth step. The method provides a data mining model building method using the ensemble model when the response variable is continuous.

Even more preferably, the input training data is generated as boost data by using a weight.

Also, more preferably, the learner is either a decision tree or a neural network model.

Also, more preferably, the ensemble model is finally constructed when the loss function of the current ensemble model is the smallest.

Hereinafter, an ensemble construction device and method for data mining according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1. Introduction

The data mining model construction algorithm proposed by the present invention solves the problems of the above-mentioned boosting technique and at the same time, is a new algorithm whose predictive power is superior to boosting. In addition, when the decision tree is adopted as the basic learner in terms of interpretation power, the decision tree is further improved than the decision tree, which is known to have the best interpretation power (of course, the prediction is very poor) among the conventional data mining techniques. In particular, it is possible to extract various kinds of association rules that can explain a given data and to analyze the given data from various angles.

Theoretically, the algorithm proposed by the present invention is completely different from the basic idea of the boosting technique. While the boosting technique is to fuse a plurality of weak learners to make a new strong learner, the algorithm proposed in the present invention is to fuse a plurality of strong learners to make a stronger learner.

The basic learner used in the ensemble technique proposed by the present invention will be described taking the case of using a decision tree as an example. The reason for this is the simplicity of the algorithm and the high interpretation power, which are the advantages of decision trees. However, the selection of the basic learner can be changed according to the type of material. For example, a neural network model can be used for text recognition or speech recognition.

Fast computations are essential to using decision trees as basic learners. In general, currently known decision tree construction algorithms are the cart algorithm proposed by Breiman (CART Algorithm, Breiman et al., 1986). In the case of the kart algorithm, there are three stages: tree growth, pruning and optimal decision tree selection.

In this case, the third step, the algorithm for selecting an optimal tree model, uses a cross validation technique, which requires a large amount of computation. The computation required for cross validation to generate a single decision tree is not too burdensome, but because the ensemble technique generates multiple decision trees, applying cross validation to all decision trees can lead to a surge in computation. It is necessarily accompanied.

In the present invention, in order to overcome the problem of cross-checking, the TIC (Tree Information Criteria) is newly defined and used to construct an optimal decision tree at a faster time.

However, the data mining model construction technique proposed by the present invention is not limited to the decision tree selection method using the TIC. This is because the present invention can use not only a decision tree as a basic learner but also various basic learners (for example, neural network learners). However, for convenience of description, the present invention selects and describes a decision tree as a basic learner.

7 is a flowchart schematically illustrating a method for constructing an ensemble model according to an embodiment of the present invention.

First, when learning data which is multidimensional data is input in step S701, a basic learner is constructed in step S702. In this case, a decision tree, a neural network model, and the like may be used as the basic learner. In the present application, it is assumed that the decision tree is for convenience of explanation. Therefore, the optimal decision tree is constructed in step S702. At this time, there are several methods for constructing an optimal decision tree, and the present invention introduces a new method called a tree information criterion (TIC) algorithm.

Next, in step S703, an ensemble model is constructed using the constructed basic learners, and in step S704, it is determined whether the constructed ensemble model is optimal.

If it is determined in the step S704 that it is not optimal, in step S705 a new response variable is generated, and then the flow returns to the step S702.

If it is determined that the result of determination in step S704 is optimal, then in step S706, the final model is constructed and then ends.

On the other hand, the configuration of the present application is as follows.

(1) First, TIC, a new optimal decision tree construction algorithm that replaces the cross-check method, will be described to find an optimal decision tree construction method (step S702), and (2) a boost trap used by the present invention. (Boostrap) Examine the data extraction method (front part of step S703), (3) describe the new ensemble algorithm using it (not necessarily use the TIC algorithm, that is, select the decision tree as the basic learner). (L) from step S703 to step S706. (3) Examine the association rule generation algorithm, and (4) compare the performance of various algorithms through virtual experiments. Compare the performance of the algorithm.

2.TIC (Tree Information Criteria)

This section describes the TIC method that improves the problem of the existing cross-checking method in the problem of determining the optimal decision tree. First, an overall algorithm for generating a decision tree will be described, and after reviewing a conventional cross-check algorithm, the optimal decision tree selection algorithm using TIC proposed by the present invention will be described.

2-1. Decision Tree Construction Algorithm (Breiman et al., 1984)

The decision tree construction algorithm proposed by Breiman can be largely divided into three stages.

The first is a growth algorithm, which produces the largest decision tree for a given piece of data.

The second is a pruning algorithm, in which a plurality of nested decision trees are generated by deleting unnecessary branches in order from a huge decision tree constructed through the growth algorithm. At this time, the constructed decision trees become smaller in size.

Third, an optimal tree selection algorithm is a step of selecting an optimal decision tree among decision trees obtained by the pruning algorithm.

2-2. Cross-check algorithm for choosing optimal decision trees (k-fold cross check)

8 is a flowchart schematically illustrating a cross-check process for selecting an optimal decision tree, which will be described in detail as follows.

The decision trees generated using the growth algorithm and the pruning algorithm on the input multidimensional data are called T ₁ , ..., T _m , and e _i is the cross-check error of T _i .

(1) Step S801: Initialize various variables. That is, e _i = 0, i = 1, 2, ..., m.

(2) Step S802: Divide the given n learning materials by k equal parts _k to generate _k D mutually betrayed materials D ₁ , D ₂ , ..., D _k .

(3) Step S803: D _i is used as test data and the remaining data is used as learning data.

(4) Step S804: Construct decision trees (using growth and pruning algorithms) nested using the learning materials.

(5) Step S805: A prediction error is obtained for each of the constructed decision trees by using the test data D _i .

(6) Step S806: From the constructed decision trees, the decision tree closest to the decision tree T _j is selected. At this time, the algorithm to select is described in detail in Breiman et al. (1984), and will be omitted here.

(7) Step S807: Add e _j to the prediction error of the decision tree found in step S806.

(8) Step S808: j = 1, ..., m times repeated.

(9) Step S809: i = 1, ... are repeated k times.

(10) Step S810: e ₁ , ..., e _m are called decision trees of the decision trees T ₁ , ..., T _m , and the decision tree with the smallest cross confirmation error is the best decision. Choose as a crystal tree.

On the other hand, such cross-check algorithm is also called k-fold cross-check algorithm, generally 5 or 10 fold cross-check method is mainly used.

As described above, the cross-checking algorithm for constructing the optimal decision tree must build the decision tree several times. Therefore, when the data is huge, the calculation time becomes very long, and the result varies randomly depending on how the data is divided.

In order to solve problems such as the increase in the amount of computation and the instability of the result that the cross-checking method has, the present invention newly proposes an algorithm called TIC. Hereinafter, the TIC algorithm will be introduced.

2-3. TIC Algorithm for Optimal Decision Tree Selection

The purpose of the TIC algorithm is to determine the optimal tree among several tree permutations, ie, T ₁ , ..., T _m . At this time, the posterior probability of each tree is calculated, and the tree having the largest posterior probability is selected as an optimal tree.

Post-probability means the probability of each tree for a given data. That is, the posterior probability of the tree T _i is given for the given data D _n = {(y ₁ , x ₁ ), ..., (y _n , x _n )} Becomes

9 is a flowchart showing an overall overview of a method of selecting an optimal decision tree using TIC, which will be described in detail as follows.

First, when learning data which is multidimensional data is input in step S901, a decision tree of maximum size is constructed using the data in step S902. Then, in step S903, the constructed maximum size decision trees are newly generated as Nested Trees using the pruning theory.

After calculating the posterior probabilities of the respective decision trees in step S904, the decision tree having the largest posterior probability is selected in step S905, and finally, in step S906, the unified optimal decision tree is finally obtained.

The following describes in more detail how to select such an optimal decision tree.

First, we look at the general method of calculating the posterior probability.

The posterior probability is determined by the Bayesian Theorem. = cPr (D _n │ T _i ) Pr (T _i ), where Pr (D _{n │} T _i ) is the probability of the data when the model is T _i , and Pr (T _i ) is the user before viewing the data. Is a randomly determined probability, and c is Is a constant that makes

On the other hand, the purpose of calculating the posterior probability is to determine the tree with the largest posterior probability, and the constant c need not be obtained, and the following Equation 17 will be used.

Let's find Pr (D _n | T _i ).

First, since the data are independent, Equation 18 below holds.

In addition, Equation 18 may also be written as Equation 19 below.

Here, the tree model T _i is a model representing a probability structure of y _{k for} a given input x _k , so Pr (x _k │T _i ) does not depend on T _i . That is, Pr (x _{k |} T _i ) = Pr (x _k ). Therefore, Pr (y _{k |} T _i , x _k ) can be obtained to obtain Pr (D _{n |} T _i ).

On the other hand, like the constant c, Pr (x _k ) is a value commonly applied to all trees, and is not necessary to find a tree having the largest posterior probability. Therefore, when the expression is reflected by the expression, it is expressed by Equation 20 below.

remind How to obtain is as follows.

The final set of nodes Let's do it. Then, for each given h th final node, the probability of each J group Let's do it. Then, given the input variable x _k If it belongs to the h-th final node of, Equation 21 below holds.

Where y _k represents the group to which the data belongs.

Using the foregoing, Equation 22 below holds.

In this case, n _jh is the number of data belonging to the group j among the data included in the h-th last node.

Probability of each final node Is an unknown variable, so remove it using the expected value. To get the expected value We need a distribution of Let's do it. Then, Equation 23 below holds.

here, Various distributions can be used, and using the general distribution, Equation 24 below holds.

In addition, using a one-way distribution, the following Equation 25 holds.

At this time, to be.

On the other hand, the daily distribution is defined as in Equation 26 below.

Prior Probability of Trees Let's look at how to determine.

Is not obtained from the data, but is entered by the user.

For TIC Is constructed as follows:

First, let's define the probability that at each given h th node it will be an intermediate node (ie, branching continues) as shown in Equation 27 below.

Where f _h is the number of ancestor nodes of a given node and is a constant Wow Is determined by the user.

Then, the probability that a given node becomes the final node is naturally determined as shown in Equation 28 below.

The prior probability of a given tree T _i under the same condition as Equation 28 is expressed as Equation 29 below.

At this time, Is the set of intermediate nodes (ie all nodes that are not final nodes).

Now, let's calculate the TIC using the above description.

When all of the above-described equations are put together, the final equation (30) is given. At this time, Equation 30 below uses a solar distribution.

Then, the logarithm of the last expression above is defined as TIC. That is, the TIC of the tree T _i is expressed as Equation 31 below.

In the case of two groups, that is, J = 2, the TIC is expressed as Equation 32 below.

At this time, Is the number of data in the second group among the data in the h-th last node.

The algorithm is terminated by applying the TIC defined as described above to each decision tree T ₁ , ..., T _m to select the decision tree with the maximum TIC as the optimal decision tree.

On the other hand, the conventional Bayesian theorem and the TIC method proposed in the present invention have the same invention in terms of the use of posterior probabilities, and there is a difference in the construction of the prior probabilities used in obtaining the posterior probabilities. And this difference greatly affects the calculation of the posterior probability. That is, in the conventional Bayesian theorem, the posterior probabilities are not calculated by the equation, and the calculation is performed using a computer, which takes much longer than the method using the cross check.

In the conventional Bayesian theorem, the method of building prior probabilities assigns probabilities to all possible trees. However, the total number of possible decision trees is enormous, so how to build prior probabilities is also very complex. And, inevitably, the number of decision trees that need to find the posterior probabilities greatly increases, which leads to a surge in computation.

However, the TIC method solves the problem of using the conventional Bayesian theorem and does not assign prior probabilities to all possible decision trees, but only to nested decision trees derived from pruning algorithms. Therefore, there is an effect that the method of building the prior probabilities is very easy and the calculation of the post probabilities is also simplified.

In other words, the prior probability building method used in the TIC method is a method of reducing the set of decision trees by using data, which is crucially different from the conventional Bayesian theorem.

In summary, the method of using TIC only needs to build a decision tree once, so the computation speed is significantly improved compared to the method of using cross validation. In addition, the results are always the same for the same data, so the reliability of the results is much better than the cross-checking method.

Table 1 below shows the simulation results of the conventional 5-fold cross-checking method and the optimal decision tree selection method using the TIC proposed in the present invention.

In other words, the present experimental data is for comparing the generation rate of a single tree using 5-fold cross-check and the generation rate of a single tree using the TIC proposed in the present invention.

Each experimental data represents the average time to generate the same number of iterations 500 times, with a computer specification of Pentium 3 900 MHz, 256 megabytes of main memory and Windows 2000 operating system.

According to Table 1 below, it can be seen that the TIC method proposed in the present invention takes only about 1/5 calculation time compared to the conventional 5-fold cross-checking method.

Meanwhile, simulation data corresponds to standard data widely known in data mining, and are data of 'Radius2', 'Interaction', 'Breast Cancer', 'Ionosphere' and 'Sonar', respectively. It is the most influential simulation data. This simulation material is detailed in UC Irvine's Machine Learning Web Site (http://www1.ics.uci.edu/~mlearn/MLRepository.html).

TABLE 1

data 5-fold cross check TIC Average travel time Average travel time radius2 235.4 msec 43.2 msec interaction 228.6 msec 43.3 msec breast cancer 128.3 msec 25.6 msec ionosphere 182.9 msec 34.4 msec sonar 259.3 msec 46.6 msec

3. Extracting Boost data

This chapter outlines the method of extracting boost data used by the present invention.

First, the overall algorithm for generating boost data is as follows.

If the original data is _n data of y ₁ , ..., y _n , the number m of the boost data to be generated is determined, and then the new data set D ^B is initialized to the empty set.

And, using a random number generator After creating the integer k, To And assign it to To In addition, repeat this process i = 1, ..., m times.

10 is a flowchart illustrating a process of generating boost strap data according to an embodiment of the present invention.

(1) Step S1001: Weights of w ₁ , ..., w _n are assigned to the _n multidimensional data (x ₁ , y ₁ ), ..., (x _n , y _n ) to be input. Where x _i is the explanatory variable in p dimension. That is, x _i = (x _1i , ..., x _pi ).

(2) Step S1002: The cumulative weight of weights of w ₁ , ..., w _n assigned to y ₁ , ..., y _n is calculated. That is, the new cumulative weight is calculated as in Equation 33 below.

(3) Step S1003: The number m of new data to be created is determined. The number m of boost strap data for the ensemble algorithm according to the present invention is n.

(4) Step S1004: Initialize the new data set, D ^B, into an empty set.

(5) Step S1005: Using a random number generator Generates a real random number that satisfies

(6) step S1006: , ..., medium, Determine j satisfying At this time, j = 1, ..., n.

(7) step S1007: To be assigned.

(8) step S1008: Weight corresponding to It is done.

(9) step S1009: To Add to

(10) Step S1010: i = 1, ... is repeated m times.

4. Background and Basic Principle of Ensemble Algorithm

4-1. Preface

This chapter describes the background and basic principles of the new ensemble algorithm proposed in this application. The ensemble algorithm according to the present invention will be referred to as a cam algorithm (CHEM: Convex Hull Ensemble Machine).

The cam algorithm is an ensemble algorithm that creates a new learner using several basic learners.

For classification problems (when the response variable is categorical) or regression models (when the response variable is continuous), the learning problem can be turned into a function estimation problem.

In the classification problem where the response variable has J categories, the problem is to estimate the J-dimensional function F = (F ₁ , ..., F _J ), where the function F is defined as in Equation 34 below. .

Also, if the problem is a regression model It becomes a problem of estimating.

The true value of function F If we say (true learner), Let be a set of basic learners used in the cam algorithm. That is, set the optimal basic learner for a given learning material Choose one of the Meanwhile, a method of finding an optimal basic learner is well known in the art.

Set of basic learners used for data mining Decision trees and neural network models are used. However, a major problem with these basic learners is that they are very sensitive to changes in data. The cam algorithm proposed in the present invention is to identify and overcome the cause of such instability of the basic learners.

The causes of instability of the conventional basic learners are as follows.

4-2. Causes of instability of conventional basic learners

The reason why the various algorithms used in data mining are very unstable is the basic set of learners. Many different learners inside explain the material similarly.

A more fundamental reason why different learners describe the material similarly Set of basic learners with consideration Because it is not included. Also, Among the learners included in This is because there are several learners with a small distance from each other.

FIG. 11 is a basic conceptual view illustrating that a plurality of learners having a small distance from a true learner are included in a conventional set of basic learners.

As shown in FIG. Many learners It is the shape that surrounds. In this case, even a slight change in the material may significantly change the optimal learner. That is, the material Depending on which direction it is from, the optimal learner , And Will be either.

Set of basic learners Is located as shown in Fig. 11, no matter how well constructed the best learner Cannot be built properly. However, combining multiple learners Can be built. The reason is that, end This is because they are located in the convex hull space of the learners included in the. In particular, in Fig. 11, the following equation (35) holds when the appropriate weights w ₁ , w ₂ and w ₃ are obtained.

Looking at the meaning of [Equation 35], To It can be obtained as the weighted average of several learners belonging to. The cam algorithm proposed in the present invention is an algorithm developed using this idea.

Hereinafter, the basic principle of the learner combining method according to the cam algorithm proposed in the present invention will be described in detail.

4-3. Knowing the true learner, the basic principle of how to combine learner according to cam algorithm

This section is a true learner Know this end When not included in, Weighted average of different learners Introduce how to build it. In the next section, If the image is unknown, Introduce how to estimate.

The basic assumption of the cam algorithm is end M learners included in It is expressed as a weighted average. This is shown in Equation 36 below.

Cam algorithm is used for weighted average And weights This algorithm searches sequentially. K learners built from Cam algorithm And weights (K + 1) th learner using And weights The principle of the algorithm to find step by step is as follows.

First, of the learners orthogonal to the current ensemble model F _k according to [Equation 36], the best learner Secondly, a new ensemble model is generated by Equation 37 below, with weights Wow silver And true learner Find the minimum distance between and.

This algorithm is described in more detail as follows.

(1) Build your first learner

12A is a diagram conceptually illustrating a method of constructing a first learner according to a cam algorithm proposed in the present invention.

As shown in Fig. 12A, an optimal learner (In other words, And the learner that is located closest to.

(2) Build a second learner

12B is a diagram conceptually illustrating a method for constructing a second learner according to the cam algorithm proposed by the present invention.

Optimal learner orthogonal to Find the ensemble model F ₂ according to Equation 37 Shortest weight with and Obtain At this time, and Are each and to be. Meanwhile, the ensemble model F ₂ according to Equation 37 is expressed as Equation 38 below.

Where d ₁ is Wow Distance from d ₂ Wow Is the distance from

(3) Build a third learner

12c conceptually illustrates a method of constructing a third learner according to the cam algorithm proposed in the present invention.

Optimal learner orthogonal to and Obtain And, in the ensemble model F ₃ according to [Equation 37], Weight closest to distance Wow Obtain Meanwhile, the ensemble model F ₃ according to Equation 37 is expressed as Equation 39 below.

Then, Equation 40 below holds.

(4) building the m-th learner

12D is a diagram conceptually illustrating a method of building an m-th learner according to the cam algorithm proposed in the present invention.

By repeating the above algorithm, the m th ensemble model is obtained as shown in Equation 41 below.

4-4. Fundamental principle of how to combine learner according to cam algorithm when true learner is unknown

In all learning questions Is unknown, instead n learning resources Is given. The purpose of learning is to use materials Effectively estimating This section describes how the algorithm described in the previous section is constructed given the data.

Let l be the given loss function and given the learner The division of is defined by Equation 42 below.

For categorical data, only two groups are considered.

(1) Build your first learner

Optimal learner Obtain

(2) Build a second learner

Orthogonal and find the best learner.

First, we use the residuals to find orthogonal learners. The residual r _i can be obtained from Equation 43 below when the response variable is categorical.

Where P ₁ is the probability that y is 1 using the first ensemble model.

When the response variable is continuous, it can be obtained by Equation 44 below.

For categorical data, | Optimal learner with weight | Build it. If continuous Optimal learner using as a response variable Obtain

On the other hand, the reason for finding the optimal learner using the residual is that the residual in the regression model is orthogonal to the response variable. Therefore, learner which is most suitable for residual Is the best learner for the response variable Is almost orthogonal to

Next, the best learner using the residuals After finding Constant to minimize Obtain And, = Place it. At this time, the ensemble model is determined by Equation 37. That is, the following equation (45) is obtained.

,

Meanwhile, statistically theoretical Is approximately Wow Squared with.

(3) Build a third learner

Building a third learner is how to build a second learner instead The same is true except to find the residual using. Obtained Third Learner The ensemble model F ₃ is given by Equation 46 below.

,

(4) The above algorithm is repeated repeatedly to obtain the m th ensemble model F _m as shown in [Equation 47] below.

5. Cam Algorithm in Two Group Classification Problems

The cam algorithm proposed by the present invention in two group classification problems is described in more detail as follows.

13 is a flowchart showing an outline of a cam algorithm according to an embodiment of the present invention.

First, in step S1301, after initializing various variables, in step S1302, the input multi-dimensional data is generated as a boost strap data using a weight. Then, in step S1303, the probability that the response variable is 1 with respect to the given explanatory variable is estimated using the basic learner, and then in step S1304, a correction parameter for minimizing the given division is calculated.

In step S1305, a modified learner is constructed using the corrected parameters, and then in step S1306, an ensemble model is constructed based on the corrected learner.

In step S1307, it is determined whether or not the stop rule described later is satisfied. If not, the process returns to the step 1302, and if it is satisfied, the process ends.

The cam algorithm will be described in more detail using equations and the like.

(1) Step 1: Set the response variable y _i to 1 if the data belongs to group 2 and to 0 if it belongs to group 1.

(2) Second step: initialize various variables. That is, n weights w ₁ , ..., w _{n are set} to w _i = 1 / n and F (x) = 0.

(3) Third step: Using the weight {w _i } to generate the boost data (x ₁ ^B , y ₁ ^B ), ..., (x _n ^B , y _n ^B ). The generation of boost strap data has already been described above.

(4) Stage 4: The probability that the response variable will be 1 for the given explanatory variable x using the boost strap data Is estimated using the basic learner.

(5) Step 5: A correction parameter for minimizing [Equation 48] below for a given loss function l Calculate

At this time, to be.

(6) Step 6: Rebuild the learner by using the modified parameters. When this process is expressed by an equation, it is expressed as Equation 49 below.

Place it.

Obtain as

Update with

Weights Update to.

At this time, to be.

(7) Step 7: Repeating the third to sixth steps m = 1, ..., M times to finally build the ensemble model. The final ensemble model is set to H (x) = F (x) and assigned to group 2 if H (x)> 0 for the new response variable x and to group 1 if H (x) <0.

Meanwhile, various loss functions may be used as the loss function, but an exponential loss function or a negative log-likelihood loss function may be used to obtain a better result.

Equation 50 below is an exponential loss function, and Equation 51 represents a log likelihood loss function.

In addition, in the present invention, a basic learner using a weight may be generated without using a boost strap. However, in most cases, using boost straps is much better.

In addition, a variety of basic learners used in the present invention can be used, in this embodiment a decision tree was used. Unlike conventional boosting algorithms, the basic learners are strong learners in the cam algorithm. Therefore, instead of using a simple decision tree, we use an optimal decision tree that goes through the entire process of constructing a decision tree. At this time, TIC is used to overcome the computational problems. This has already been described above.

An algorithm that extends the cam algorithm to two or more classification problems is as follows.

6. Cam Algorithm in Multiclass Classification Problems

Even in the case of the cam algorithm extended to the multi-class, the overview follows the process of FIG. However, the applied formulas and the like are slightly different from the cam algorithms in the two group classification problems, which will be described in detail as follows.

(1) Step 1: Initialize various variables. That is, weight , i = 1, ..., n, j = 1, ..., J, F _j (x) = 0, j = 1, ..., J.

(2) Step 2: If y-th data is included in j group, set y _i ^* to 1, otherwise to 0.

(3) Step 3: _Booststrap data (x ₁ ^B , y ₁ ^{* B} ), ..., (x _n ^B , y _n ^{* B} ) using weights {w _1j , ..., w _nj } Create The generation of boost strap data has already been described above.

(4) Stage 4: The probability that the response variable will be 1 for the given explanatory variable x using the boost strap data Is estimated using the basic learner.

(5) 5th step: Place it.

(6) Sixth step: repeating the second to fifth steps j = 1, ..., J times.

(7) Step 7: Corrected parameter that minimizes Equation 52 below for a given loss function l Calculate

At this time, ego, to be.

(8) Eighth Step: The learner is newly modified by using the modified parameters. If this process is expressed by an equation, Equation 53 is given below.

Place it.

Obtain as

Update with At this time, to be.

Weights Update to.

At this time, ego, Is 1 if the i th observation belongs to the j th group, or 0 otherwise.

(9) Ninth Step: The second to eighth steps are repeated m = 1, ..., M times to finally construct an ensemble model. In this case, for the new explanatory variable x Assign to a group.

The algorithm that extends the cam algorithm to the continuous variable problem is as follows.

7. Cam Algorithm for Continuous Variable Problems

This chapter describes an algorithm (Regression CHEM) that creates an ensemble model when the response variable is continuous. The case where the response variable is continuous is called a regression model. The basic model is as follows.

14 is a flowchart schematically illustrating an ensemble algorithm for a continuous variable according to an embodiment of the present invention, which will be described in detail as follows.

First, in step S1401, various variables are initialized and response variables are defined. In step S1402, a regression model is constructed using a basic learner. Then, in step S1403, after calculating a correction parameter that minimizes the given division, in step S1404, a new response variable is updated based on the correction parameter, and in step S1405, an ensemble model is generated based on the updated response variable. Build.

In step S1406, it is determined whether or not the stop rule described later is satisfied. If not, the process returns to step S1402, and when it is satisfied, the process ends.

This will be described in more detail as follows.

First, suppose that _n multidimensional data to be input are learning data (x ₁ , y ₁ ), ..., (x _n , y _n ). Where x _i is an explanatory variable of p dimension, that is, x _i = (x _1i , ..., x _pi ) and the response variable is y _i .

The purpose of this algorithm is to find the relationship that best describes the response variable as the explanatory variable using n learning materials. In other words, we use the training materials to create the optimal function H: R ^p → R. And given the new explanatory variable x, we estimate the response of this data as H (x).

On the other hand, in statistical theory, to be. In other words, estimating conditional expectation is the purpose of regression analysis.

The cam algorithm for continuous variables is

(1) Step 1: Set a new response variable z _i = y _i .

(2) Step 2: Regression model with response variable z _i and explanatory variable x _i Obtain using the basic learner.

(3) Step 3: A correction parameter that minimizes Equation 54 below for a given loss function l Find it.

(4) fourth step: the correction parameters Use to update the new response variable. This is shown in Equation 55 below.

Place it.

Calculate

Update to.

New response variable Update to.

(5) 5th step: repeating said 2nd step-4th step m = 1, ..., M times.

(6) Construct the final ensemble model as H (x) = F (x).

Meanwhile, the loss function l is or Various known loss functions can be used.

8. Stop Rules

This chapter describes the algorithm that determines the number of basic learners required for the final ensemble model. The basic idea behind this stop rule is to stop updating the ensemble and stop the entire algorithm when the current ensemble model's division is the smallest.

The stop rule is described as follows.

(1) Step 1: Determine a positive integer K value.

(2) Step 2: F _m is called an ensemble model constructed with the first m basic learners, and for the given loss function l, the deviation of the ensemble model F _m is calculated according to Equation 56 below. do.

(3) Third step: , Put it on, The ensemble model F _m is defined as the final ensemble model for the first m that satisfies, and the algorithm is stopped.

On the other hand, the positive integer K is a value defined by the user.

[Table 2] below is a simulation data comparing the predictive power when the stop rule is not applied to the cam algorithm and when it is applied to various data. As shown in Table 2 below, the two predictive powers are almost the same. Therefore, when the stop rule is applied, an optimal ensemble model can be constructed using a small number of basic learners, and this results in a large improvement in the calculation speed.

TABLE 2

9. Association rule generation algorithm

9-1. Preface

This chapter describes algorithms to find various kinds of association rules that can be used to describe data using the basic learner used to create the final ensemble model.

On the other hand, the association rule generation algorithm presented in this chapter can be applied not only to the ensemble model constructed by the cam algorithm proposed in the present invention but also to a conventional ensemble model construction method.

9-2. Association rule generation algorithm

The algorithm for finding the association rule is

15 is a flowchart showing an outline of an association rule generation algorithm.

(1) Step S1501: Initialize various variables. That is, if the response variable is categorical data, determine the group of interest as g, the minimum allowable data number as m, and the minimum allowable reliability p. In addition, if the response variable is continuous, the region of interest (g _L , g _U ) is determined instead of the group g.

(2) Step S1502: Construct the total set S of the basic rules. This will be described in more detail as follows.

The basic learner is searched and selects all nodes of all the basic learners used in the ensemble with more than m and more than p probability of group g.

At this time, in the case of two group classification problems, since there are as many basic learners as the ensembles, all the basic learners are searched and all nodes matching the conditions are selected.

If the classification problem is more than two groups, there are basic learners multiplied by the number of ensembles and the number of groups, and among all of them, all nodes satisfying the condition are selected using the basic learners corresponding to the number of ensembles of interest group g. .

The node selected in the above-described method is the total set S of the basic rules.

(3) Step S1503: For all nodes selected according to the condition, nodes higher than the corresponding node are removed from the set of rules. This will be described in detail as follows.

All rules For (i = 1, ..., N), repeat the following N times.

If s _k nodes up to k = 1, ..., N for the selected s _i are upper nodes of s _i , then remove s _k from S.

(4) Step S1504: The reliability for all the rules included in S is calculated. In this case, when the reliability of each condition is n, the total number of data belonging to the rule is n, and the number of data belonging to the group of interest (that is, group g) is n _g , and the reliability is calculated as n _g / n.

(5) Step S1505: Arrange the rules of the calculated S set. In other words, the calculated S set rules are sorted in order of high reliability to low reliability. In this case, the sorted rules are referred to as o ₁ , ..., o _H.

(6) Step S1506: Assume that the association rule set is R. If R is empty, add o _h to set R, and if R is not empty, compare the similarity with o _h for all the association rules contained in set R. If not similar to all the rules contained in R, add node o _h to R.

Meanwhile, the similarity comparison method is as follows.

The two rules o and r given for the explanatory variable x = (x ₁ , ..., x _p ) are defined as in Equation 57 below.

At this time, x _i is the set of data D _o, x _i included in R _r D _oi Let the set of data contained in R _ri. In this case, R _oi and R _ri are a subset of R.

First, maximum allowable similarity After determining The number of materials included in If dividing by the number of data contained in is greater than or equal to s, two conditions o and r are determined to be similar for x _i , and if less than s, two conditions o and r are not similar to x _i . Determine. This process is repeated i = 1, ..., p times.

And if it is determined that the results of the similarity determination are all similar for all x _i , it is determined that rule o and r are similar, and if it is determined that no x _i is not similar, o and r are not similar. It is determined that it is not.

(7) Step S1507: Data are interpreted in order of reliability using all the rules included in the association rule set R. FIG.

9-3. Performance Experiment of Association Rule Generation Algorithm

This section describes experimental data to show the interpretation power of the association rule generation algorithm. As a contrasting prior art, the CART algorithm was selected, and one decision tree constructed in CART was applied to the association rule generation algorithm. In addition, the actual data were tested using German data.

The results of the association rules of the actual data are summarized in [Table 3] and [Table 4] below.

[Table 3] Search Results of Association Rules Using CART

[Table 4] Search results applying the association rule algorithm to the cam algorithm

German data consists of 700 good and 300 bad credit customers, based on 1,000 credit transactions, and is analyzed using the minimum number of allowable data and the minimum acceptable reliability under the same conditions for accurate comparison of association rules. It was. The minimum allowable number of data for analyzing bad credit customers 'data was 50 people (5%) and the minimum allowable reliability was 50%. The minimum allowable data for analyzing good credit customers' data was 50 people (5%). The minimum acceptable reliability was 85%.

The relevance rule search result of CART was searched by 1 bad credit customer group and 1 good credit customer group. Appeared as a customer group.

It can be seen that the relevance rule of the cam algorithm corresponds to an extensive search including the relevance rule of CART, and the result of the cam algorithm finds an association rule that includes the result of the CART. In addition, since the data corresponding to the association rule retrieved by the CART is generated from one basic learner, the data is composed of mutually exclusive data, whereas the association rule algorithm proposed in the present invention includes data included in the rule. It appears as a set, not a betrayal.

In other words, the cam algorithm finds association rules by using several basic learners, and thus, it is possible to find a wide variety of association rules, compared to CART, which relies on one basic learner. You can do it.

10. Performance Comparison of Ensemble Techniques

This chapter compares the performance of various ensemble techniques through various experiments.

10-1. Performance comparison through virtual experiments

The following model is used for the virtual experiment.

(1) Model 1: Radius 2

The number of learning materials is 1,000, the number of test materials is 5,000, and the number of groups is two.

Explanatory variables: x = (x ₁ , ..., x ₁₀ ), each of which is independent and follows a standard normal distribution.

Reaction Variables: If so, the probability is 0.9 and y is 1, and the probability is 0.1 and y is -1. Also, If so, the probability is 0.9 and y = -1, and the probability is 0.1 and y = 1. Where c is Constant that satisfies

(2) Model 2: Interaction

The number of learning materials is 1,000, the number of test materials is 5,000, and the number of groups is 2.

Explanatory variables: x = (x ₁ , ..., x ₁₀ ), each of which is independent and follows a standard normal distribution.

Reaction Variable: The response variable Is a 0-1 variable, and F (x) is expressed by Equation 58 below.

(3) Model 3: Two Normal

The number of learning materials is 1,000, the number of test materials is 5,000, and the number of groups is two.

The first 500 data belong to group 1 and the explanatory variables x = (x ₁ , ..., x ₁₀ ), each of which is independent and follows a standard normal distribution. The remaining 500 data belong to group 2, with explanatory variables x = (x ₁ , ..., x ₁₀ ), each following a normal distribution with independent, average 0, and variance 2.

(4) Model 4: Simple Quadratic

The number of learning materials is 1,000, the number of test materials is 5,000, and the number of groups is two.

The explanatory variable x follows the standard normal distribution.

The response variable is a 0-1 variable according to Equation 59 below, and is given by F (x) = -x ² + 2.

16A-16D are graphs showing the results of these four virtual experiments.

As can be seen from Figures 16a to 16d, it can be seen that the cam algorithm proposed in the present invention operates very stably. In particular, in Model 4, all other ensemble algorithms perform poorly as the number of trees increases, but this problem does not occur at all for the cam algorithm.

Table 5 below is a table representing the results of FIGS. 16A to 16D numerically.

[Table 5] Results of the virtual experiment

On the other hand, it can be seen that in most models, the predictive power of the cam algorithm is superior to the conventional ensemble algorithm. Comparing Deviance as well as predictive power, it can be seen that the cam algorithm is much better than the conventional algorithm. That is, in the conventional algorithms other than the cam algorithm, the division continues to increase. This means that the function estimation is rarely done.

In contrast, it can be seen that the deviation value in the cam algorithm is stably output. A good predictive power, but an increase in division, is found in all ensemble algorithms other than the Cam algorithm, which means that the boundary between the two groups can be found in the classification problem, but all other information is lost.

For example, for the new explanatory variable x, the probability that the response variable y belongs to the k-th group is not estimated in all ensemble algorithms other than the cam algorithm.

In conclusion, our experimental results show that the cam algorithm proposed in the present invention has not only superiority of predictive power but also stability superiority.

10-2. Comparison through analysis of actual data

The cam algorithm is compared with the existing algorithm by analyzing several real data. The actual data used was obtained from the above-mentioned 'Machine Learning Web Site' (http://www1.ics.uci.edu/~mlearn/MLRepository.html) of UC Irvine.

[Table 6] below is a chart showing information about actual data.

[Table 6] Information on the actual data

17A to 17I are graphs showing the results of actual data analysis.

In most cases it can be seen that the cam algorithm works very reliably. In Ionospher data, it works a bit badly, but the difference is not so great. In case of 'German data', the cam algorithm is more predictive and stable than other algorithms. As in the above-described virtual experiment, it can be seen that the deviation value has the smallest cam algorithm.

Table 7 below is a table numerically expressing the results of FIGS. 17A to 17I.

[Table 7] Actual data analysis results

In conclusion, the cam algorithm is very stable (not too badly predicted from any data), with very good predictive power, and function estimation (i.e. small division) is possible.

Function estimation means estimation of probability, and unlike the conventional ensemble algorithm, the cam algorithm can probabilistically represent the value of a response variable in a given data. This can be very useful in analyzing actual data.

In addition, Table 8 below is a chart showing the experimental results of the continuous variable.

[Table 8] Result of continuous variable analysis

As the conventional algorithm shown in Table 8, the LS boost algorithm was used.

The Friedman model is a virtual model with 500 training materials and 5,000 test data. Also, the explanatory variables x = (x ₁ , ..., x ₁₀ ), which are independent of each other and follow an even distribution at [0, 1]. In addition, the response variable ego, to be.

The actual data were used to examine the impact of various environmental variables on house prices in the Boston area (Boston Housing Data). These data are also readily available on the Internet.

The number of data was 506, and 5 fold cross check was used to find the test error. Results of hypothetical experiments and Boston house data analysis show that the cam algorithm works well for regression models as well as classification problems. In particular, in the conventional LS boosting method, the user needs to define a reduction parameter, but there is almost no user specification in the cam algorithm.

While the invention has been described above based on the preferred embodiments thereof, these embodiments are intended to illustrate rather than limit the invention. It will be apparent to those skilled in the art that various changes, modifications, or adjustments to the above embodiments can be made without departing from the spirit of the invention. Therefore, the protection scope of the present invention will be limited by the appended claims, and should be construed as including all such changes, modifications or adjustments.

As described above, the present invention has the following effects.

First, the proposed cam algorithm has much better predictive power than the conventional ensemble construction algorithm and works very stably. In other words, the possibility of overfitting problems is significantly reduced.

Secondly, the proposed cam algorithm overcomes the degradation of the analysis power that the conventional ensemble construction method has in common, and shows the superior analysis power than any data mining technique by using the association rule algorithm.

Third, the cam algorithm is naturally applied even in the continuous variable problem, so that it can be easily applied in general industrial fields.

Claims (78)

In the data mining model building apparatus using an ensemble model, Includes M ensemble model building means, The first ensemble model building means constructs the first learner from the input multidimensional training material, and constructs the constructed first learner itself as the first ensemble model; The k th ensemble model construction means receives a k-1 ensemble model that is the result of the k-1 ensemble model construction means, constructs an orthogonal k th learner in a multidimensional space, and constructs the k th learner and the k-1 th An apparatus for constructing data mining models using an ensemble model, comprising constructing a k-th ensemble model by combining an ensemble model. At this time, M is an integer of 2 or more, and k is an integer of 2 or more and M or less. The method of claim 1, The k th ensemble construction means, And an orthogonal learner is constructed in a multidimensional space using residuals of the k-1 th ensemble model, and then a k th ensemble model is constructed using the k-th ensemble model. The method of claim 2, k + 1th learner building means, k th learner Take the input, Constant to minimize After obtaining An apparatus for constructing a data mining model using an ensemble model, comprising constructing a k + 1 th learner using a. At this time, d is division and k is an integer of 2 or more and M-1 or less. The method of claim 2, If the response variable is categorical, data mining model building apparatus using the ensemble model characterized in that the residual is determined by the following [Equation 1]. [Equation 1] Here, the input training materials are (x ₁ , y ₁ ), ..., (x _n , y _n ), r _i is the i th residual, and P _k-1 is the k-1 th ensemble model. is the probability that y will be 1. The method of claim 4, wherein An apparatus for constructing data mining models using an ensemble model, comprising: building an optimal learner using the absolute value of the residual as a weight. The method of claim 2, If the response variable is continuous, the data mining model building apparatus using the ensemble model characterized in that the residual is determined by the following [Equation 2]. [Equation 2] Where F _k-1 is the k-1 th ensemble model. The method of claim 6, An apparatus for constructing a data mining model using an ensemble model, wherein the optimal learner is constructed using the residual as a response variable. The method of claim 1, The k-th ensemble model building means, An apparatus for constructing a data mining model using an ensemble model, wherein the division is calculated using a loss function, and then the k th ensemble model is constructed using the loss function. The method of claim 8, The k-th ensemble model building means, An apparatus for constructing data mining models using an ensemble model, wherein the division is calculated according to Equation 3 below. [Equation 3] here, Is the division and l is the loss function. The method of claim 9, The k-th ensemble model building means, k-1st ensemble model F _k-1 and kth learner Data mining model building apparatus using the ensemble model, characterized in that by constructing the k th ensemble model F _k by [Equation 4] below. [Equation 4] Where w _k is And u _k-1 is to be. The method of claim 1, and a stop means for constructing a final ensemble model with m learners, when m learners are constructed, and if the given loss function is the smallest, the apparatus for constructing a data mining model using the ensemble model. The method of claim 11, The stop means, An apparatus for constructing a data mining model using an ensemble model, wherein the ensemble model F _m is constructed as a final ensemble model for the first m satisfying Equation 5 below. [Equation 5] , Put it on, Find the m. At this time, Where l is a loss function and K is a predetermined constant. In the data mining model building method using an ensemble model, Including the M ensemble model building phase, The first ensemble model building step may include: constructing the first learner from the input multidimensional learning material, and constructing the constructed first learner itself as the first ensemble model; In the k-th ensemble model construction step, the k-1 th learner, which is the result of the k-1 th ensemble model construction step, is input, constructs an orthogonal k th learner in a multidimensional space, and constructs the k th learner and k. A method of constructing a data mining model using an ensemble model, comprising constructing a k-th ensemble model by combining a -1 th ensemble model. At this time, M is an integer of 2 or more, and k is an integer of 2 or more and M or less. The method of claim 13, The k-th ensemble model building step, A method of constructing a data mining model using an ensemble model, comprising constructing an orthogonal learner in a multidimensional space using residuals of a k-1 th ensemble model, and then constructing a k th ensemble model using the same. The method of claim 14, The k + 1th ensemble model building phase k th learner Take the input, Constant to minimize After obtaining A method of constructing a data mining model using an ensemble model, comprising constructing a k-th learner using a. At this time, d is division and k is an integer of 2 or more and M-1 or less. The method of claim 14, If the response variable is categorical, the data mining model construction method using an ensemble model characterized in that the residual is determined by the following [Equation 6]. [Equation 6] Here, the input training materials are (x ₁ , y ₁ ), ..., (x _n , y _n ), r _i is the i th residual, and P _k-1 is the k-1 th ensemble model. is the probability that y will be 1. The method of claim 16, A data mining model building method using an ensemble model, characterized in that to construct an optimal learner using the absolute value of the residual as a weight. The method of claim 14, If the response variable is continuous, the data mining model construction method using an ensemble model characterized in that the residual is determined by the following equation (7). [Equation 7] Where F _k-1 is the k-1 th ensemble model. The method of claim 18, A data mining model building method using an ensemble model, characterized in that to construct an optimal learner using the residual as a response variable. The method of claim 13, The k-th ensemble model building step, A method of constructing a data mining model using an ensemble model, wherein the division is calculated using a loss function, and then the k th ensemble model is constructed using the loss function. The method of claim 20, The k-th ensemble model building step, A method of constructing a data mining model using an ensemble model, wherein the division is calculated according to Equation 8 below. [Equation 8] here, Is the division and l is the loss function. The method of claim 21, The k-th ensemble model building step, k-1st ensemble model F _k-1 and kth learner Data mining model building apparatus using the ensemble model, characterized in that by constructing the k th ensemble model F _k by the following equation (9). [Equation 9] Where w _k is And u _k-1 is to be. The method of claim 13, When the m learners are constructed, if the given loss function is the smallest, the method further includes terminating the entire algorithm in the m th step and constructing a final ensemble model with the m learners. How to Build a Mining Model. The method of claim 23, A method for constructing a data mining model using an ensemble model, comprising constructing an ensemble model F _m as a final ensemble model for the first m satisfying Equation 10 below. [Equation 10] , Put it on, Find the m. At this time, Where l is a loss function and K is a predetermined constant. In the data mining model building apparatus in two groups classification using an ensemble model, Input means for receiving learning materials that are multidimensional data; Weight calculation means for calculating a weight using a residual of the ensemble model; Probability estimating means for estimating a probability that the response variable becomes 1 with respect to the explanatory variable x given from the data input by the input means using a basic learner and the weight; Correction parameter calculation means for calculating a correction parameter that minimizes a given loss function with respect to the probability estimated by the probability estimation means; Learner updating means for updating and rebuilding a learner using the calculated correction parameter; And And an ensemble model building means for finally building an ensemble model based on the updated learner. The method of claim 25, Two group classification using the ensemble model further comprises generating the input training data as boost data by using the weight and then transmitting the boost data to the probability estimating means. Mining model building device in The method of claim 25, The correction parameter calculating means is a correction parameter that minimizes the following Equation 11 for a given loss function l. Apparatus for constructing a data mining model in two group classifications using an ensemble model, characterized in that for calculating. [Equation 11] At this time, ego, Is the probability that the response variable will be 1 for the given explanatory variable x. The method of claim 27, And the learner updating means updates the learner according to the following [Equation 12]. [Equation 12] here, Is the correction parameter. The method of claim 28, And the ensemble model building means constructs an ensemble model according to Equation 13 below. [Equation 13] Obtain as Update with The method of claim 29, The weight calculation means is a data mining model building apparatus for classification of two groups using the ensemble model, characterized in that by updating the weight according to the following equation (14). [Equation 14] Weights Update to. At this time, to be. The method of claim 25, The learner is a data mining model building apparatus in two groups classification using an ensemble model, characterized in that the decision tree (Decision Tree). The method of claim 25, The learner is a data mining model building apparatus in two groups classification using an ensemble model, characterized in that the neural network model. The method of claim 25, The ensemble model building means, Apparatus for constructing a data mining model in two group classifications using an ensemble model, wherein the ensemble model is finally constructed when the division of the current ensemble model is smallest. The method of claim 33, wherein An apparatus for constructing data mining models in two group classifications using an ensemble model, comprising constructing an ensemble model F _m as a final ensemble model for the first m satisfying Equation 15 below. [Equation 15] , Put it on, Find the m. At this time, , L is a loss function, K is a preset constant, Is the m-th ensemble model. In the method of building a data mining model in two group classifications using an ensemble model, A first step of receiving learning materials that are multidimensional data; Calculating a weight using a residual of the ensemble model; A third step of estimating a probability that the response variable becomes 1 with respect to the explanatory variable x given from the data input in the first step using a basic learner and the weight; A fourth step of calculating a correction parameter that minimizes a given loss function with respect to the probability estimated in the third step; And And a fifth step of updating and reconstructing the learner using the modified parameters calculated in the fourth step. 36. The method of claim 35 wherein The third step, After generating the input learning data as a boost (Boostrap) data by using a weight, using the basic learner to estimate the probability that the response variable is 1 for the given explanatory variable x using this Data mining model construction method in two group classification using ensemble model. 36. The method of claim 35 wherein The fourth step, Modification parameter that minimizes Equation 16 below for a given loss function l A method of constructing a data mining model in two group classifications using an ensemble model, comprising [Equation 16] At this time, ego, Is the probability that the response variable will be 1 for the given explanatory variable x. The method of claim 37, The fifth step, A method of constructing a data mining model in two group classifications using an ensemble model characterized by updating a learner according to Equation 17 below. Formula 17 here, Is the correction parameter. The method of claim 38, And a sixth step of constructing the ensemble model according to Equation 18 below. [Equation 18] Obtain as Update with The method of claim 39, The second step, A method of constructing a data mining model in two group classifications using an ensemble model, wherein the weight is updated and calculated according to Equation 19 below. [Equation 19] Weights Update to. At this time, to be. 36. The method of claim 35 wherein The learner is a decision tree (Decision Tree) characterized in that the data mining model building method in two groups classification using an ensemble model. 36. The method of claim 35 wherein And the learner is a neural network model. 36. The method of claim 35 wherein A method of constructing a data mining model in two group classifications using an ensemble model, wherein the ensemble model is finally constructed when the division of the current ensemble model is smallest. The method of claim 43, A method of constructing a data mining model in two group classifications using an ensemble model, comprising constructing an ensemble model F _m as a final ensemble model for the first m that satisfies Equation 20 below. [Equation 20] , Put it on, Find the m. At this time, , L is a loss function, K is a preset constant, Is the m-th ensemble model. An apparatus for constructing a data mining model in a multi-class classification using an ensemble model, Input means for receiving learning materials that are multidimensional data; Weight calculation means for calculating a weight using a residual of the ensemble model; Probability estimating means for estimating a probability that the response variable belongs to the j group with respect to the explanatory variable x given from the data input by the input means using a basic learner and the weight; Correction parameter calculation means for calculating a correction parameter that minimizes a given loss function for the probability estimated by the probability estimation means; Learner updating means for updating and rebuilding a learner using the calculated correction parameter; And And an ensemble model building means for finally building an ensemble model based on the updated learner. The method of claim 45, In the multi-class classification using the ensemble model, further comprising boosting data generation means for generating the input learning data as a boost (Boostrap) data by using a weight, and then transmitting it to the probability estimation means Data mining model building device. The method of claim 45, The correction parameter calculating means is a correction parameter that minimizes the following [Equation 21] for a given loss function l. Apparatus for constructing a data mining model in a multiclass classification using an ensemble model, characterized in that for calculating a. Formula 21 Where p _mj is the probability that the response variable belongs to j group for the given explanatory variable x. The method of claim 47, The learner updating means is a data mining model building apparatus in a multi-class classification using an ensemble model characterized in that the learner is updated according to Equation 22 below. Formula 22 here, Is the correction parameter. 49. The method of claim 48 wherein And the ensemble model building means updates the ensemble model according to the following [Equation 23]. Formula 23 Obtain as Update with At this time, to be. The method of claim 49, The weight calculation means is a data mining model building apparatus in the multi-class classification using the ensemble model, characterized in that for updating the weight according to the following formula (24). Formula 24 Weights Update to. At this time, to be. Is 1 if the i th observation belongs to the j th group, or 0 otherwise. The method of claim 45, The learner is a data mining model building apparatus in a multi-class classification using an ensemble model, characterized in that the decision tree (Decision Tree). The method of claim 45, The learner is a data mining model building apparatus in a multi-class classification using an ensemble model, characterized in that the neural network model. The method of claim 45, The ensemble model building means, An apparatus for constructing a data mining model in a multi-class classification using an ensemble model, wherein the ensemble model is finally constructed when the division of the current ensemble model is smallest. The method of claim 53, wherein An apparatus for constructing data mining models in a multi-class classification using an ensemble model, wherein the ensemble model F _m is constructed as the final ensemble model for the first m satisfying the following Equation 25. [Equation 25] , Put it on, Find the m. At this time, , L is a loss function, K is a preset constant, Is the m-th ensemble model. In the data mining model construction method in the multi-class classification using the ensemble model, A first step of receiving learning materials that are multidimensional data; Calculating a weight using a residual of the ensemble model; A third step of estimating, using the basic learner and the weight, a probability that the response variable belongs to the j group with respect to the explanatory variable x given from the data input in the first step; A fourth step of calculating a correction parameter that minimizes a given loss function with respect to the probability estimated in the third step; A fifth step of updating and rebuilding a learner using the correction parameter calculated in the fourth step; And And a sixth step of finally constructing an ensemble model based on the updated learner. The method of claim 55, The third step, After generating the input learning data as a boost (Boostrap) data by using a weight, using the basic learner to estimate the probability that the response variable is 1 for the given explanatory variable x using this Data Mining Model Construction in Multi-Class Classification Using Ensemble Model. The method of claim 55, The fourth step, Correction parameter that minimizes Equation 26 below for a given loss function l A method of constructing a data mining model in a multiclass classification using an ensemble model, comprising: Formula 26 Where p _mj is the probability that the response variable belongs to j group for the given explanatory variable x. The method of claim 57, The fifth step, A data mining model building method in a multi-class classification using an ensemble model, characterized in that the learner is updated by the following [Equation 27]. [Equation 27] here, Is the correction parameter. The method of claim 58, The sixth step, A method of constructing a data mining model in a multi-glass classification using an ensemble model, characterized by updating the ensemble model according to Equation 28 below. [Equation 28] Obtain as Update with At this time, to be. The method of claim 59, The second step, A data mining model construction method in a multi-class classification using an ensemble model, wherein the weight is updated by Equation 29 below. Formula 29 Weights Update to. At this time, to be. Is 1 if the i th observation belongs to the j th group, or 0 otherwise. The method of claim 55, The learner is a decision tree (Decision Tree) characterized in that the data mining model building method in a multi-class classification using an ensemble model. The method of claim 55, The learner is a neural network model, data mining model building method in a multi-class classification using an ensemble model. The method of claim 55, The sixth step, A method of constructing a data mining model in a multi-class classification using an ensemble model, wherein the ensemble model is finally constructed when the division of the current ensemble model is smallest. The method of claim 63, wherein A method for constructing a data mining model in a multi-class classification using an ensemble model, comprising constructing an ensemble model F _m as a final ensemble model for the first m satisfying Equation 30 below. Formula 30 , Put it on, Find the m. At this time, , L is a loss function, K is a preset constant, Is the m-th ensemble model. In the data mining model building apparatus using an ensemble model when the response variable is a regression, Input means for receiving learning materials that are multidimensional data; Residual calculation means for calculating a residual of the ensemble model; Regression model building means for constructing a regression model using a basic learner and the residual from data input by the input means; Correction parameter calculation means for calculating a correction parameter that minimizes a given loss function for the regression model constructed by the regression model construction means; Learner updating means for rebuilding a learner by updating a response variable using the calculated correction parameter; And And an ensemble model building means for finally constructing an ensemble model based on the reconstructed learner. 66. The method of claim 65, The correction parameter calculation means, Modification parameter that minimizes Equation 31 below for a given loss function l Apparatus for constructing a data mining model using an ensemble model when the response variable is continuous. Formula 31 66. The method of claim 65, The learner updating means, The above correction parameters An apparatus for constructing a data mining model using an ensemble model when the response variable is continuous by updating a new learner by using Equation 32 below and then constructing an ensemble model using the same. Formula 32 Place it. Calculate Update to. New response variable Update to. In this case, the input training data is (x ₁ , y ₁ ), ..., (x _n , y _n ), x _i is an explanatory variable of p dimension, y _i is a response variable, Is a regression model. 66. The method of claim 65, The learner is a decision tree (Decision Tree) data mining model building apparatus using an ensemble model when the response variable is continuous. 66. The method of claim 65, The learner is a neural network model, the data mining model building apparatus using the ensemble model when the response variable is continuous. 66. The method of claim 65, The ensemble model building means, An apparatus for constructing a data mining model using an ensemble model when the response variable is continuous, characterized in that the ensemble model is finally constructed when the deviation of the current ensemble model is smallest. The method of claim 70, An apparatus for constructing a data mining model using an ensemble model when the response variable is continuous, wherein the ensemble model F _m is constructed as the final ensemble model for the first m that satisfies Equation 33 below. [Equation 33] , Put it on, Find the m. At this time, , L is a loss function, K is a preset constant, Is the m-th ensemble model. In the method of constructing a data mining model using an ensemble model when the response variable is a regression, A first step of receiving learning materials that are multidimensional data; Calculating a residual of the ensemble model; A third step of constructing a regression model using a basic learner and the residual from data input in the first step; A fourth step of calculating a correction parameter that minimizes a given loss function for the regression model constructed in the third step; A fifth step of rebuilding the learner by updating the response variable using the correction parameter calculated in the fourth step; And And a sixth step of finally constructing the ensemble model based on the learner reconstructed in the fifth step. The method of claim 72, The fourth step, Correction parameter that minimizes Equation 34 below for a given loss function l A method of constructing a data mining model using an ensemble model when the response variable is continuous. [Equation 34] The method of claim 72, The fifth step, The above correction parameters A method of constructing a data mining model using an ensemble model when the response variable is a continuous form, after which the learner is updated by using Equation 35 below and the ensemble model is constructed using the learner. Formula 35 Place it. Calculate Update to. New response variable Update to. In this case, the input training data is (x ₁ , y ₁ ), ..., (x _n , y _n ), x _i is an explanatory variable of p dimension, y _i is a response variable, Is a regression model. The method of claim 72, The learner is a decision tree (Decision Tree) characterized in that the data mining model building method using an ensemble model when the response variable is continuous. The method of claim 72, The learner is a neural network model, when the response variable is a continuous type data mining model building method using an ensemble model. The method of claim 72, The sixth step, A method of constructing a data mining model using an ensemble model when the response variable is continuous, characterized in that the ensemble model is finally constructed when the deviation of the current ensemble model is smallest. 76. The method of claim 75 wherein A method for constructing a data mining model using an ensemble model when the response variable is continuous, wherein the ensemble model F _m is constructed as a final ensemble model for the first m satisfying the following Equation 36. Formula 36 , Put it on, Find the m. At this time, , L is a loss function, K is a preset constant, Is the m-th ensemble model.