CN103679267A - Method and device for constructing RBF neural network based on unmarked samples - Google Patents

Abstract

The invention discloses a method and device for constructing an RBF neural network based on unmarked samples. The method takes the sensitivity of the hidden layer neurons in the RBF neural network as a benchmark, and gradually cuts out neurons with low sensitivity in the hidden layer neurons to achieve The purpose of gradually simplifying the network structure and improving the performance of the classifier. Compared with the traditional method, in the process of calculating the sensitivity of RBF hidden layer neurons, the present invention not only uses marked samples, but also uses a large number of unmarked samples, which greatly improves the accuracy of sensitivity calculation. The device includes an initial module, a training module, a hidden layer neuron selection module and an output module, which can effectively improve the construction efficiency of the neural network and improve the performance of the RBF neural network.

Description

RBF neural network construction method and device based on unlabeled samples

technical field

The invention relates to a network construction method and a device for designing a neural network, in particular to a network construction method and a device that can effectively improve the classification efficiency or regression efficiency of a neural network, and belongs to the field of machine learning in intelligent science and technology.

Background technique

When designing the RBF neural network classifier, how to determine the structure of the neural network is an important and critical step. Constructing a suitable network for specific problems is of great help to improve classification accuracy and generalization ability. Currently, three-layer neural networks are widely used. The literature has proved that the three-layer neural network can approach any continuous function when the number of neurons in the second layer (also known as the hidden layer and the middle layer) increases. In a specific application, the number of neurons in the first layer of the three-layer neural network depends on the dimension of the input variable, and the number of neurons in the third layer depends on the dimension of the output variable. Because the dimensions of the input variables and output variables are generally known, the number of neurons in the first layer and the third layer is generally determined. The network construction of the three-layer neural network is actually a process of determining the number of neurons in the second layer.

When training neural networks, supervised learning methods are usually used. The supervised learning method refers to adjusting the network parameters by telling the network input and corresponding output in the process of training the neural network, so as to achieve the purpose of training the neural network. This process requires the use of labeled training samples. The labeling of training samples is generally done by experts, which often costs a lot of money and time. In practical applications, unlabeled samples are much easier to obtain than labeled samples. For example, in some Internet applications, the labeled samples marked by experts account for only a small part compared to the unlabeled samples. It becomes necessary to be able to use those unlabeled samples to help determine the optimal structure of the network.

For the three-layer RBF neural network, the methods for determining the hidden layer neural network mainly include:

1) Branching method. The process of determining the number of hidden neurons in this method is to first choose a small number of hidden neurons. Because the number of neurons in the hidden layer is too small, the structure of the network is too simple, resulting in the failure of training the neural network with training samples. The mathematical characterization is that the error does not converge. On the basis of the number of neurons in the hidden layer, increase the number of neurons in the hidden layer one by one, and retrain the network every time a neuron in the hidden layer is added, until the number of neurons in the hidden layer increases to a certain number, and the neural network can be trained successfully. . The minimum number of hidden layer neurons that can make the neural network training successful is the number of hidden layer neurons we need to find.

2) Branch reduction method. This method is opposite to the branching method. Its operation method is to first determine a sufficiently large number of neurons in the hidden layer to construct a three-layer neural network. Under this structure, the neural network can be easily trained using labeled samples. Then remove one of the neurons in the hidden layer, and use the marked samples for training on the basis of the removed network, so that the network is trained again. The above removal process is repeated until the network training cannot be completed. At this time, the minimum number of neurons in the hidden layer that can be completed is taken as the final number of neurons in the hidden layer. The theoretical basis behind the branching method and the branching method is that the statistical learning theory requires a classifier to have an appropriate complexity for a specific classification problem, ensuring that it is neither overfitting nor underfitting. Only such a classifier can have the best generalization ability. For a classifier such as a three-layer RBF neural network, the complexity of the network is reflected in the number of neurons in the hidden layer. If the number of neurons is too small, the network is underfitting, and the training cannot be completed. If the number of neurons is too large, the network is overfitting. , poor generalization ability.

3) Empirical method. This method to determine the number of neurons in the hidden layer requires a deep understanding of the domain involved in the specific problem, so that the number of neurons in the hidden layer can be determined empirically. Even this cannot guarantee that the number of neurons in the hidden layer is optimal.

For the above method, the branch reduction method is currently used more. In the specific branch reduction process, which hidden layer neuron is subtracted first, and which hidden layer neuron is subtracted second is very important for determining the final network structure. It is generally believed that each hidden layer neuron plays a different role or importance in the training process. In theory, removing neurons that have no effect on classification or are unimportant can make the generalization performance of the final trained neural network better. How to use unlabeled samples to help determine hidden layer neurons to better determine the network becomes very important.

Contents of the invention

Purpose of the invention: Aiming at the problems existing in the prior art, the present invention provides a method and device for constructing a RBF neural network based on unlabeled samples.

Technical solution: a method for constructing a RBF neural network based on unlabeled samples, comprising the following steps:

(S101) Select a sufficiently large positive integer m (the maximum number of m does not exceed the number of training samples) as the number of neurons in the hidden layer, construct a three-layer RBF neural network, and give initial network parameters;

(S103) Using the labeled sample set to train the neural network until the cost function converges to a given small threshold e (the value of e is less than 10 to the power of -2), and a trained classifier is obtained;

(S105) Calculate the sensitivity of neurons in the hidden layer using labeled and unlabeled samples, and sort the sensitivity from small to large;

(S107) removing the least sensitive hidden layer neurons to obtain a new structured RBF neural network;

(S109) On the basis of the original parameters, the new RBF neural network is trained again using the marked sample set. If the cost function can converge to a small threshold e, a classifier with updated parameters is obtained and the steps are repeated (S107), (S109); if it cannot converge, go to the next step;

(S111) Taking the RBF neural network structure with the smallest number of neurons in the hidden layer and capable of convergence as the final network structure, and its network is the final output classifier.

The method of the invention provides a basis for cutting hidden layer neurons to determine the network structure, and improves the classification performance of the neural network classifier.

The present invention also discloses a RBF neural network construction device based on unlabeled samples, including: an initial module, a training module, a hidden layer neuron selection module and an output module; the above-mentioned modules are sequentially constructed in the following order: initial module, training module , a hidden layer neuron selection module, and an output module.

(1) Construct the initial module: it selects a sufficiently large positive integer m as the number of neurons in the hidden layer, constructs a three-layer RBF neural network, and gives initial network parameters;

(2) Build a training module: it uses a labeled sample set to train the neural network until the cost function converges to a given small threshold e, and obtains a trained classifier;

(3) Hidden layer neuron selection module: it uses labeled samples and unlabeled samples to calculate the sensitivity of hidden layer neurons, sorts the sensitivity from small to large, and removes the least sensitive hidden layer neurons to form RBF neural network with a new structure; re-use the labeled sample set to train the RBF neural network with a new structure, if the cost function can converge to a small threshold e, get a classifier with updated parameters and repeat this step; if the cost If the function cannot converge to a small threshold e, enter the next step;

(4) Build the output module: take the neural network structure with the smallest number of neurons in the hidden layer that can converge as the final network structure, and output this network as the final classifier.

The RBF neural network construction device based on unlabeled samples of the present invention effectively utilizes the information contained in unlabeled samples, so that it is more accurate and convenient than previous methods in judging the importance of individual neurons in the hidden layer.

Compared with the prior art, the present invention has the following beneficial effects:

The RBF neural network construction method based on unmarked samples in the invention provides a basis for clipping hidden layer neurons to determine the network structure, and improves the classification performance of the neural network classifier.

The RBF neural network construction device based on unlabeled samples of the present invention effectively utilizes the information contained in unlabeled samples, so that it is more accurate and convenient than previous methods in judging the importance of individual neurons in the hidden layer. The working method of the sample RBF neural network construction device The pruned neural network has better generalization.

Description of drawings

Fig. 1 is the structural diagram of RBF neural network;

FIG. 2 is a flow chart of a method for constructing an RBF neural network based on unlabeled samples according to an embodiment of the present invention.

Detailed ways

Below in conjunction with specific embodiment, further illustrate the present invention, should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention, after having read the present invention, those skilled in the art will understand various equivalent forms of the present invention All modifications fall within the scope defined by the appended claims of the present application.

Taking the RBF neural network as an example, the sample selection method of the feedforward neural network according to the present invention is described.

The RBF neural network is a fully connected feedforward neural network, which is suitable for object classification. The structure of the RBF neural network is shown in Figure 1. It is a three-layer forward network: the input layer MA is composed of input pattern nodes, and x _i represents the i-th component of the input pattern vector (i=1,2,.. .,n); the second layer is the hidden layer MB, which consists of m nodes b _j (j=1,2,...,m). The third layer is the output layer MC, which consists of p nodes c _k (k=1,2,...,p).

Each element of the input vector needs to be normalized before training, where each element is normalized to [-1,1].

The standard BP algorithm is used here for the training of the above-mentioned RBF neural network.

Below we define the sensitivity of the neurons in the hidden layer of the RBF neural network above, and such a definition can be easily extended to other forward neural networks.

When the training of the neural network is completed, its mapping relationship is determined. Let the mapping relationship function be F(X) (where X is the input vector). Assuming that the jth hidden layer neuron is cut off, the mapping relationship function of the neural network becomes F _j (X) after removing the jth hidden layer neuron, and the sensitivity of the jth hidden layer neuron is defined as:

S _j (X)＝E(||F(X)-F _j (X)|| ² ) (1)

||·|| ² is an operator for obtaining the Euclidean norm of ·. E is the operator for obtaining expectations.

From the definition of sensitivity, it can be seen that the sensitivity actually represents the difference between the function output of the jth hidden layer neuron and the function output without the jth hidden layer neuron. The smaller the difference, the less important the jth hidden layer neuron is, and vice versa.

In the process of calculating S _j (X), S _j (X) can be transformed into:

S _j (X)＝ _∫Ω (F(X)-F _j (X)) ² p(X)dX (2)

Where Ω is the domain, and p(X) is the density function of X in the domain Ω. Generally speaking, p(X) is unknown. Due to the small number of marked samples, it is very difficult to estimate p(X) only relying on marked samples. The number of unlabeled samples is very large, and it also contains the distribution information of the samples. Here we use labeled samples and unlabeled samples to estimate p(X), which can greatly improve the estimation accuracy.

Suppose (X ₁ ,y ₁ ),(X ₂ ,y ₂ ),…(X _b ,y _b ) are labeled sample sets, X _b+1 ,X _b+2 …X _N are unlabeled sample sets , then a good estimate of S _j (X) is:

${\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

In this paper, formula (3) is used to calculate the sensitivity of neurons in the hidden layer.

FIG. 2 is a flow chart of the neural network construction method based on unlabeled samples in the present invention.

In step S101, a sufficiently large positive integer m is selected as the number of neurons in the RBF hidden layer, a three-layer neural network is constructed, and initial network parameters are given.

In step S103, the neural network is trained using the labeled sample set until the cost function converges to a given small threshold e, and a trained classifier is obtained.

In step S105, the sensitivities of neurons in the hidden layer are calculated by using the labeled and unlabeled samples, and sorted according to the sensitivity from small to large.

In step S107, the neuron in the hidden layer with the least sensitivity is removed to obtain an RBF neural network with a new structure.

In step S109, on the basis of the original parameters, the new RBF neural network is trained again using the marked sample set. If the cost function can converge to a small threshold e, the classifier with updated parameters is obtained and Repeat steps S107 and S109. If it cannot converge, go to the next step.

In step S111, the RBF neural network structure with the smallest number of neurons in the hidden layer and capable of convergence is taken as the final network structure, and its network is the final output classifier.

The method of this embodiment is based on the neural network construction device based on unlabeled samples. The device includes: an initial module, a training module, a hidden layer neuron selection module, and an output module; the above modules are sequentially constructed in the following order: initial module, training module, hidden layer neuron selection module, and output module.

(1) Construct the initial module: it selects a sufficiently large positive integer m as the number of neurons in the RBF hidden layer, constructs a three-layer neural network, and gives initial network parameters;

(2) Build a training module: it uses a labeled sample set to train the RBF neural network until the cost function converges to a given small threshold e to obtain a trained classifier;

(3) Hidden layer neuron selection module: it uses labeled samples and unlabeled samples to calculate the sensitivity of hidden layer neurons, sorts the sensitivity from small to large, and removes the least sensitive hidden layer neurons to form RBF neural network with a new structure; re-use the labeled sample set to train the neural network with a new structure, if the cost function can converge to a small threshold e, get a classifier with updated parameters and repeat this step; if the cost function If it cannot converge to a small threshold e, enter the next step;

(4) Build the output module: take the RBF neural network structure with the smallest number of neurons in the hidden layer and can converge as the final network structure, and output this network as the final classifier.

Claims (3)

1. A RBF neural network construction method based on unmarked samples, is characterized in that, comprises the following steps: (S101) Select a sufficiently large positive integer m as the number of neurons in the hidden layer, construct a three-layer RBF neural network, and give initial network parameters; (S103) Using the labeled sample set to train the neural network until the cost function converges to a given small threshold e to obtain a trained classifier; (S105) Calculate the sensitivity of neurons in the hidden layer using labeled and unlabeled samples, and sort the sensitivity from small to large; (S107) removing the least sensitive hidden layer neurons to obtain a new structured RBF neural network; (S109) Retrain the new RBF neural network on the basis of the original parameters using the marked sample set, if the cost function can converge to a small threshold e, get a classifier with updated parameters and repeat the steps (S107), (S109); if it cannot converge, go to the next step; (S111) Taking the RBF neural network structure with the smallest number of neurons in the hidden layer and capable of convergence as the final network structure, and its network is the final output classifier. 2. the RBF neural network construction method based on unmarked samples as claimed in claim 1, is characterized in that, after neural network training is finished, set mapping relationship function as F (X), X is input vector, assumes that cut out is the jth hidden layer neuron, then the mapping relationship function of the neural network becomes F _j (X) after removing the jth hidden layer neuron, and the sensitivity of the jth hidden layer neuron is defined as: S _j (X)＝E(||F(X)-F _j (X)|| ² ) (1) ||·|| ² is the operator to obtain the Euclidean norm of ·, and E is the operator to obtain the expectation; In the process of calculating S _j (X), S _j (X) can be transformed into: S _j (X)＝ _∫Ω (F(X)-F _j (X)) ² p(X)dX (2) Where Ω is the definition domain, p(X) is the density function of X in the definition domain Ω, and p(X) is estimated by using marked samples and unmarked samples together; Suppose (X ₁ ,y ₁ ),(X ₂ ,y ₂ ),…(X _b ,y _b ) are labeled sample sets, X _b+1 ,X _b+2 …X _N are unlabeled sample sets , then a good estimate of S _j (X) is: ${\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$ The sensitivity of neurons in the hidden layer is calculated by formula (3). 3. A RBF neural network construction device based on unlabeled samples, characterized in that, comprising: an initial module, a training module, a hidden layer neuron selection module and an output module; said modules are constructed in the following order: initial module, training module , hidden layer neuron selection module, output module; (1) Initial module: it selects a sufficiently large positive integer m as the number of neurons in the hidden layer, constructs a three-layer RBF neural network, and gives initial network parameters; (2) Training module: it uses the labeled sample set to train the neural network until the cost function converges to a given small threshold e, and obtains a trained classifier; (3) Hidden layer neuron selection module: it uses labeled samples and unlabeled samples to calculate the sensitivity of hidden layer neurons, sorts the sensitivity from small to large, and removes the least sensitive hidden layer neurons to form RBF neural network with a new structure; re-use the labeled sample set to train the RBF neural network with a new structure, if the cost function can converge to a small threshold e, get a classifier with updated parameters and repeat this step; if the cost The function cannot converge to a small threshold e, and the final classifier through the output module; (4) Output module: take the neural network structure with the smallest number of neurons in the hidden layer and can converge as the final network structure, and output this network as the final classifier.

Images