CN110619059A - Building marking method based on transfer learning - Google Patents

Abstract

A method for building calibration based on transfer learning, comprising the following steps: 1) constructing a training set and a test set from a self-made data set, the data set comprising a building design drawing from a relevant design institute and an online real-shot building picture, Apply it as the test data for the recommendation system; 2) Build a building calibration technology architecture model, which includes image feature calibration, image feature extraction, database matching, Mahalanobis distance setting and final image authentication; 3) The dataset is divided into small The blocks are sequentially input into the building recognition neural network and the parameters are updated by the back-propagation Adam algorithm, and the last changed classifier is trained by the set number of training rounds; 4) The trained model is calibrated to the actual building picture .

Description

A method of building calibration based on transfer learning

technical field

The invention relates to the field of image classification, in particular to a building calibration method based on migration learning

Background technique

Building recognition is also one of the research hotspots in the field of computer vision and pattern recognition. It enables people to quickly obtain the location, name, description and other related information of buildings based on images. important application value, and how to effectively calibrate is the key issue of building identification.

The key technology of building recognition lies in the extraction of features. The most common image features include color features, texture features, and shape features. Most of the established image annotation and image extraction systems are based on these features, and most of their performances are It depends on the representation method of the extracted features. For color features, texture features, and shape features, there are mainly the following widely used feature descriptors: HOG descriptor, LBP descriptor, HSV descriptor, SIFT descriptor. Using these descriptors, combined with dimensionality reduction, building pyramids, and building bag-of-words models, generate efficient feature extractors. However, traditional features have great limitations. Artificial feature engineering is time-consuming and labor-intensive and requires high professional domain knowledge. The extracted features are relatively simple, and as the complexity of the data set increases, the feature effect is worse. Therefore, the use of traditional features alone cannot meet the needs of feature extraction in the current computer vision field. In recent years, with the development of deep learning technology, the neural network structure based on end-to-end learning relies on the advantages of big data and high-dimensional parameter space to gradually abstract and synthesize advanced features from the bottom layer to the top layer. The data-driven self-learning method ensures that the convolutional neural network has excellent feature extraction ability. Therefore, combining deep learning with building calibration has practical application value.

SUMMARY OF THE INVENTION

The invention solves the shortcomings of the existing building identification technology with low precision and requires a large number of artificial feature engineering, and provides a building calibration method based on migration learning.

The invention combines the convolutional neural network of migration learning for feature extraction, and the multi-feature calibration technology for feature division, so that the model can better capture the potential correlation information between the features to improve the prediction accuracy of the model, fine-tune the network model, through The Inception network model obtains the input of the last layer, which is defined as Bottlenecks, and then uses Bottlenecks to train the last changed softmax layer, and conducts simulation experiments on the spyder visualization platform

The technical scheme adopted by the present invention to solve its technical problems is:

A method for building calibration based on transfer learning, comprising the following steps:

Step 1. Self-made data set. The building data set comes from Google pictures and Baidu pictures. Based on the existing data set, it is divided into seven types of common buildings in life, namely churches, residential buildings, hospitals, hotels, and books. Pavilions, villa housing and shopping malls, and make uniform treatment.

Step 2. Build a multi-feature calibration neural network model based on transfer learning. The entire network architecture is divided into image feature calibration, image feature extraction, database matching, and graphic authentication.

Image feature calibration: Single feature calibration is carried out for buildings with obvious architectural styles, and multi-feature calibration is carried out for various styles, and then the feature matrix of the image is formed.

Image feature extraction: The image after feature calibration is used as the network input. The multi-layer neural network learns the features of the original image by combining a series of convolutional layers and downsampling layers, and combines the classic BP algorithm to adjust the parameters and complete the weights. Update, the BP network update weight formula is:

ω(t+1)=ω(t)+ηδ(t)x(t) (1)

where ω(t) is the connection weight, x(t) is the output of the neuron, δ(t) is the error term of the neuron, η is the learning rate, and the network structure of the convolutional layer in the network adopts the discrete convolutional type, expressed as:

where M _β represents a choice of input features, k represents the convolution kernel, γ represents the number of layers of the network, and b represents the bias added by each input feature map. For a specific output map, the input map features can be applied with different volumes The kernel convolution is obtained. f represents the activation function used by the convolutional neuron, and here each block uses the ReLu activation function.

Database matching: The database here is the feature vector of each image obtained from the training data set. After network training, the feature information of each image will be saved in the label file in the form of 1024-dimensional data points. When the newly trained image feature information will be matched to the database by calculating its Mahalanobis distance.

Image authentication: After the image feature extraction module completes the feature extraction, two output results are obtained, one is the feature vector of the image, and the other is the label after image classification. The image authentication module first classifies the image to be authenticated into the corresponding category of the image library according to the existing tags of the image, and then calculates the Mahalanobis distance with other images of this category, and the result is calculated as D _i (i=1, 2, 3..., m) , take out D _min . According to the calculation result, it is judged whether the image to be authenticated comes from the database or the same source image as the image in the database. Judgment method The threshold method is used in this paper, and the threshold Th _m is set according to a large number of experimental results. If D _min >Th _m , that is, the distance between the image and all images in this class is large, and it does not belong to the database. On the contrary, the image belongs to the database, and the image corresponding to D _min is the same image or comes from the same image.

Regarding the definition of Mahalanobis distance: it is known that there are M sample vectors X ₁ ～X _m , the covariance matrix is denoted as S, and the mean value is denoted as vector μ, then the Mahalanobis distance D(X) from the sample vector X _i to μ is expressed as for:

The Mahalanobis distance between the vectors X _i and X _j is defined as:

If the covariance matrix is a unit matrix, that is, independent and identical distribution among the sample vectors, the formula becomes:

That is the Euclidean distance. If the covariance matrix is a diagonal matrix, the formula becomes the normalized Euclidean distance. Therefore, compared with the Euclidean distance, Mahalanobis distance is an effective method to calculate the similarity between two unknown sample sets. Unlike the Euclidean distance, it takes into account the relationship between various characteristics and is not affected by dimensions. This process mainly tests the accuracy of the image authentication module. Grayscale transformation and other operations, synthesizing a new data set to simulate the tampered image, and the image authentication accuracy mainly depends on the results of the previous image classification and the selection of the threshold, so the purpose of this experiment is to find a suitable threshold, and the initial value of the threshold passes Calculate the distance between the transformed image and the original image, and set it after statistical analysis. Through a lot of experimental analysis, the threshold of the Mahalanobis distance between the original image and the preprocessed transformation is about 1.2, while the Mahalanobis distance of two completely different images is basically the same. It is more than 3, so the setting of the threshold can be allowed to be rough, so we can set the threshold to about 2.0 and conduct experiments.

Step 3. Network training, the process is as follows:

The number of training rounds of the network is set to 10,000 times, the training batch-size is 100, and a round of prediction is performed every 10 times. At the same time, 80% of the total data is used as the main training sample T, and the other 10% is used as a The cross-validation sample V of V, and the remaining 10% are used as the test dataset K for predicting the performance of the classifier in the real world. The algorithm used for training is SGD (stochastic gradient descent algorithm). The input of this network is 229x229x3 in size. The convolutional layers in the block all use 3x3 small convolution kernels for parameter reduction. By migrating the GoogleNet weights from imageNet training and fine-tuning, the network starts training from the optimal point, retrains the classifier, and calculates the training set T. log loss loss function value, update the weights and biases of the multi-layer neural network according to the back-propagation algorithm.

Calculate the log loss loss function value of the verification set V to determine whether the model has converged. If the converged model training ends, otherwise enter the next batch of data on the training set T for training until the log loss loss function value on the verification set V tends to converge. Finally, the accuracy of the model is verified with the test set K.

Step 4. Calibrate the trained model to the actual building image.

The advantages of the present invention are: using the "pre-training-fine-tuning mode based on migration learning to train new classifiers and network weights, this method has high versatility and stability compared with traditional machine learning methods, and the present invention proposes here. The multi-feature calibration technology makes the feature vector passing through the bottleneck layer of the network more representative by performing artificial feature calibration on the multi-functional building, which greatly improves the accuracy of network classification.

Description of drawings

FIG. 1 is a schematic diagram of the experimental Build-7 dataset of the present invention.

FIG. 2 is a representation of a building characteristic matrix of the present invention.

Fig. 3 is a building identification neural network diagram of the present invention.

FIG. 4 is a structural diagram of the building calibration technology of the present invention.

Detailed ways

In order to better illustrate the technical solution of the present invention, the present invention will be further described below through an embodiment with reference to the accompanying drawings.

A method for building calibration based on transfer learning, comprising the following steps:

Step 1. Take the data as shown in Figure 1 in the self-made dataset Build-7 to construct a training set and a test set respectively. The data set contains the building design drawings from the relevant design institutes and the actual building pictures on the Internet, and is used as a recommendation system test. data.

Step 2. Model feature calibration part As shown in Figure 2, multi-feature calibration is performed on the image to be trained and marked in the form of a feature matrix. 1 indicates that this feature exists in this building, and -1 indicates that this feature does not exist. The feature-labeled picture is input to the network (as shown in Figure 3). The Inception module group migrated in the network is a series of multiple Inception modules. This structure combines the convolution (1x1, 3x3, 5x5), pooling and The operations (3x3) are stacked together (the same size after convolution, pooling, adding the channels), which increases the width of the network on the one hand, and the adaptability of the network to scale on the other hand. The network in the convolutional layer of the network can extract every detail of the input, and the 5x5 filter can also cover most of the input of the receiving layer. For the convolutional layer, hundreds or thousands of convolution kernels are usually used to generate a large number of Feature maps are used to capture various features of an image, therefore, the local activation values of these feature maps can be aggregated as feature vectors to build more discriminative representations than hand-extracted descriptors and features directly extracted from fully connected layers of CNNs.

Step 3. The labeling technology framework of the present invention is shown in Figure 4. The data set is divided into small blocks and input into the building recognition neural network in turn, and the parameters are updated using the back-propagation algorithm. The input of this network is 229x229x3 in size. All convolution layers use 3x3 small convolution kernels for parameter reduction. After feature calibration and feature extraction, the Mahalanobis distance is set to 2.0. In addition, the training round is set to 10,000 times, and the size of each block (batch size) includes 100 pieces of data, of which 80% of the dataset is used as a training sample set, 10% is used as a cross-validation set in training, and the remaining 10% is used as a test data set for predicting the performance of the classifier in the real world, with predictions made every 10 epochs , the Adam algorithm is used to train the model during the training process, and the learning rate is set to 0.01. At the end of each training round, the validation error on the validation set is calculated.

Step 4. Predict the trained model on the test set and compare the results

The content described in the embodiments of the present specification is only an enumeration of the realization forms of the inventive concept, and the protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments, and the protection scope of the present invention also extends to those skilled in the art. Equivalent technical means that can be conceived by a person based on the inventive concept.

Claims (2)

1. A method for building calibration based on transfer learning, comprising the following steps: Step 1. Self-made data set. The building data set comes from Google pictures and Baidu pictures. Based on the existing data sets, it is divided into seven types of common buildings in life, namely churches, residential buildings, hospitals, hotels, and libraries. , villa housing and shopping malls, and make uniform treatment; Step 2. Build a multi-feature calibration neural network model based on migration learning, and the entire network architecture is divided into image feature calibration, image feature extraction, database matching, and graphic authentication; Image feature calibration: Single feature calibration is performed for buildings with obvious architectural styles, and multi-feature calibration is performed for various styles, and then the feature matrix of the image is formed; Image feature extraction: The image after feature calibration is used as the network input. The multi-layer neural network learns the features of the original image by combining a series of convolutional layers and downsampling layers, and combines the classic BP algorithm to adjust the parameters and complete the weights. Update, the BP network update weight formula is: ω(t+1)=ω(t)+ηδ(t)x(t) (1) where ω(t) is the connection weight, x(t) is the output of the neuron, δ(t) is the error term of the neuron, η is the learning rate, and the network structure of the convolutional layer in the network adopts the discrete convolutional type, expressed as: where M _β represents a choice of input features, k represents the convolution kernel, γ represents the number of layers of the network, and b represents the bias added by each input feature map. For a specific output map, the input map features can be applied with different volumes The product kernel convolution is obtained; f represents the activation function used by the convolutional neuron, and each block uses the ReLu activation function here; Database matching: The database here is the feature vector of each image obtained from the training data set. After network training, the feature information of each image will be saved in the label file in the form of 1024-dimensional data points. When the newly trained image feature information will be matched to the database by calculating its Mahalanobis distance; Image authentication: After the image feature extraction module completes the feature extraction, two output results are obtained, one is the feature vector of the image, and the other is the label after image classification; the image authentication module first classifies the image to be authenticated into the image library according to the existing label of the image In the corresponding category, the Mahalanobis distance calculation is performed with other images of this category, and the result is counted as D _i (i=1, 2, 3..., m), and D _min is taken out; according to the calculation result, it is judged whether the image to be authenticated comes from the database or from the same source image as the image in the database; the judgment method uses the threshold method in this paper, and the threshold Th _m is set according to a large number of experimental results; if D _min >Th _m , that is, the distance between this image and all images in this class is large, and it does not belong to this category. database; otherwise, the image belongs to the database, and the image corresponding to D _min is the same image or comes from the same image; The definition of Mahalanobis distance: It is known that there are M sample vectors X ₁ ～X _m , the covariance matrix is denoted as S, and the mean value is denoted as vector μ, then the Mahalanobis distance D(X) from the sample vector X _i to μ is expressed as : The Mahalanobis distance between the vectors X _i and X _j is defined as: If the covariance matrix is a unit matrix, that is, independent and identical distribution among the sample vectors, the formula becomes: That is, the Euclidean distance; if the covariance matrix is a diagonal matrix, the formula becomes the standardized Euclidean distance; therefore, compared with the Euclidean distance, the Mahalanobis distance is an effective method for calculating the similarity of two unknown sample sets. Method; Unlike Euclidean distance, it takes into account the relationship between various characteristics and is not affected by dimensions, and tests the accuracy of the image authentication module. The image authentication accuracy mainly depends on the results of the previous image classification and the selection of the threshold, so the purpose of the experiment is to find a suitable threshold and the initial value of the threshold By calculating the distance between the transformed image and the original image, it is set after statistical analysis; Step 3. Network training, the process is as follows: The number of training rounds of the network is set to 10,000 times, the training batch-size is 100, and a round of prediction is performed every 10 times. At the same time, 80% of the total data is used as the main training sample T, and the other 10% is used as the cross in the training process. The validation sample V, the remaining 10% is used as the test dataset K for predicting the performance of the classifier in the real world, the algorithm used for training is the stochastic gradient descent algorithm SGD, the input of the network is 229x229x3, the convolution in the block All layers use 3x3 small convolution kernels for parameter reduction. By migrating the GoogleNet weights from imageNet training and fine-tuning, the network starts training from the optimal point, retrains the classifier, and calculates the log loss function value of the training set T. , update the weights and biases of the multi-layer neural network according to the back-propagation algorithm; Calculate the log loss loss function value of the verification set V to determine whether the model has converged. If the convergent model training ends, otherwise enter the next batch of data on the training set T for training until the log loss loss function value on the verification set V tends to converge; finally use Test set K to verify the accuracy of the model; Step 4. Calibrate the trained model to the actual building image. 2. A method for calibrating buildings based on transfer learning, characterized in that: the initial value of the threshold described in step 2 is 2.0.