CN110059558A - A kind of orchard barrier real-time detection method based on improvement SSD network - Google Patents

Abstract

The invention discloses a real-time detection method for orchard obstacles based on an improved SSD. The improved SSD deep learning target detection method is used to identify obstacles in an orchard environment, and a lightweight network MobileNetV2 is used as the basic network in the SSD model to reduce extraction. The time and computational complexity of the image feature process, the auxiliary layer uses the reverse residual structure combined with the hole convolution as the basic structure for position prediction, so that the multi-scale features can be integrated and the information loss caused by the downsampling operation can be avoided. The corresponding image The improved SSD target detection model is trained on the data set, and the images collected by the camera are input into the trained model to detect the target position, which solves the traditional obstacle detection algorithm that is prone to background interference, inaccurate positioning of obstacles and difficult to locate. Realize problems such as simultaneous detection of multiple obstacle categories.

Description

A Real-time Detection Method of Orchard Obstacles Based on Improved SSD Network

technical field

The invention belongs to the fields of computer vision and deep learning, and specifically relates to an obstacle detection method for intelligent operations of mobile robots in an outdoor orchard environment.

Background technique

Accurate identification of obstacles in farmland is one of the essential key technologies for unmanned agricultural vehicles. With the proposal of precision agriculture theory and the development of intelligent robots, the automatic navigation of intelligent agricultural vehicles has attracted more and more attention at home and abroad. Autonomous navigation agricultural vehicles have the characteristics of replacing labor, improving operational efficiency, and reducing agricultural production costs. In order to ensure the safety of intelligent vehicles operating in the field without human intervention, real-time obstacle detection is necessary. Obstacle detection in the field environment is challenging to implement due to its complex natural environment, variability of obstacle shapes, and large-scale changes in external conditions such as illumination. In the field environment, ultrasonic sensors have shortcomings such as poor accuracy in detecting the spatial position of obstacles and being susceptible to interference. Although lidar sensors can detect obstacles more intuitively, the cost of radar systems is expensive. Compared with other obstacle detection methods, computer vision detection has the advantages of low cost and effective use of color and texture information in the environment. In this paper, the computer vision method combined with deep learning is used to detect pedestrian obstacles during the automatic operation of unmanned agricultural machinery.

In the field of target detection, the accuracy of deep learning-based methods greatly exceeds that of traditional detection methods based on artificially designed features such as HOG and SIFT. Target detection based on deep learning mainly includes two categories, one is the convolutional network structure based on region generation, and the representative network is the R-CNN series (R-CNN, fast R-CNN, faster R-CNN); The class is to regard the detection of the target position as a regression problem, directly use the CNN network structure to process the entire image, and predict the category and position of the target at the same time. Representative networks include YOLO, SSD (Single Shot MultiBox Detector), etc. Its speed Generally faster than the former method.

Since the SSD target detection model does not require time-consuming region generation and feature resampling steps, it directly convolves the entire image and predicts the category and corresponding coordinates of the objects contained in the image, thereby greatly improving the detection speed. By using small-sized convolution kernels, multi-scale prediction, etc., the accuracy of target detection is greatly improved. The SSD network structure is divided into two parts: the base network and the auxiliary network: the base network is some typical networks with high classification accuracy in the field of image classification and its classification layer is removed; the auxiliary network is the base network. Based on the added convolutional network structure for object detection, the size of these layers is gradually reduced to enable multi-scale prediction. The SSD network has excellent performance in the comprehensive performance of detection speed and accuracy, and its detection speed and accuracy need to be further improved, and its computational load needs to be reduced to meet the requirements of its deployment and operation on mobile devices.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention uses the lightweight network MobileNetV2 as the basic network in the SSD model to reduce the time and computational complexity of the process of extracting image features, and the auxiliary layer uses the reverse residual structure combined with the hole convolution as the basic structure for position prediction. It can integrate multi-scale features while avoiding information loss caused by down-sampling operations, so as to perform real-time obstacle detection and ensure the safety of intelligent vehicles operating in the field without manual intervention, reducing the amount of parameters and calculations of deep learning models. Therefore, the hardware requirements of the deep learning model can be reduced and the real-time performance can be achieved to meet its application on outdoor mobile devices.

The technical scheme of the present invention is: a real-time detection method for orchard obstacles based on an improved SSD network, comprising the following steps:

Step 1, construct a data set about the orchard environment and divide the data set into a training set and a test set;

Step 2: On the basis of the TensorFlow deep learning framework, build an SSD network target detection model, use MobileNetV2 as the feature extraction network, use the reverse residual structure for the auxiliary layer of the SSD and combine the hole convolution as the basic convolution structure;

Step 3: Initialize the parameters in the network model to obtain a pre-training model;

Step 4: Using the training set and test set in Step 1, use the batch gradient descent algorithm to train the pre-trained model, and use the difficult sample mining strategy during the training process to enhance the model's ability to discriminate false positives;

Step 5: Deploy the SSD network target detection model, collect images through the camera and send them to the SSD network target detection model, and use the non-maximum suppression algorithm to remove redundant bounding boxes to obtain the detection results.

Further, the specific process of step 1 is:

1.1) Obtain a large number of video images in the orchard environment with different scenes on the camera installed on the corresponding orchard agricultural machinery. Obtain a large number of videos in the orchard environment, and extract pictures according to 7.5 frames/second, and take all pictures according to the ratio of 2:1:1 Divided into training set, validation set and test set;

1.2) Manually label all the above images, the labelled object is the obstacle target to be detected, and the specific labeling information is the category of the target in the image and the upper left and lower right coordinate values of the bounding box of the target;

1.3) Preprocess the images of the training set, including horizontal flipping and translation to increase the number of samples and corresponding processing of the annotation information, and increase the quality of the image through adaptive histogram equalization and reduce the impact of illumination changes on the image. .

Further, in step 2, the MobileNetV2 is used as the feature extraction network, the reverse residual structure is used for the auxiliary layer of the SSD, and the hole convolution is used as the basic convolution structure. The specific method is:

2.1) After removing the convolutional layer used for classification of MobileNetV2, leave the feature extraction layer as the basic network of SSD;

2.2) Use the reverse residual structure combined with the hole convolution and apply the hierarchical feature fusion strategy to solve the computational discontinuity problem caused by the hole convolution, so as to serve as the basic structure of the auxiliary layer for the location and the feature extracted by the basic network. category detection.

Further, the specific method of step 3 is:

3.1) Train MobileNetV2 on ImageNet large-scale classification dataset to achieve high classification accuracy;

3.2) Remove the classification convolution layer of MobileNetV2, and assign the convolution layer parameters used for feature extraction to the feature extraction layer corresponding to SSD;

3.3) The parameters of each layer of the SSD auxiliary layer are randomly initialized using a Gaussian distribution with 0 as the mean and 0.01 as the standard deviation.

Further, the specific method of step 4 is:

4.1) The objective function used in the training process of the batch gradient descent algorithm is:

where N is the number of matching default bounding boxes, when N is 0, directly set L to 0, c is the label category, l is the predicted bounding box, g is the labeled bounding box, and L _loc is the corresponding The smooth _L1 error of position prediction, L _conf is the corresponding softmax multi-class error function:

Among them: Pos is the positive example in the sample, cx, cy are the coordinates of the center point of the prediction frame, w is the width of the prediction frame, h is the height of the prediction frame, Whether the predicted box matches the jth real box with respect to category K, is the prediction box, for the real box

in: Whether the predicted box i and the real box j match the category p, Neg is the negative example in the sample, To predict that there is no object in the box, The calculation formula is:

in: The probability that the target is the pth class in the ith prediction box for the target.

4.2) In the training process, first use the initial positive and negative samples to train the detection model, and then use the trained model to detect and classify the samples, and continue to put those samples that are wrongly detected into the negative sample set for training, so as to strengthen the model to discriminate false positives. positive ability.

Further, the specific method of step 5 is:

5.1) Remove the operation used in the training process to prevent overfitting and fix the network parameters to obtain the SSD network target detection model for deployment;

5.2) The image is collected by the camera and used as the input of the model, so as to obtain the category confidence and bounding box coordinates of several targets;

5.3) Use the non-maximum suppression algorithm to remove redundant detection frames to obtain more accurate detection results.

Further, the non-maximum value suppression algorithm is specifically: sorting the detection results according to the confidence levels corresponding to the detection results from high to low, and calculating the corresponding overlap rate, setting the threshold value of the overlap rate to 0.5, in This detection result is adopted when the detection result has a high confidence and a high overlap rate threshold.

The advantages of this scheme are:

1) Through the transfer learning technology, the parameters that MobileNetV2 performs better in Imagenet classification are transplanted into the feature extraction network model of SSD, thereby simplifying the training process of the target detection model and shortening the training time.

2) By improving the feature extraction network of the original SSD, the more lightweight MobileNetV2 network model is used for feature extraction, and the auxiliary layer uses the improved reverse residual structure for convolution operation, so that multi-feature information can be used and the amount of computation can be reduced, Thereby, the accuracy and detection speed of model detection are improved.

3) The improved SSD target detection model is used to detect pedestrian obstacles in the field environment. The model occupies a small space and is lightweight, and is suitable for deployment on mobile devices. The model has good robustness and can better Realize the detection of obstacles in the orchard environment, and provide a basis for obstacle avoidance decisions.

Description of drawings

Fig. 1 is a step diagram of the present invention.

Figure 2 Improved reverse residual structure diagram

Figure 3 Hierarchical feature fusion structure diagram

The hole convolution layer structure is expressed as (#input channel, receptive field, #output channel), where the effective receptive field of the hole convolution kernel is nk*nk,nk=(n-1)*2k-1+1,k= 1,...,K.

Figure 4 The improved SSD object detection model.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

The present invention provides a real-time detection method for orchard obstacles based on improved SSD, and the method mainly comprises the following steps:

Step 1: Construct the dataset and divide the dataset into training set and test set. This step includes the following sub-steps:

1.1) Obtain a large number of video images in the orchard environment with different scenes on the camera installed on the corresponding orchard agricultural machinery. Obtain a large number of videos in the orchard environment, and extract pictures according to 7.5 frames/second, and take all pictures according to the ratio of 2:1:1 Divided into training set, validation set and test set;

1.2) Manually annotate all the above images, the annotated object is the obstacle target to be detected, and the specific annotation information is the category of the target in the image and the coordinate values of the upper left and lower right of the bounding box of the target.

1.3) Preprocess the images of the training set, including horizontal flipping and translation to increase the number of samples and corresponding processing of the annotation information, and increase the quality of the image through adaptive histogram equalization and reduce the impact of illumination changes on the image. .

Step 2: On the basis of the TensorFlow deep learning framework, MobileNetV2 is used as the feature extraction network, and the reverse residual structure is used for the auxiliary layer of SSD and combined with hole convolution as the basic convolution structure.

2.1) After removing the convolutional layer used for classification of MobileNetV2, leave the feature extraction layer as the basic network of SSD;

2.2) Use the reverse residual structure combined with the hole convolution and apply the hierarchical feature fusion strategy to solve the computational discontinuity problem caused by the hole convolution, so as to serve as the basic structure of the auxiliary layer for the location and the feature extracted by the basic network. category detection.

It mainly includes the following steps:

1. Build an improved SSD target detection algorithm in the TensorFlow deep learning framework, and remove the convolutional layers conv2d 1x1, avgpool 7x7, and conv2d 1x1 of the lightweight network model MobileNetV2 for classification as the base layer of SSD for feature extraction.

2. Improve the convolution structure of the auxiliary layer of the SSD target detection model, and use the reverse residual structure set hole convolution structure to improve the convolution structure, and use it as the basic convolution structure unit of the auxiliary layer, as shown in the attached figure. 2, the application of atrous convolution can increase the receptive field of the convolution kernel without downsampling operation, so as to reduce the information loss caused by nonlinear transformation in the learning process, and the convolution kernel has a multi-scale receptive field.

3. Due to the introduction of hole convolution, the operation of the convolution kernel will be discontinuous. The hierarchical feature fusion algorithm is further used to eliminate the negative impact of hole convolution. The specific implementation method is for each convolution of the hole convolution layer. The outputs of the units are summed in turn, and each summed result is concatenated to obtain the final output result. See Figure 3.

Among them, the activation function used by the improved reverse residual structure is ReLU6. Compared with ReLU, ReLU6 has better robustness in low-precision operation scenarios. In addition, the size of the convolution kernel still uses a typical 3x3 volume. Product kernel, ReLU6 function is shown in formula (1).

ReLU6=min(max(features,0),6) (1)

The final improved SSD target detection model is shown in Figure 4.

Step 3: Initialize the parameters in the network model to obtain a pre-trained model. It mainly includes the following steps:

3.1) Train MobileNetV2 on ImageNet large-scale classification dataset to achieve high classification accuracy;

3.2) Remove the classification convolution layer of MobileNetV2, and assign the convolution layer parameters used for feature extraction to the feature extraction layer corresponding to the SSD; for the MobilenetV2 part of the basic network, use the MobilenetV2 network that has been trained on the ImageNet classification task dataset. And extract the parameters corresponding to the network structure as the initialization value of the basic network.

3.3) The parameters of each layer of the SSD auxiliary layer are randomly initialized using a Gaussian distribution with 0 as the mean and 0.01 as the standard deviation.

Step 4: Use the batch gradient descent algorithm to train the pre-trained model, and use the difficult sample mining strategy during the training process to enhance the model's ability to discriminate false positives. The specific training process is:

4.1) The objective function used in the training process of the batch gradient descent algorithm is:

where N is the number of matching default bounding boxes, when N is 0, directly set L to 0, c is the label category, l is the predicted bounding box, g is the labeled bounding box, and L _loc is the corresponding The smooth _L1 error of position prediction, L _conf is the corresponding softmax multi-class error function:

Among them: Pos is the positive example in the sample, cx, cy are the coordinates of the center point of the prediction frame, w is the width of the prediction frame, h is the height of the prediction frame, Whether the predicted box matches the jth real box with respect to category K, is the prediction box, for the real box

in: Whether the predicted box i and the real box j match the category p, Neg is the negative example in the sample, To predict that there is no object in the box, The calculation formula is:

in: The probability that the target is the pth class in the ith prediction box for the target.

4.2) In the training process, first use the initial positive and negative samples to train the detection model, and then use the trained model to detect and classify the samples, and continue to put those samples that are wrongly detected into the negative sample set for training, so as to strengthen the model to discriminate false positives. positive ability.

The above batch gradient descent algorithm sets the sample batch size to 128, the impulse to 0.9, the weight decay coefficient to 2×10 ^-3 , the maximum number of iterations to 100k, the initial learning rate to 0.004, and the decay rate to 0.95. After every 10,000 iterations Attenuate once, and save the model every 10,000 iterations, and finally select the model with the highest accuracy.

The hard negative mining strategy is used in the training process, that is, the initial positive and negative samples are used to train the detection model during the training process, and then the trained model is used to detect and classify the samples, and those samples that are detected incorrectly are placed in the The negative sample set is used for training, which can strengthen the ability of the model to discriminate false positives.

Step 5: Deploy the SSD model, collect images through the camera and send them to the SSD target detection model, and use the non-maximum suppression algorithm to remove redundant bounding boxes to obtain the detection results. The specific implementation is as follows:

5.1) Remove the operations used in the training process to prevent over-fitting and fix the network parameters to obtain the network model for deployment; 5.2) Collect images through the camera and use them as the input of the model, so as to obtain the category confidence of several targets and bounding box coordinates; 5.3) Use non-maximum suppression algorithm to remove redundant detection boxes to get more accurate detection results.

1. Fix the model parameters trained in step 4 and remove operations such as dropout to prevent overfitting to obtain the final network model.

2. Test and evaluate the network model. The evaluation indicators are precision (P) and recall (R) and their harmonic mean F ₁ , as shown in equations (2), (3), and (4) respectively.

In equations (2) and (3), TP is the number of correctly detected pedestrians, FP is the number of non-pedestrian targets mistakenly detected as pedestrian targets, FN is the number of pedestrians mistakenly detected as the background, and the value of F1 is the number _of The harmonic mean of rate and recall, the closer it is to 1, the better the performance of the model.

3. Fix the expected network model parameters and deploy them in the corresponding mobile devices, obtain real-time pictures in the orchard environment through the camera and input them into the model, and use non-maximum suppression to remove redundant detection frames, where the IOU threshold is selected. is 0.4, and the confidence threshold is selected as 0.5.

In summary, the real-time detection method of orchard obstacles based on the improved SSD of the present invention is mainly suitable for the automatic navigation scene of the unmanned agricultural machine in the orchard environment. The obstacle detection method is based on the SSD detection network, and the network structure and training process are improved to reduce the calculation amount to speed up the detection speed, improve the detection accuracy, meet the real-time requirements, and reduce the hardware requirements of the deep learning model. Thereby, it can meet the deployment application on the mobile. The method first collects the corresponding video data, and extracts pictures at a certain frame rate for labeling, and produces a data set for training. And initialize the deep learning model through transfer learning, then use the batch gradient descent algorithm to train the model, and finally use the trained model for the actual obstacle detection task. The invention can quickly and accurately detect the obstacles ahead in the orchard environment, and is a powerful measure for realizing intelligent agriculture and improving its reliability.

Claims (7)

1. a kind of based on the orchard barrier real-time detection method for improving SSD network, characterized in that the following steps are included:

Step 1, it constructs the data set about orchard environment and data set is divided into training set and test set；

Step 2: on the basis of TensorFlow deep learning frame, SSD network objectives detection model is built, it will MobileNetV2 uses reversed residual error structure to the auxiliary layer of SSD and combines empty convolution as base as feature extraction network Plinth convolutional coding structure；

Step 3: the parameter in initialization network model obtains pre-training model；

Step 4: using the training set and test set in step 1, pre-training model being instructed using batch gradient descent algorithm Practice, enhances the ability of Model checking false positive using difficult sample Mining Strategy in the training process；

Step 5: deployment SSD network objectives detection model passes through camera collection image and is sent into SSD network objectives detection mould Type, and remove excess edge frame using non-maxima suppression algorithm, obtain testing result.

2. according to claim 1 a kind of based on the orchard barrier real-time detection method for improving SSD network, feature exists In the detailed process of step 1 are as follows:

1.1) video under the orchard environment by obtaining a large amount of different scenes on the camera that is mounted on corresponding orchard agricultural machinery Image obtains the video under a large amount of orchard environment, and extracts picture according to 7.5 frames/second, by all pictures according to 2:1:1 ratio point For training set, verifying collection and test set；

1.2) above-mentioned all images are manually marked, the object of mark is obstacle target to be detected, specific to mark Infuse the coordinate value that information is the upper left and bottom right of the bounding box of the classification and target of target in image；

1.3) image of training set is pre-processed, including flip horizontal and translation are to increase sample size while also to mark Information carries out corresponding processing, and increases the quality of image by adaptive histogram equalization, reduces illumination variation to image Influence.

3. according to claim 1 a kind of based on the orchard barrier real-time detection method for improving SSD network, feature exists In, it is described using MobileNetV2 as feature extraction network in step 2, reversed residual error structure is used to the auxiliary layer of SSD And combine empty convolution as basic convolutional coding structure method particularly includes:

2.1) basic network for leaving feature extraction layer as SSD after removing the convolutional layer for being used to classify of MobileNetV2；

2.2) empty convolution is combined with reversed residual error structure and application level Fusion Features strategy solves brought by empty convolution Discontinuous problem is calculated, so that the feature that the basic structure as auxiliary layer is used to extract basic network carries out position and class Other detection.

4. according to claim 1 a kind of based on the orchard barrier real-time detection method for improving SSD network, feature exists In step 3 method particularly includes:

3.1) being trained on the extensive categorized data set of ImageNet to MobileNetV2 makes it to reaching higher classification Accuracy；

3.2) the classification convolutional layer for removing MobileNetV2 takes it to be used for the convolutional layer parameter assignment of feature extraction corresponding to SSD Feature extraction layer；

It 3.3) is mean value with 0 to each layer parameter use of SSD auxiliary layer, 0.01 is random initial for the Gaussian Profile progress of standard deviation Change.

5. according to claim 1 a kind of based on the orchard barrier real-time detection method for improving SSD network, feature exists In step 4 method particularly includes:

4.1) batch gradient descent algorithm is trained objective function used in process are as follows:

Wherein N is the number of matched default boundary frame, when wherein when N is 0, it is mark classification that directly setting L, which is 0, c, and l is The bounding box of prediction, g are the bounding box of mark, L_locFor the smooth of corresponding position prediction_L1Error, L_confIt is corresponding The more error in classification functions of softmax:

Wherein: Pos is the positive example in sample, and cx, cy are the center point coordinate of prediction block, and w is the width of prediction block, and h is prediction block Height,Whether matched with j-th of true frame about classification K for which prediction block,For prediction block,For true frame

Wherein:Whether being matched with true frame j about classification p for prediction block i, Neg is the negative example in sample,For in prediction block There is no object,Calculating formula are as follows:

Wherein:It is the probability of p-th of classification for target in i-th of prediction block of target.

4.2) first with initial positive and negative sample training detection model in training process, then using the model trained to sample into Row detection classification continues wherein detection those of mistake sample to be put into negative sample set being trained, so as to reinforce mould The ability of type differentiation false positive.

6. according to claim 1 a kind of based on the orchard barrier real-time detection method for improving SSD network, feature exists In step 5 method particularly includes:

5.1) it removes used for preventing the operation of over-fitting and fixed network parameter has been obtained for portion in training process The SSD network objectives detection model of administration；

5.2) by camera collection image and the input as model, to obtain classification confidence level and the boundary of several targets Frame coordinate；

5.3) extra detection block is removed using non-maxima suppression algorithm, obtains more accurate testing result.

7. according to claim 6 a kind of based on the orchard barrier real-time detection method for improving SSD network, feature exists In non-maxima suppression algorithm specifically: confidence level corresponding in testing result carries out according to confidence testing result Degree is ranked up from high to low, and calculates corresponding Duplication, and setting Duplication threshold value is 0.5, is had in testing result This testing result is adopted when high confidence level and high Duplication threshold value.