CN118038379A - Vehicle small target detection method and device based on lightweight network design - Google Patents

Abstract

The present invention discloses a vehicle small target detection method and device with a lightweight network design, which relates to the field of target detection technology, including: obtaining a vehicle image to be tested; using a trained network model to classify the vehicle image to be tested, and outputting a classification result; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain a classification result. The present invention can meet the requirements of real-time and accuracy for vehicle target detection at the edge end where storage and calculation are limited.

Description

A vehicle small target detection method and device with lightweight network design

Technical Field

The present invention belongs to the technical field of target detection, and in particular relates to a vehicle small target detection method and a device thereof with a lightweight network design.

Background Art

Vehicle retrieval technology is a technology that uses computer vision algorithms to determine whether there is a specific vehicle target in the image to be tested. This technology has important application value in the fields of traffic planning and environmental testing. In addition, drones are small in size and highly maneuverable, so they are suitable for collecting information in complex areas and are applicable to various civil and military fields. It is of great research significance to combine drones with vehicle retrieval technology to study the problem of drone vehicle retrieval under drone-mounted cameras in different geographical environments and weather conditions. The research content of aerial vehicle small target retrieval in complex scenes includes three tasks: accurate and fast retrieval in complex scenes, building an efficient target retrieval database, and determining the target candidate frame. In actual application environments, due to the influence of drone flight altitude, shooting angle and background noise, the drone vehicle target retrieval effect in complex scenes is poor, the missed detection rate is high, and the positioning is not accurate enough, which cannot meet actual needs. In addition, the existing detection task network computing cost is high, which is not friendly to the edge end where storage and computing are limited.

At present, the research on small target detection of unmanned aerial vehicles has been improved from the following perspectives. First, in terms of data sets, since target detection data sets are not abundant, most research work will first expand the data sets in order to better train the accuracy of model recognition, such as using the Poisson fusion method based on the self-made data set for expansion; secondly, the model structure is improved, such as by adding a feature map attention layer (featuremap attention, FMA) at the end of the backbone to improve the network feature extraction capability to modify the YOLOV5 algorithm, or from the perspective of network versatility and scene complexity, using a simpler and more practical ResNet-50 network structure to replace the original backbone network DarkNet-53, which is convenient for the implementation of subsequent joint algorithms. In order to better deploy the neural network model on the edge, it can also be improved from a lightweight perspective, such as improving the backbone network in YOLOv3, replacing some convolution operations with linear operations with less computation without reducing the accuracy of the original model, so that the model has fewer parameters and consumes less computing power; or using a surface target detection model compression method based on deep learning, using an improved network with deep separable convolution and lightweight attention model to replace the feature extraction network DarkNet, and compressing the model through multi-scale feature fusion. Generally speaking, in order to speed up the calculation speed of the final prediction, the images in the Convolutional Neural Networks (CNN) almost always undergo a similar conversion process in the backbone network, and the information of the spatial dimension is gradually transmitted to the channel; and each time the spatial (width and height) compression of the feature map and the expansion of the channel dimension will cause the loss of some semantics, the simple use of DSC operation can increase the speed of the detector, but these models expose the real problem of low accuracy in specific applications. MobileNets uses a large number of 1*1 dense convolutions to fuse independently calculated channel information; ShuffleNets uses "channel shuffle" to achieve channel information interaction; GhostNet uses the "halved" standard convolution (SC) operation to retain the interaction information between channels; however, 1*1 dense convolution takes up more computing resources, and the effect of using "channel shuffle" still does not touch the result of standard convolution; many lightweight models only use depth-separable convolution from beginning to end, but whether it is used for image classification or target detection, the defects of depth-separable convolution are directly amplified in the backbone.

Therefore, there is an urgent need to provide a vehicle small target detection method with a lightweight network design to reduce costs.

Summary of the invention

In order to solve the above problems existing in the prior art, the present invention provides a vehicle small target detection method and device with a lightweight network design. The technical problem to be solved by the present invention is achieved by the following technical solutions:

In a first aspect, the present invention provides a vehicle small target detection method with a lightweight network design, comprising:

Acquire an image of a vehicle to be tested;

The trained network model is used to classify the vehicle image to be tested and the classification result is output; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain the classification result.

In a second aspect, the present invention further provides a vehicle small target detection device with a lightweight network design, comprising:

An image acquisition module, used to acquire an image of a vehicle to be tested;

The image processing module is used to use the trained network model to classify the vehicle image to be tested and output the classification result; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain the classification result.

Beneficial effects of the present invention:

The present invention provides a method and device for detecting small vehicle targets with a lightweight network design. Aiming at the problems that the existing deep convolutional neural network has a large number of parameters and high computational cost, and the shallow lightweight network has a weak feature extraction capability for the target and poor detection performance, a method for detecting small vehicle targets with a lightweight network design is proposed. While reducing the computational cost, the feature extraction of the target has the advantages of high computational efficiency, semantic adaptation and global effectiveness. The output end constructs an attention-based bounding box loss function to make positioning more accurate and increase the generalization ability of the model. While reducing the complexity of the model, the entire network can meet the requirements of real-time and accuracy of vehicle target detection at the edge where storage and computing are limited.

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a vehicle small target detection method with a lightweight network design provided by an embodiment of the present invention;

FIG2 is a schematic diagram of a backbone network module provided by an embodiment of the present invention;

FIG3 is a schematic diagram of an adaptive frequency band filtering operator provided by an embodiment of the present invention;

FIG4 is a schematic diagram of a neck network module provided by an embodiment of the present invention;

FIG5 is a schematic diagram of a preset network model for training provided in an embodiment of the present invention;

FIG6 is a schematic diagram of an example image of a data set provided by an embodiment of the present invention;

FIG7 is a schematic diagram of a test result provided by an embodiment of the present invention;

FIG8 is a schematic diagram of a vehicle small target detection device with a lightweight network design provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

In the existing technology, due to the limited computer resources and the mismatch of software and hardware resources in the early days, target detection can only be achieved by designing artificial features and using various acceleration technologies. In 2012, with the introduction of convolutional networks, some famous convolutional neural networks such as AlexNet, VGGGoogLeNet, and ResNet gradually came into people's sight. Since convolutional neural networks can extract richer information and features, many researchers have turned their attention to using convolutional neural networks to extract the features of target objects. In 2014, RGirshick et al. proposed the R-CNN algorithm, which was the first to break the deadlock and use the area of convolutional neural network features for target detection. Since then, target detection algorithms have developed rapidly, and more and more algorithms have emerged, and target detection algorithms have been well developed.

Target detection algorithms can generally be divided into two categories: two-stage detection methods represented by R-CNN and one-stage detection methods represented by the YOLO series. These two types of algorithms have different detection ideas and therefore have different characteristics and advantages and disadvantages. The two-stage detection method generates candidate boxes based on a certain algorithm, extracts features from these candidate boxes, and finally generates position boxes and predicts the corresponding types and regresses position information; this type of algorithm has high accuracy but slow speed. In 2016, Tao Kong et al. proposed the HyperNet algorithm. The core idea of the HyperNet algorithm is to fuse information at different feature levels through a multi-branch network to improve the accuracy of small target detection. Specifically, the HyperNet algorithm first extracts shallow features and deep features through the basic network. Then, the feature information at different levels is fused to obtain a more accurate fusion network feature representation. The fusion network adopts a multi-branch design, and each branch fuses different feature levels. Finally, the target is detected and located through the classification network and regression network. In 2017, Lin et al. proposed a feature pyramid network structure. Through a top-down feature extraction network and a bottom-up feature fusion network, feature pyramids of different sizes can be obtained. In this process, the high-level feature map is upsampled to the same scale as the low-level feature map, and then fused with the low-level feature map to obtain a feature map with more semantic information; finally, the target is detected and located through the detection network. Law H et al. proposed the CornerNet algorithm in 2018. The CornerNet algorithm is based on the idea of Keypoint. It determines the position and size of the target by detecting the corner points of the target, thereby realizing the detection of small targets. The advantage of the CornerNet algorithm is that it can accurately represent the position and size of the target by means of corner points, thereby realizing the accurate detection of small targets. The Vision Transformer (ViT) model proposed by the Google team in 2020 is a classic pure transformer technology solution for visual tasks; however, ViT has high computational complexity and high computational overhead. EfficientNetV2 (TanM, LeQ. Efficientnetv2: Smallermodels and faster training), published in April 2021, is a lightweight network model that proposes an improved progressive learning method that dynamically adjusts the regularization method according to the size of the training image. In addition, it verifies that the use of deep separable convolution in shallow networks is very slow, so a new network module Fused-MBConv is proposed for shallow networks to increase the training speed of the network and improve the accuracy of the network. However, because its network architecture is obtained through neural structure search, it is not fully suitable for small target recognition tasks of ground vehicles.

In summary, the disadvantages of the prior art include:

1. On the one hand, although convolutional neural networks require fewer parameters to be trained, they have local spatial perception, and global spatial perception is crucial for tasks such as image classification and semantic segmentation. On the other hand, the Transformer structure has high computational complexity and high computational overhead, which is more costly.

2. The computational complexity brought by deep networks is high, but shallow networks will bring problems such as poor ability to extract target features, resulting in poor detection performance, and shallow networks are slow when using deep separable convolution.

3. The existing algorithms have poor results in retrieval of drone vehicle targets, high missed detection rates, and inaccurate positioning, which cannot meet actual needs.

4. Most neural network algorithms are difficult to deploy and implement, and the training data of anchor-based detection algorithms inevitably contains low-quality examples. Therefore, geometric metrics such as distance and aspect ratio will increase the penalty for low-quality examples, thereby reducing the generalization performance of the model.

In view of this, the present invention proposes a method for detecting small vehicle targets using a lightweight network design, which reduces the computational cost while having the advantages of high computational efficiency, semantic adaptation, and global effectiveness in extracting target features. The output end constructs an attention-based bounding box loss function to make positioning more accurate and increase the generalization ability of the model. While reducing the complexity of the model, it can meet the requirements of real-time and accuracy of vehicle target detection at the edge where storage and computing are limited.

Please refer to FIG. 1 , which is a flow chart of a vehicle small target detection method with a lightweight network design provided by an embodiment of the present invention. The present invention provides a vehicle small target detection method with a lightweight network design, comprising:

S101, obtaining an image of a vehicle to be tested.

Specifically, in this embodiment, the vehicle image to be tested is a small target image.

S102. Use the trained network model to classify the image of the vehicle to be tested and output the classification result; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain the classification result.

In an optional embodiment of the present invention, please refer to FIG. 2, which is a schematic diagram of a backbone network module provided in an embodiment of the present invention. The backbone network module includes a first module, a second module and a third module. The image of the vehicle to be tested input into the backbone network module is processed by the first module to obtain a first feature, the first feature is input into the second module for processing to obtain a second feature, and the second feature is input into the third module for processing to obtain a third feature; wherein,

The first module, the second module and the third module all include a downsampling module, an AFF module and a fusion module. The number of AFF modules set in the first module, the number of AFF modules set in the second module and the number of AFF modules set in the third module are all different. The downsampling module includes an MBConv layer.

Specifically, in this embodiment, the backbone network module is first tokenized using a convolutional system, i.e., Conv Stem in FIG2 , which consists of a 3×3 convolutional layer with a stride of 2, followed by four MBConv layers, a downsampling module; it can be understood that MBConv is the abbreviation of a mobile inverted bottleneck convolution (MBConv) module with a kernel size of 3. After the input features are tokenized, they are processed by the main body of a three-stage cascaded lightweight network, where each stage consists of an MBConv layer with a stride of 2 for spatial downsampling and N _i AFF modules, i.e., the first module, the second module, and the third module are all composed of MBConv layers with a stride of 2; wherein, the number of AFF modules of the first module N ₁ =2, the number of AFF modules of the second module N ₂ =4, and the number of AFF modules of the third module N ₃ =3. In this embodiment, two groups of linear layers (also called 1×1 convolutional layers) and ReLU functions are used to learn the proposed adaptive frequency filtering mask. It should be noted that the use of a linear layer group is crucial because it can prevent nonlinearity from destroying too much information; the linear layer group uses a 1x1 convolution kernel to further compress the number of channels, reduce the amount of calculation and the number of parameters; in this embodiment, layer normalization (LN) is used for channel mixing, which is then fed to the adaptive frequency band filter operator for global sequence mixing to obtain the output of the lth AFF module; in this way, in this embodiment, jump connections of channel mixing and sequence mixing are used to speed up the training convergence of the model.

In an optional embodiment of the present invention, the AFF module includes a first normalization layer, a second normalization layer and an adaptive frequency band filter operator; wherein the first normalization layer and the second normalization layer are used for channel mixing, the adaptive frequency band filter operator is used for global sequence mixing, and the characteristic expression output by the AFF module is:

in, represents the output features after the lth downsampling layer and the first normalization layer, MBConv ^l (·) represents the output features after processing by the lth MBConv layer, X ^ll represents the features input to the downsampling layer, AFF ^l (·) represents the output features after processing by the lth AFF module, LN (·) represents the output features after processing by the second normalization layer, and X ^l represents the final output features after processing by the lth AFF module.

Specifically, in this embodiment, the backbone network composed of MBConv and adaptive frequency band filtering operators can perform hierarchical extraction of global features and local features, which can effectively reduce computational complexity, overhead costs and computational costs; on the basis of achieving the same effect as the deep network, the model calculation cost is low and the number of model parameters is small.

In an optional embodiment of the present invention, please refer to FIG. 3, which is a schematic diagram of an adaptive frequency band filter operator provided by an embodiment of the present invention. The adaptive frequency band filter operator includes a Fourier fast transform module, a frequency domain filter and an inverse Fourier transform module; the processing process of the adaptive frequency band filter operator on the input feature includes:

Use the Fourier transform module to transform the input features from the spatial domain to the frequency domain to obtain the frequency domain features, which are expressed as:

Among them, X _F represents the frequency domain features, X represents the input features, represents Fourier transform;

Use the frequency domain filter to process the frequency domain features and obtain the mask tensor learned by the frequency domain feature X _F

The mask tensor Frequency domain characteristics Perform point multiplication to obtain the filtered features Its expression is:

Among them, ⊙ represents matrix product;

Use the inverse Fourier transform module to transform the filtered features from the frequency domain to the spatial domain to obtain the spatial domain features Its expression is:

in, Represents inverse Fourier transform, spatial domain features That is the feature output after being processed by the adaptive frequency band filter operator.

Specifically, please continue to refer to FIG. 3 , the process of processing the input self-features by the adaptive frequency band filter operator in this embodiment includes:

S111. Perform fast Fourier transform on the input features.

First, the input features are transformed into the frequency domain by Fast Fourier Transform (FFT). For a given feature X∈R ^H×W×C , where H and W represent the height and width of the feature, and C represents the number of channels of the feature, the expression of the frequency domain feature after Fast Fourier Transform is:

Among them, u and v represent the height and width of the feature respectively. The features at different spatial positions correspond to different frequency components of X. The complexity is The transformation of X integrates the global information of X. This method can reduce the complexity of sequence mixing from Reduce to To reduce the computational complexity.

S112, through the learnable frequency domain filter By dot-multiplying the frequency domain features of the input obtained by S111, we can get:

in, represents the mask tensor learned from X _F , has the same shape as X _F. As shown in Figure 3, in order to make the network as lightweight as possible, It is effectively implemented by a group of 1×1 convolutional (linear) layers, followed by a group of RELU functions and another group of linear layers. It is understandable that the use of a linear layer group is crucial because it prevents nonlinearity from destroying too much information. The linear layer group uses a 1x1 convolution kernel to further compress the number of channels, reducing the amount of calculation and the number of parameters.

S113, convert back to the spatial domain through Inverse Fast Fourier Transform (IFFT).

Applying the S112 convolution theorem, the frequency representation _XF of X is filtered using a learnable instance-adaptive mask, thereby achieving effective global sequence mixing of X; the filtered _XF is further inverse Fourier transformed to obtain the updated feature representation in the original latent space. The process can be described as:

where ⊙ represents matrix product, represents the inverse Fourier transform, It can be seen as the result of a global adaptive sequence mixture of X. The use of linear layer groups is crucial to prevent nonlinearity from destroying too much information.

S114, completing the construction of the adaptive frequency domain filtering operator structure.

At this time, the result obtained through S111, S112 and S113 Mathematically equivalent to the output result obtained by using large kernel dynamic convolution, since the multiplication of two signals in the Fourier domain is equal to the Fourier transform of the convolution of the two signals in their original domain. When it is applied to frequency domain multiplication, there exists:

According to the above formula, we can get:

in, It is a tensor of the same shape as X and can be regarded as a dynamic deep convolution kernel as large as X in space, which is adaptive to the information of X. Due to the properties of Fourier transform, circular padding is used for X here, and the operation of inverse Fourier transform enables the module to have a global range of sequence mixing operations with semantically adaptive weights.

In this embodiment, the adaptive frequency band filter operator is equivalent to the large spatial convolution kernel, which has higher computational complexity and greater computational overhead. The adaptive filter operator is used instead to reduce the computational cost of the network and the computational overhead, thereby achieving the same effect as the dynamic large spatial convolution kernel. While effectively reducing the computational cost and reducing resource waste, it achieves the characteristics of efficient, global and adaptive target feature extraction.

In an optional embodiment of the present invention, please refer to FIG. 4, which is a schematic diagram of a neck network module provided in an embodiment of the present invention. The neck network module includes three ASFF modules, namely a first ASFF module, a second ASFF module and a third ASFF module; wherein the neck network module processes the input features including:

The first ASFF module fuses the first feature, the second feature and the third feature to obtain a first fused feature, the second ASFF module fuses the first feature, the second feature and the third feature to obtain a second fused feature, and the third ASFF module fuses the first feature, the second feature and the third feature to obtain a third fused feature.

Specifically, please continue to refer to Figure 4. In this embodiment, the semantic information of the features changes from low-dimensional to high-dimensional as the network level deepens. High-level features provide rich semantic information for object classification, while low-level features provide rich fine-grained information for object positioning. Feature fusion is also particularly important, so an efficient feature fusion structure is constructed, and an adaptive spatial feature fusion (ASFF) module is introduced. After the fusion modules of each stage (Stage) of the backbone network, the features are imported into the ASFF module, and self-learning weights are set for each fused feature map to perform weighted fusion, so that the semantic information is more deeply integrated to achieve the purpose of improving detection performance.

Please continue to refer to Figure 4. ASFF1, ASFF2, and ASFF3 correspond to the outputs X1, X2, and X3 of stage 1, stage 2, and stage 3, respectively. Taking ASFF3 as an example, to obtain the output of ASFF3, first adjust the number of channels of X1 and X2 through 1×1 convolution, then adjust the width and height of the feature map through upsampling to make it consistent with X3, then multiply the adjusted feature map by the corresponding weight coefficient, and finally add the low-resolution feature map with position information and the high-resolution feature map with semantic information to obtain the fused feature map; the calculation formula of the ASFF output layer is as follows:

in, Represents the new l-th layer feature map obtained after fusion, It represents the feature vector with the same size and dimension as the feature map of the lth layer after the size and dimension of the feature map of the nth layer are adjusted. and They represent the weight coefficients of the lth layer obtained by adaptive learning, The expression of weight coefficient is:

in, represents the self-learning weight control parameter of the lth layer, which is obtained by 1×1 convolution of the adjusted feature vector. Indicates the definition of level l The self-learning weight control parameters, Indicates the definition of level l The self-learning weight control parameters.

In this embodiment, the above-mentioned fusion method not only prevents different network layers from learning repeated gradient information, but also eliminates the computational bottleneck, reduces memory consumption, promotes the fusion of low-level information, and obtains stronger feature extraction capabilities; it also alleviates the problem of feature loss of target position information and contour information to a certain extent.

In an optional embodiment of the present invention, please refer to FIG. 5, which is a schematic diagram of training a preset network model provided in an embodiment of the present invention. The trained network model is obtained by training the preset network model. The training of the preset network model includes:

Obtain a data set, which includes a training sample set and a test sample set;

Use the training sample set to train the preset network model;

During the training process, the loss function is used for iterative optimization to obtain the optimal weights of the preset network model to build a trained network model;

Use the test sample set to test the performance of the trained network model.

In an optional embodiment of the present invention, the loss function includes a boundary loss function, a confidence loss function and a classification loss function, and the confidence loss and the classification loss are both calculated by a cross entropy loss function. The expression of the loss function is:

Loss＝ _LWIoUv +Loss _cls +Loss _obj ;

The data inevitably contains low-quality samples, and geometric factors such as distance and aspect ratio will aggravate the penalty for low-quality samples, thereby reducing the generalization performance of the model. When the anchor box and the target box overlap well, a good loss function should weaken the penalty of geometric factors, and less intervention during training will enable the model to obtain better generalization ability. Therefore, distance attention R _WIoU is introduced, and WIoUv with a two-layer attention mechanism is obtained, which will significantly amplify the L _IoU of the normal quality anchor box. The expression of the boundary loss function L _WIoUv is:

L _WIoUv ＝R _WIoU L _IoU

Among them, R _WIoU represents the coefficient based on distance setting, L _IoU represents the initial intersection-over-union loss function, x represents the horizontal coordinate of the center of the true box, x _gt represents the horizontal _coordinate of the predicted center point area, y represents the vertical coordinate of the center of the true box, y _gt represents the vertical coordinate of the predicted center point area, W _g represents the width of the minimum anchor box, H _g represents the height of the minimum anchor box, W _g represents the width of the minimum anchor box, and H g represents the height of the minimum anchor box; in order to prevent R _WIoU from generating gradients that hinder convergence, W _g and H _g are separated from the calculation graph (superscript * indicates this operation). Because it effectively eliminates factors that hinder convergence, this operation does not introduce new indicators, such as aspect ratio. This loss function makes the regression of the predicted rectangular box more stable, effectively avoids problems such as divergence during training, and accelerates the regression convergence process of the predicted box;

The expression of the confidence loss function Loss _obj is:

Where b represents the confidence loss weight when the element value in the mask matrix is true, loss _BCE (z,x,y) represents the cross entropy loss function, Lable _(z,x,y) represents the confidence label matrix, and P _(z,x,y) represents the prediction confidence matrix;

Classification loss, taking the VisDrone2019 dataset as an example, there are 10 categories. The input image is divided into 80*80 grids, each grid has 3 preset anchor boxes, then the dimension of the prediction probability matrix P is 3*80*80*10; the dimension of the label probability matrix Label is consistent with the prediction probability matrix. Before calculating the loss, the label value needs to be converted into one-hot encoding (One-Hot Encoding), and then smoothed on the basis of one-hot encoding. Label is the one-hot encoding value, a represents the smoothing coefficient, set to 0.1, n represents the number of categories, and the smoothing formula is as follows:

Among them, Label _smooth represents the label probability matrix after label smoothing, P represents the prediction probability matrix, n represents the number of categories, a represents the smoothing coefficient, and the V data set has 10 categories as an example. x, y, and z represent the matrix subscript values, which are used to determine the position of the element. z takes values from 0 to 3, x and y take values from 0 to 80, and n takes values from 0 to 10. The loss calculation formula for each position in the matrix is as follows:

When the grid size is 80×80 and the number of categories is 6, the classification loss function formula is as follows:

Among them, the larger the confidence value is, the higher the possibility that the target object exists in the rectangular frame is.

In an optional embodiment of the present invention, the detection module includes a convolution layer, a pooling layer and a fully connected layer, and the detection module processes the input features including:

The convolution layer processes the first, second and third fusion features of the input, the pooling layer processes the features processed by the convolution layer, and the fully connected layer processes the features processed by the pooling layer to obtain the classification results.

Specifically, in this embodiment, it is finally sent to a 1*1 convolution, a pooling operation and a full connection operation are performed to obtain a classification result, and the network model construction is completed.

In an optional embodiment of the present invention, specifically, please continue to refer to Figure 5, use the VisDrone2019 dataset and the AU-AIR dataset to screen out 10,000 high-quality dataset images, perform simple data enhancements such as random rotation and superimposed noise, and divide them into training set, verification set and test set at 6:2:2. The label content in the dataset is readjusted so that the self-made dataset contains six different categories of vehicle targets in different scenarios. The training set consists of 6,000 images, and the verification set and test set each consist of 2,000 images. An example image of the dataset is shown in Figure 6, which is a schematic diagram of an example image of a dataset provided in an embodiment of the present invention.

Combining the advantages of the current lightweight network and taking into account the network detection performance, the mobile inverted bottleneck convolution (MBConv) module and the designed adaptive frequency band filter operator are used as the main framework of the network. The backbone network is shown in Figure 2. Specifically, a convolution system is used for tokenization, which consists of a 3×3 convolution layer with a stride of 2, followed by four MBConv layers with a kernel size of 3. After tokenization, three stages are cascaded as the main body of the lightweight network, where each stage consists of an MBConv layer with a stride of 2 for spatial downsampling and N _i AFF modules. And the number of AFF modules of the first module is set to N ₁ = 2, the number of AFF modules of the second module is set to N ₂ = 4, and the number of AFF modules of the third module is set to N ₃ = 3. Two groups of linear layers (also called 1×1 convolution layers) and ReLU functions are used to learn the masks of the proposed adaptive frequency filtering. The use of linear layer groups is crucial because it prevents nonlinearity from destroying too much information. This linear layer group uses a 1x1 convolution kernel to further compress the number of channels, reduce the amount of calculation and the number of parameters. Layer normalization (LN) is used for channel mixing, and then it is fed to the proposed adaptive frequency band filter operator for global sequence mixing to obtain the output of the lth AFF module. The jump connection of channel mixing and sequence mixing is used to speed up the training convergence of the model. The backbone network is responsible for feature extraction, and the neck network is responsible for feature fusion. The improved adaptive spatial feature fusion Adaptively Spatial Feature Fusion (ASFF) is introduced as the neck fusion module. Finally, it is sent to the 1*1 convolution, pooling operation and full connection operation to obtain the classification result and complete the network model construction.

Iteratively train the network and save the best weights.

At the output end, the designed loss function is used to establish end-to-end training to make the network converge, save the best training weights, and complete the model training.

The trained network model is predicted, and mean Average Precision (mAP) is used as an evaluation index to judge the network performance. Please refer to FIG7 , which is a schematic diagram of the test results provided by an embodiment of the present invention, (a) represents the original image of the real scene, and (b) represents the test result diagram of the present invention.

In the target detection task, TP (True positives) refers to the positive samples that are correctly identified, TN (Truenegatives) refers to the negative samples that are correctly identified, FP (False positives) refers to the negative samples that are incorrectly identified, and FN (False negatives) refers to the positive samples that are incorrectly identified. Precision (P) refers to the proportion of correct detections among all detected targets. The calculation formula is as follows:

The mean average precision (mAP) is used to measure the average precision of all categories. The higher the mAP value, the better the performance of the network model. mAP is the average value of the average precision (AP) of multiple categories. If there are N categories in a target detection task, the AP of the i-th category is represented by the mAP formula as follows:

A tensor is randomly generated using a deep learning framework and sent to the trained network model obtained above to calculate the parameter quantities Params and FLOPs (used to measure the complexity of the model). In order to more objectively and fully evaluate the advantages of the method of the present invention, two currently advanced lightweight network models MobileNetV3 and ShuffleNetV2 are used to compare the spatial domain convolution method and the method of the present invention on the data set obtained in the above steps. The comparison results are shown in Figure 1.

Table 1 Evaluation indicators on self-made datasets

Params FLOPs mAP ShuffleNetV2 5.5 0.6 44.5 MobileNetv3 5.4 0.2 45.2 Using spatial convolution 10.7 2.7 48.6 The present invention 5.5 1.5 49.8

Compared with the most advanced lightweight network models, the present invention sacrifices a small amount of computation while having similar parameters to the advanced lightweight models ShuffleNetV2 and MobileNetv3, and the accuracy of the network for small vehicle targets is improved. When the adaptive frequency band filter operator of the present invention replaces the equivalent spatial convolution, the number of parameters will decrease, the computational cost will also decrease, and the detection accuracy will also increase. It can meet the requirements of real-time and accuracy of small vehicle target detection at the edge where storage and computing are limited. The accuracy and efficiency of the present invention can achieve a better trade-off.

In summary, in order to solve the problems that the existing deep convolutional neural network has large parameters and high computational cost, and the shallow lightweight network has weak feature extraction ability and poor detection performance for the target, a vehicle small target detection method with lightweight network design is proposed. While reducing the computational cost, the feature extraction of the target has the advantages of high computational efficiency, semantic adaptation and global effectiveness. The output end constructs an attention-based bounding box loss function to make the positioning more accurate and increase the generalization ability of the model. While reducing the complexity of the model, the entire network can meet the requirements of real-time and accuracy of vehicle target detection at the edge where storage and computing are limited.

Based on the same inventive concept, FIG8 is a schematic diagram of a vehicle small target detection device with a lightweight network design provided by an embodiment of the present invention. The present invention also provides a vehicle small target detection device with a lightweight network design, which is applied to the vehicle small target detection method with a lightweight network design provided by the above embodiment of the present invention. The embodiment of the method is referred to above, and the present invention will not be repeated here; the device includes:

An image acquisition module 101 is used to acquire an image of a vehicle to be tested;

The image processing module 201 is used to classify the image of the vehicle to be tested using the trained network model and output the classification result; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain the classification result.

It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that the article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the article or device including the elements. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The orientation or position relationship indicated by "up", "down", "left", "right", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims (10)

1. A vehicle small target detection method with a lightweight network design, characterized by comprising: Acquire an image of a vehicle to be tested; Use the trained network model to classify the vehicle image to be tested and output the classification result; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain the classification result. 2. The vehicle small target detection method with lightweight network design according to claim 1 is characterized in that the backbone network module includes a first module, a second module and a third module, the vehicle image to be detected input into the backbone network module is processed by the first module to obtain a first feature, the first feature is input into the second module for processing to obtain a second feature, the second feature is input into the third module for processing to obtain a third feature; wherein, The first module, the second module and the third module all include a downsampling module, an AFF module and a fusion module. The number of AFF modules set in the first module, the number of AFF modules set in the second module and the number of AFF modules set in the third module are all different. The downsampling module includes an MBConv layer. 3. The vehicle small target detection method with lightweight network design according to claim 2 is characterized in that the AFF module includes a first normalization layer, a second normalization layer and an adaptive frequency band filter operator; wherein the first normalization layer and the second normalization layer are used for channel mixing, the adaptive frequency band filter operator is used for global sequence mixing, and the characteristic expression output by the AFF module is: in, represents the output features after the lth downsampling layer and the first normalization layer, MBConv ^l (·) represents the output features after processing by the lth MBConv layer, X ^ll represents the features input to the downsampling layer, AFF ^l (·) represents the output features after processing by the lth AFF module, LN (·) represents the output features after processing by the second normalization layer, and X ^l represents the final output features after processing by the lth AFF module. 4. The vehicle small target detection method with lightweight network design according to claim 3 is characterized in that the adaptive frequency band filter operator includes a Fourier fast transform module, a frequency domain filter and an inverse Fourier transform module; the processing process of the adaptive frequency band filter operator on the input self-features includes: The Fourier transform module is used to transform the input features from the spatial domain to the frequency domain to obtain the frequency domain features, which are expressed as follows: Among them, X _F represents the frequency domain features, X represents the input features, represents Fourier transform; The frequency domain feature is processed using the frequency domain filter to obtain a mask tensor for learning the frequency domain feature X _F The mask tensor With the frequency domain characteristics Perform point multiplication to obtain the filtered features Its expression is: Among them, ⊙ represents matrix product; The inverse Fourier transform module is used to transform the filtered features from the frequency domain to the spatial domain to obtain the spatial domain features. Its expression is: in, represents the inverse Fourier transform, the spatial domain features That is, the feature output after being processed by the adaptive frequency band filter operator. 5. The vehicle small target detection method with lightweight network design according to claim 2 is characterized in that the neck network module includes three ASFF modules, namely a first ASFF module, a second ASFF module and a third ASFF module; wherein the neck network module processes the input features including: The first ASFF module fuses the first feature, the second feature and the third feature to obtain a first fused feature, the second ASFF module fuses the first feature, the second feature and the third feature to obtain a second fused feature, and the third ASFF module fuses the first feature, the second feature and the third feature to obtain a third fused feature. 6. The vehicle small target detection method with lightweight network design according to claim 1 is characterized in that the characteristic expression output by the ASFF module is: in, Represents the new l-th layer feature map obtained after fusion, It represents the feature vector with the same size and dimension as the feature map of the lth layer after the size and dimension of the feature map of the nth layer are adjusted. and They represent the weight coefficients of the lth layer obtained by adaptive learning, The expression of the weight coefficient is: in, Indicates the definition of level l The self-learning weight control parameters, Indicates the definition of level l The self-learning weight control parameters, Indicates the definition of level l The self-learning weight control parameters. 7. The vehicle small target detection method with lightweight network design according to claim 1 is characterized in that the trained network model is obtained by training a preset network model, and the training of the preset network model comprises: Acquire a data set, wherein the data set includes a training sample set and a test sample set; Using the training sample set to train the preset network model; During the training process, the loss function is used for iterative optimization to obtain the optimal weight of the preset network model to construct the trained network model; The performance of the trained network model is tested using the test sample set. 8. The vehicle small target detection method with lightweight network design according to claim 7 is characterized in that the loss function includes a boundary loss function, a confidence loss function and a classification loss function, and the expression of the loss function is: Loss＝ _LWIoUv +Loss _cls +Loss _obj ; The expression of the boundary loss function L _WIoUv is: Among them, R _WIoU represents the coefficient based on distance setting, L _IoU represents the initial intersection-over-union loss function, x represents the horizontal coordinate of the center of the true box, x _gt represents the horizontal coordinate of the predicted center point area, y represents the vertical coordinate of the center of the true box, y _gt represents the vertical coordinate of the predicted center point area, W _g represents the width of the minimum anchor box, and H _g represents the height of the minimum anchor box; The expression of the confidence loss function Loss _obj is: Where b represents the confidence loss weight when the element value in the mask matrix is true, loss _BCE (z,x,y) represents the cross entropy loss function, Lable _(z,x,y) represents the confidence label matrix, and P _(z,x,y) represents the prediction confidence matrix; The expression of the classification loss function loss _cls is: Among them, Label _smooth represents the label probability matrix after label smoothing, P represents the prediction probability matrix, n represents the number of categories, and a represents the smoothing coefficient. 9. The vehicle small target detection method with lightweight network design according to claim 5 is characterized in that the detection module includes a convolution layer, a pooling layer and a fully connected layer, and the processing process of the input features of the detection module includes: The convolution layer processes the first fusion feature, the second fusion feature and the third fusion feature of the input, the pooling layer processes the features processed by the convolution layer, and the fully connected layer processes the features processed by the pooling layer to obtain a classification result. 10. A vehicle small target detection device with a lightweight network design, characterized by comprising: An image acquisition module, used to acquire an image of a vehicle to be tested; An image processing module is used to classify the vehicle image to be tested using a trained network model and output a classification result; wherein the trained network model includes a backbone network module, a neck network module and a detection module, the backbone network module is used to extract hierarchical global features and local features, the neck network module is used to fuse hierarchical global features and local features, and the detection module is used to process the fused global features and local features to obtain a classification result.