CN115601657A - A method for ship target detection and recognition in bad weather - Google Patents

Abstract

The invention relates to the fields of image defogging, target detection and the like, and provides a method applied to ship target detection and identification in severe weather, aiming at the problems of difficult ship detection, low precision and the like in severe weather. The method comprises two stages: firstly, converting a fog image into a clear fog-free image through a defogging model; secondly, an improved detection network is used for carrying out target detection tasks on the processed clear input, and interesting ship targets are identified and positioned. The defogging model consists of a CNN branch, a transformer branch and a fusion branch. The CNN branch is responsible for local feature extraction, the transformer branch is used for long-distance global feature dependence, and the fusion branch realizes feature adaptive mode fusion. The detection model is based on an original YOLOV5 framework, and a multi-branch convolution structure is used for replacing an original feature extraction module, so that the detection performance is improved. The method can solve the problems of low detection precision and the like of the ship target in severe weather, and has higher practical value.

Description

A Method for Ship Target Detection and Recognition in Bad Weather

technical field

The present invention mainly relates to technical fields such as intelligent image processing, deep learning, and target detection. Combining image defogging and target detection models, a method for ship target detection and recognition in severe weather is proposed.

Background technique

Object detection technology is one of the important branches in the field of computer vision, and it has gradually become a key research issue of various universities and research institutes. In recent years, target detection algorithms based on single-stage and two-stage strategies have emerged in an endless stream, and have achieved excellent detection results, which are effectively used in military and civilian fields such as border security, crowd analysis, and traffic flow estimation. Specifically, the existing target detection technology mainly uses clear and clean images as the target input, that is, the images input to the detection network are not disturbed by noise, which ensures high image quality. However, due to complex and harsh weather conditions, during the ship target detection and recognition process, the acquired remote sensing images are easily blocked by clouds, haze and other vapors, which greatly affects the quality of the image, thus affecting the follow-up The detection task causes problems such as false detection and missed detection. Therefore, how to improve the detection effect of ship targets in severe weather and improve the anti-interference ability of the detection algorithm to severe weather conditions has become a key problem that needs to be solved urgently.

Contents of the invention

In order to meet the needs of the target detection algorithm in foggy days, solve the problems of low detection rate and high missed detection rate caused by cloud and fog occlusion, and achieve better ship target detection performance. The invention provides a method for ship target detection and recognition in bad weather.

A method for ship target detection and recognition in severe weather, including: in the first stage, the obtained fogged remote sensing image is cleared, that is, the fogged image is converted into a clear and fog-free image by a dehazing model Image; in the second stage, the obtained clear image is used as the image input of the detection network, that is, an improved detection network is used to perform target detection tasks on the processed clear input to identify and locate the ship target of interest.

The first stage described includes: the construction of fog-clear data set based on the atmospheric scattering model, the establishment of the dehazing model combined with CNN and transformer structure, the design of global feature and local feature fusion module; used to realize the defog image fog treatment.

The second stage described includes: the collection and labeling of remote sensing ship data, the use of improved YOLOv5 for ship target detection tasks, the training and test deployment of the detection model.

In addition, the dehazing model in the first stage and the detection model in the second stage adopt the mode of cascading end-to-end simultaneous training, that is, the loss function is combined and added to optimize the overall model. Specifically, the model total loss function is as follows:

L _total =L _h +L _d (1.1)

Among them, L _total represents the total loss function of the model; L _h represents the loss function of the dehazing model; L _d represents the loss function of the target detection model.

The technical scheme adopted in the present invention is, a kind of method that is applied to ship target detection and recognition under bad weather, specifically implements according to the following steps:

The first-stage dehazing model shown above:

Step 1: Fog-clear dataset construction based on atmospheric scattering model. In the existing dehazing algorithm research, the atmospheric scattering model as shown below is often used,

I(x)=J(x)t(x)+A(1-t(x)) (1.2) Among them, I(x) represents the fogged image, J(x) represents the clear and fog-free image, t(x ) represents the transmission matrix, and A represents the atmospheric scattering coefficient. From the parameter analysis, it can be known that the clear fog-free image is known, and the corresponding fogged image can be simulated and generated by setting the corresponding atmospheric scattering coefficient and projection coefficient by using its depth map. Therefore, the collected remote sensing images are subjected to the fogging process in the above formula to construct a fogged-clear remote sensing image data set for training and testing.

Step 2: Establish a dehazing model that combines CNN and Transformer structures. Thanks to the local feature perception ability of the convolutional neural network CNN, it can be effectively applied to various image restoration tasks. However, the CNN structure lacks the ability to capture long-distance dependencies. Existing methods mainly alleviate this problem by increasing the number of layers of the network, but this simple idea can easily lead to network redundancy and loss of more local details. . The proposal of Transformer can solve this problem very well, and the global dependence of features is well described through the self-attention mechanism. Therefore, the proposed defogging model is composed of CNN branch and Transformer branch, making full use of the advantages of their respective structures to obtain a better defogging effect. In addition, using the fusion branch, the obtained features are superimposed and fused to achieve stronger feature representation capabilities.

Step 3: Design of global feature and local feature fusion module. In the process of feature fusion, the effective features extracted by CNN and Transformer structures are used at the same time, which makes the fused features more compact and the network expression ability stronger. Specific operations include channel attention module, spatial attention module, conventional convolution operation, residual connection, etc.

The second-stage object detection model shown above:

Step 1: Collection and labeling of remote sensing ship data. First, capture the remote sensing image containing the ship target on the satellite map to obtain the original image data. Secondly, use the labeling tool to mark the ship target in the collected image data to get the corresponding image features. Finally, one-to-one correspondence between the image and the marked target position information is established to establish a remote sensing ship target data set;

Step 2: Use the improved YOLOv5 for the ship target detection task. In order to meet the requirements of the algorithm's subsequent deployment of airborne hardware, the original YOLOv5 feature backbone extraction network is simplified and adjusted. Among them, based on the topology paradigm, the original feature extraction module is replaced by a multi-branch convolution structure. At the same time, the operators of the overall backbone network are adjusted to achieve efficient reasoning on hardware while maintaining efficient multi-scale feature fusion capabilities;

Step 3: Detection model training and test deployment. Use the marked ship data set to train the detection model, and obtain the corresponding training weight after the model training is completed. To speed up the inference speed of the model, perform the TensorRT configuration operation. Among them, TensorRT can be regarded as a deep learning framework with only forward propagation reasoning. The framework can analyze the network model of Caffe and TensorFlow, and then perform one-to-one mapping with the corresponding layers in TensorRT, and uniformly convert the models of other frameworks to TensorRT. In order to accelerate the deployment.

The present invention is also characterized in that:

The dehazing model and the ship target detection model adopt a joint training mode. First, the weights of the dehazing sub-network are initialized with Gaussian random variables. Secondly, the target detection subnetwork does not perform random initialization of weights, but fine-tunes the model obtained through pre-training on the COCO dataset by downsampling to achieve weight initialization. Finally, the overall model is trained end-to-end on the constructed dataset, simultaneously learning image dehazing enhancement, ship object classification and localization purposes.

In summary, the main contribution of the present invention is:

(1) The present invention provides a method for ship target detection and recognition in severe weather, cascading the defogging model and the detection model, the fogged image is used as the input of the defogging model, and the clear output of the defogging model is used as In the next stage, the input of the detection model is output, and the category positioning result of the ship target is obtained as an output.

(2) The present invention provides a method for ship target detection and recognition in bad weather. The dehazing model used is composed of CNN and Transformer structures, which includes three branches: CNN branch, Transformer branch and fusion branch . Fully integrate and utilize the structural characteristics of CNN and Transformer to achieve better defogging effect.

(3) The present invention provides a method for ship target detection and recognition in severe weather, using the topology paradigm to carry out lightweight adjustments to the detection model, reducing network memory, and deploying the trained model in TensorRT, Accelerate model inference on hardware.

Description of drawings

Fig. 1 is a schematic structural diagram of a method for ship target detection and identification under bad weather proposed by the present invention;

Fig. 2 is a schematic diagram of the overall structure of the defogging method model proposed by the present invention;

Fig. 3 is a schematic diagram of the feature fusion structure proposed by the present invention;

detailed description

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be pointed out that the described embodiments are only intended to facilitate the understanding of the present invention, rather than limiting it in any way. The accompanying drawings all adopt very simplified forms and use inaccurate scales, and are only used to facilitate and clearly assist the purpose of illustrating the embodiments of the present invention, and the structures shown in the accompanying drawings are part of the actual structure. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

The present invention provides a method for ship target detection and recognition in bad weather. The overall model of the proposed method is shown in Figure 1, which includes two stages: the first stage, the obtained atomized remote The image is cleared, that is, the fogged image is converted into a clear and fog-free image through a dehazing model; in the second stage, the clear image obtained by the dehazing model is used as the image input of the detection network, that is, an improved detection The network performs the target detection task on the processed clear input to identify and locate the ship target of interest. Through the above technical steps, the purpose of intelligent detection and positioning of ship targets in severe weather is realized.

Fogged remote sensing image defogging and clearing processing stage, the fogged image is restored to a clear fog-free image.

The physical mechanism of the atmospheric scattering model is analyzed, and the fog-clear image data pairs used for training and testing are constructed in the form of simulation generation. Among them, it mainly includes two steps of original remote sensing image acquisition and fog image construction. Raw remote sensing image collection: through satellite remote sensing maps, intercept and obtain remote sensing images of interest. Note that the images contain ship targets as much as possible, which is convenient for subsequent construction of ship target detection data sets. Fog image construction: calculate the respective depth maps from the collected remote sensing images, set the corresponding atmospheric scattering coefficient and transmission coefficient, and based on the atmospheric scattering model, simulate and calculate the corresponding fog image. In order to better simulate the realistic haze scene, the parameter setting should cover the interval from 0 to 1 as much as possible to meet the needs of remote sensing images with different concentrations of haze.

Build a dehazing model to achieve the purpose of image dehazing. The proposed dehazing model is shown in Figure 2, which includes three branches: CNN branch, transformer branch and fusion branch.

Specifically, in the CNN branch, feature accumulation is achieved by stacking N layers of conventional residual modules, where each residual module is preceded by a pooling layer with a downsampling coefficient of 2 to reduce the size of the feature . The feature representation extracted by CNN is mainly about the local features of the image, such as contours, boundaries, etc., and cannot perceive its global information well. Therefore, the transformer branch is needed to strengthen and supplement the features.

被划分为N个图像块，图像尺寸变 换为

S的大小设置为16。随后，将这些图像块进行拉伸并输入到一个 嵌入层中，输出得到一个嵌入向量

其中，D表示维度大小。为保证图 像的空间先验只是，加入一个可学习的位置编码向量，其大小与图像嵌入向量 保持一致，将两者结合作为编码器的共同输入。编码器由多头注意力模块与多 层感知器组成，其中，自注意力模块作为transformer的核心操作，可以表示为：Specifically, in the transformer branch, the transformer structure follows the classic codec structure. First, in the encoder, the input image is

is divided into N image blocks, and the image size is transformed into

The size of S is set to 16. Subsequently, these image patches are stretched and input into an embedding layer, which outputs an embedding vector

Among them, D represents the dimension size. In order to ensure the spatial prior of the image, a learnable position encoding vector is added, whose size is consistent with the image embedding vector, and the two are combined as the common input of the encoder. The encoder consists of a multi-head attention module and a multi-layer perceptron. The self-attention module is the core operation of the transformer, which can be expressed as:

在解码器中，我们遵循时序的上采样方法，即逐渐提升特 征的尺寸大小。具体来说，首先将特征返回原始的尺寸大小

然后 使用上采样的卷积层依次提升特征的空间分辨率，最后将所获取得到特征按照 其尺寸大小传递给下阶段的融合模块。In addition, in addition to the above operations, we also added an instance regularization operation. Finally, the output of the encoder is obtained

In the decoder, we follow a temporal upsampling method, which gradually increases the size of the features. Specifically, first return the features to their original size

Then use the upsampled convolutional layer to sequentially increase the spatial resolution of the features, and finally pass the acquired features to the fusion module in the next stage according to their size.

与

依次表示这两个特 征流。其中，H,W,C分别表示特征的高宽及通道数大小。首先，考虑到F_c只 包含有局部的特征信息，利用空间注意力模块加强特征像素级别的依赖作用； 考虑到F_t包含有更多全局依赖信息，利用通道注意力模块，增强特征在不同通 道之间的相互作用关系。其次，将特征流进行像素级别的相乘操作，探索特征 之间的相互作用，再利用残差模块进行特征加强作用。最后，将这三种特征流 进行通道拼接处理，得到最终融合结果。Specifically, in the fusion branch, the specific steps are shown in Figure 3. The two features come from the CNN branch and the transformer branch respectively, using

and

These two feature streams are represented in turn. Among them, H, W, and C represent the height and width of the feature and the number of channels, respectively. First of all, considering that F _c only contains local feature information, the spatial attention module is used to strengthen the dependence of the feature pixel level; considering that F _t contains more global dependency information, the channel attention module is used to enhance features in different channels the interaction between them. Secondly, the feature stream is multiplied at the pixel level to explore the interaction between features, and then the residual module is used to enhance the feature. Finally, the three feature streams are processed by channel splicing to obtain the final fusion result.

Build a detection model to achieve the purpose of ship target recognition and positioning. Among them, the following three steps are included: the collection and labeling of remote sensing ship data, the use of improved YOLOv5 for ship target detection tasks, the training and test deployment of detection models

Specifically, the collection and labeling process of remote sensing ship data mainly relies on manual operations. On the open source Google satellite map, capture remote sensing images containing ship targets, and save all remote sensing images at the same resolution. Using the labeling software labelme, the task of labeling the acquired remote sensing image ship target is carried out to obtain the xml file with position and size information.

Specifically, in the target detection stage, the method uses the improved YOLOv5 for ship target detection tasks. In the structure of the decoupling detection head, the design is simplified. The detection head of the original YOLOv5 is realized through the fusion and sharing of classification and regression branches. Although the accuracy of target detection is improved, the network delay is increased to a certain extent, and the reasoning speed is slowed down. Therefore, the method comprehensively considers the balance between the representation ability of related operators and the computing overhead on the hardware, and adopts the Hybrid Channels strategy to redesign a more efficient decoupling head structure, which reduces the delay while maintaining accuracy. , which alleviates the extra delay overhead caused by the conventional convolution in the original decoupling head, and improves the speed of the network.

Specifically, during the training and test deployment of the ship target detection model, the proposed detection method is transplanted and deployed in TensorRT. The deployment is divided into two phases: the model preprocessing phase and the model inference phase. The general process is as follows: 1) Export the detection network definition and related trained weights; 2) Re-analyze the detection network definition and related weights; 3) Construct the optimal execution plan according to the graphics card operator used; The serialization of the plan is stored in the graphics card; 5) The reverse serialization executes the target detection plan; 6) The forward reasoning process of the ship detection model is carried out.

In summary, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (6)

1. A method for detecting and identifying ship targets in severe weather is characterized by comprising the following steps:

step 1: cascading the defogging model and the detection model, wherein the fogging image is used as the input of the defogging model, the clear output of the defogging model is used as the input of the detection model of the next stage, and the category positioning result of the ship target is obtained through output;

step 2: the defogging model is synthesized by combining CNN and Transformer structures, and comprises three branches: CNN branch, transformer branch and fusogenic branch. Structural characteristics of CNN and Transformer are fully fused and utilized to realize more excellent defogging effect;

and 3, step 3: and the detection model is adjusted in a light weight manner by utilizing the topological structure paradigm, the network memory is reduced, and in addition, the trained model is deployed in a TensorRT manner, so that the reasoning speed of the model on hardware is accelerated.

2. The cascade defogging model and the detection model in the step 1 are characterized in that: and initializing the weight of the defogging subnetwork by using a Gaussian random variable in a joint training mode. The target detection subnet does not use random initialization weight operation, but carries out class down-sampling fine adjustment on a model obtained by pre-training on a COCO data set so as to realize weight initialization. The whole model carries out end-to-end training on the constructed data set and simultaneously learns the purposes of image defogging, ship target classification and positioning.

3. The defogging model construction according to the step 2 is characterized in that: in the CNN branch, local feature information about an image, such as contours, boundaries, etc., is mainly characterized by stacking N layers of conventional residual modules to achieve feature accumulation.

4. The defogging model construction method according to the step 2 is characterized in that: in the transform branch, based on the coding and decoding structure, the long-distance dependent global feature representation is realized by using operations such as a multi-head self-attention module, a multi-layer perceptron and an upsampling convolutional layer. Supplementing the characteristics of the CNN branch to achieve better defogging performance.

5. The defogging model construction according to the step 2 is characterized in that: in the feature fusion branch, based on a feature adaptive fusion strategy, a channel attention module, a space attention module, a conventional convolution operation, residual connection and the like are utilized, and effective features extracted by CNN and Transformer structures are utilized simultaneously.

6. The construction of the ship target detection model in the step 3 is characterized in that: and replacing the original feature extraction module by using a multi-branch convolution structure based on the topological structure paradigm. Meanwhile, operators of the whole backbone network are adjusted, and high-efficiency multi-scale feature fusion capability is kept.

Images