CN116503799A - Contact net dropper defect detection method based on CNN and Transformer fusion - Google Patents

Abstract

The invention discloses a catenary suspension string defect detection method based on the fusion of CNN and Transformer, and relates to the technical field of defect detection. The method collects the catenary suspension string image and performs image enhancement processing to obtain a suspension string defect sample set; The convolutional network constructs the convolutional module, and at the same time builds the self-attention module based on the improved high-efficiency multi-head self-attention mechanism, and deeply integrates the convolutional module and the self-attention module based on the optimal module allocation ratio to generate a multi-block cross hybrid network to Improve the FasterRCNN network. Based on the suspension string defect sample set, the improved network is trained and verified, the trained model is obtained and deployed in the suspension string detection equipment, the catenary suspension string image is captured in real time and input into the suspension string detection equipment for suspension string defect detection, Identify catenary dropper defects. The application can be applied to complex natural scene environments, and improves the accuracy and recall rate of suspension string defect identification under real complex natural conditions.

Description

Catenary suspension string defect detection method based on CNN and Transformer fusion

technical field

The present invention relates to the technical field of defect detection, in particular to a method for detecting defects of suspension strings of high-speed railway catenary based on deep cross-fusion of CNN and transformer.

Background technique

The suspension string is a key component of the high-speed rail catenary, which ensures the smooth and continuous flow of the high-speed rail EMU, and relieves the vibration between the contact wire and the catenary cable. However, the suspension string is affected by temperature, climate, high-frequency vibration, etc., and the suspension string frequently loosens, breaks, and falls off, which may affect the current flow of the pantograph at the slightest level, or damage the pantograph or damage the contact wire at the worst, resulting in train failure. Therefore, Real-time detection and early warning of suspension string defects to ensure the safety and reliability of the catenary is of great significance to the safe operation of high-speed railways.

Most of the existing catenary suspension string defect detection methods use machine learning and neural network models to detect suspension string defects, but there are the following shortcomings:

(1) Affected by the actual scene, such as in rainy days, foggy days, strong sunlight, night and other natural scene environments, the defect recognition effect on suspension strings is poor;

(2) When the suspension strings are blocked, the defects of the suspension strings cannot be accurately identified, and there are cases where defects are missed.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art, to provide a catenary suspension string defect detection method based on the fusion of CNN and Transformer, which helps to solve the problem of the current catenary suspension string defect detection method being blocked and complex scenes The effect of defect recognition is not good, and the recognition accuracy is low.

The purpose of the present invention is achieved through the following technical solutions:

The invention provides a catenary suspension string defect detection method based on the fusion of CNN and Transformer, including:

Collecting catenary suspension string images and performing image enhancement processing on the suspension string images to obtain suspension string defect sample sets;

Construct a convolution module using a variable convolutional network based on constraints, and construct a self-attention module based on an improved high-efficiency multi-head self-attention mechanism, and deeply fuse the convolution module with the self-attention module based on the optimal module allocation ratio , generating a multi-block cross-hybrid network;

Using the multi-block cross hybrid network to improve the FasterRCNN network, and based on the hanging string defect sample set, the improved FasterRCNN network is trained and verified, and the trained FasterRCNN improved model is obtained;

The improved FasterRCNN model is deployed in the suspension string detection equipment, and the catenary suspension string image of the high-speed railway is captured in real time and input to the suspension string detection equipment for suspension string defect detection, and the suspension string defect in the catenary is identified.

Further, the collecting the catenary suspension string image and performing image enhancement processing on the suspension string image to obtain the suspension string defect sample set specifically includes:

Collect the catenary hanging string image during the operation of the high-speed railway;

Based on the improved image enhancement algorithm, image enhancement processing is performed on the catenary hanging string image, N mask areas are randomly generated in any hanging string image, and the mask area that blocks the key features of the hanging string is filtered according to the number N of mask areas. Obtain a sample set of hanging string defects;

The hanging string defect sample set is divided into training samples and validation samples.

Further, according to the number N of mask regions, filtering the mask regions that block the key features of the hanging strings specifically includes:

If N=1, filter out the single mask area that completely blocks the hanging string in the hanging string image;

If N=2, then filter out the two mask regions that block the upper and lower ends of the hanging string in the hanging string image;

If N≥3, filter out a single mask area that completely blocks the hanging string in the hanging string image, and filter out any two mask areas that block the upper and lower ends of the hanging string in the hanging string image.

Further, the convolution module is constructed using a variable convolution network based on constraints, and the self-attention module is constructed according to the improved high-efficiency multi-head self-attention mechanism, and the convolution module is combined with the self-attention module based on the optimal module allocation ratio. The attention module is deeply fused to generate a multi-block cross-hybrid network, specifically including:

According to the aspect ratio of the height-width ratio of the hanging string, the constraint relationship is used to constrain the height-width coordinate ratio of the sampling point position in the variable convolutional network, and at the same time limit the height-width coordinates of the sampling point position to not exceed the height-width of the input feature map , obtain a constraint-based variable convolution network, and construct a convolution module through a constraint-based variable convolution network;

Space dimensionality reduction operations are performed on the key vector K and value vector V in the original Transformer self-attention mechanism to obtain an improved high-efficiency multi-head self-attention mechanism and build a self-attention module based on the improved high-efficiency multi-head self-attention mechanism;

Based on the backbone network architecture in the FasterRCNN model, the number of convolution modules and the number of self-attention modules in the backbone network architecture are allocated according to the optimal module allocation ratio, and the convolution modules and self-attention modules are fused in a new paradigm to generate multiple blocks cross-hybrid network.

Further, the optimal module allocation ratio is specifically: number of convolution modules: number of self-attention modules=7:2.

Further, the improved FasterRCNN network is improved by using the multi-block cross hybrid network, and the improved FasterRCNN network is trained and verified based on the hanging string defect sample set, and the trained FasterRCNN improved model is obtained, which specifically includes:

The backbone network architecture of the FasterRCNN network is replaced by the network architecture of the multi-block cross hybrid network to obtain the improved FasterRCNN model;

The improved Faster RCNN network is trained based on the training samples in the hanging string defect sample set. After the training is completed, the model is verified by the verification samples, and the trained and verified Faster RCNN improved model is obtained.

Beneficial effects of the present invention: the present invention provides a catenary suspension string defect detection method based on the fusion of CNN and Transformer. The method collects the catenary suspension string image and performs image enhancement processing on the suspension string image to obtain a suspension string defect sample set; Construct a convolution module using a variable convolutional network based on constraints, and construct a self-attention module based on an improved high-efficiency multi-head self-attention mechanism, and deeply fuse the convolution module with the self-attention module based on the optimal module allocation ratio , generate a multi-block cross-mixed network; use the multi-block cross-mixed network to improve the FasterRCNN network, and based on the hanging string defect sample set, the improved FasterRCNN network is trained and verified to obtain a trained FasterRCNN improved model; The improved FasterRCNN model is deployed in the suspension string detection equipment, and the catenary suspension string image of the high-speed railway is captured in real time and input to the suspension string detection equipment for suspension string defect detection, and the suspension string defect in the catenary is identified. This application obtains the hanging string defect sample set by performing image enhancement processing on the hanging string image, increases the hanging string defect samples, and solves the problem that the hanging string is blocked and the number of hanging string training samples is small. At the same time, the application meets the characteristic requirements of the hanging string and improves the ability to identify the defect of the hanging string by using a variable convolution network based on constraints to construct a convolution module. At the same time, the self-attention module is constructed according to the improved high-efficiency multi-head self-attention mechanism, and the convolution module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross hybrid network, and CNN and transformer are cross-fused to achieve Solve the problem of poor defect recognition effect in natural scene environments such as rainy days, foggy days, strong sunlight, and night, and can also accurately identify the defects of suspended strings that are blocked and small long-distance targets, thereby improving the detection of hanging string defects recall and precision rates.

Description of drawings

Fig. 1 is the catenary suspension string defect detection method flow chart based on CNN and Transformer fusion of the present invention;

Figure 2 is a schematic diagram of the existing types of suspension string defects;

Fig. 3 is a schematic diagram of the identification problem of hanging string defects in a real application environment;

Figure 4 is an example diagram of the existing model misidentifying the hanging string;

Figure 5 is a diagram of the FasterRCNN improved model architecture;

Fig. 6 is a sample diagram of a hanging string image after image enhancement;

Figure 7 is a schematic diagram of convolutional sampling using different convolutional neural networks for defect identification;

Fig. 8 is a schematic diagram of the principle of bilinear interpolation;

Figure 9 is a comparison diagram of the backbone network architecture of a traditional hybrid network and a multi-block cross-fusion hybrid network;

Figure 10 is a schematic diagram of the network structure formed by the cross fusion of CB blocks and TB blocks in a single stage of the backbone network of the present application;

Fig. 11 is a schematic diagram of the CB block structure of the present invention;

Fig. 12 is a schematic diagram of the TB block structure of the present invention.

Detailed ways

In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation manners of the present invention will now be described with reference to the accompanying drawings.

The catenary is an important part of the electrified railway and the source of power for the high-speed operation of the train. The suspension string is the transmitter of vibration, force and current between the catenary wire and the catenary cable. It is an important component to improve the current and force performance of the catenary. Includes hanging string and hanging string clips at both ends of the hanging string. In the flexible catenary system, the hanging string can increase the suspension points of the catenary in each span without adding pillars, so as to improve the overall elasticity of the catenary and the sag of the catenary.

In the process of railway transportation, once the suspension string breaks, if it cannot be found and repaired in time, the contact line at the position of the suspension string breakage will sink, causing the contact line to be non-parallel to the track, which will affect the flow of the pantograph and completely After the broken hanging string droops, under the action of wind, it is easy to get entangled with the contact wire, which will damage the pantograph or damage the contact wire, resulting in train operation failure and endangering the lives and property of passengers. Therefore, it is of great significance for the safe operation of high-speed railways to detect and warn the suspension string defects in real time to ensure the safety and reliability of the catenary.

Therefore, this application cross-integrates CNN and transformer, which can identify the defects of hanging strings in natural scene environments such as rainy days, foggy days, strong sunlight, and nights, and can also detect hidden and long-distance small targets. Accurate identification of string defects improves the recall rate and precision of hanging string defect identification.

Referring to Figure 1, Figure 1 shows a catenary suspension string defect detection method based on the fusion of CNN and Transformer, including:

S1: Collect the catenary suspension string image and perform image enhancement processing on the suspension string image to obtain a suspension string defect sample set;

S2: Construct a convolution module using a variable convolutional network based on constraints, and construct a self-attention module based on an improved high-efficiency multi-head self-attention mechanism, and combine the convolution module with the self-attention module based on the optimal module allocation ratio Deep fusion to generate multi-block cross hybrid network;

S3: Utilize the multi-block cross hybrid network FasterRCNN network to improve, and train and verify the improved FasterRCNN network based on the hanging string defect sample set, and obtain a trained FasterRCNN improved model;

S4: Deploy the improved FasterRCNN model on the suspension string detection equipment, capture the catenary suspension string image of the high-speed railway in real time and input it into the suspension string detection equipment to detect suspension string defects, and identify the suspension string defects in the catenary.

Among them, the existing FasterRCNN model applied to complex real scenes cannot meet the requirements of hanging string defect detection, and there are many problems of low recall rate and low precision recognition, and real-time image capture under high-speed motion requires low inference delay. The reasoning delay of the existing FasterRCNN model cannot meet this requirement. Therefore, this application mainly realizes the defect detection of hanging strings by improving the FasterRCNN network.

Specifically, as shown in Figure 2, since the suspension strings are exposed to the external environment for many years, they are greatly affected by climate factors and operating environments. There are problems such as breakage and shedding, and the number of defects increases geometrically. Among them, there are many types of suspension string defects. For the convenience of classification and identification, we divide them into five major defects: suspension string breakage, suspension string drop-off (the upper end of the suspension string clip falls off, and the lower end falls off), suspension string bending, suspension string The installation of loose strands and hanging strings is not standardized. Drawings in part a1 of Fig. 2 show defects of broken suspension strings, drawings of parts d1 show defects of bent suspension strings, drawings of parts e1 and f1 show defects of loose suspension strings, b1 and c1 Part of the drawings show the defects of the upper shedding and the lower shedding of the suspension string respectively.

Due to the wide coverage and long mileage of China's high-speed rail, the climate and landforms are complex and diverse. Affected by factors such as natural weather, seasonal changes, illumination, and suspension strings being blocked, it is very difficult to identify suspension string defects in practical application scenarios. Figure 3 lists the problems encountered in the identification of hanging strings in practical applications. Part a2 in Figure 3 presents the problem of image dragging and blurring when the train is moving at high speed; part b2 shows the problem that the tunnel is dark and the whole image is dim; part c2 shows that the lens is dirty , the problem that part of the image is occluded; the drawings in part d2 show that the image encounters a complex background, and the target object is intertwined with the background, and it is difficult to distinguish the problem of the hanging string target; the drawings in parts e2 and f2 show the hanging string in the image The problem of being blocked by the pantograph or being cut off by the arc. Parts of the drawings in g2, h2, i2 and j2 show the problem that the hanging string image is blurred and difficult to identify due to the influence of heavy fog, night, rainstorm, strong sunlight and other bad weather. It can be seen that in the real application environment, the identification process of hanging strings will encounter various complex situations, which greatly increases the difficulty of identifying the defects of hanging strings.

With reference to Fig. 4, and prior art adopts FasterRCNN and its optimization and improvement model, or CNN and transformer fusion model carry out defect recognition to suspension string, all there is following problem: 1, the positioning tube, electric connecting line joint, electric connecting line and The windproof stay wire is misidentified as a suspension string, as shown in the attached drawings of a3, b3, d3 and e3 in Figure 4; 2. After the suspension string is blocked, it is misidentified as a broken suspension string, as shown in Figure 4 as c3 Part of the drawings are shown; 3. Missing recognition of long-distance small targets, as shown in part f3 of Fig. 4 . Aiming at the problems of low recall and precision in hanging strings and their defects in real scenarios, the existing models cannot meet the requirements for the identification of hanging string defects. Therefore, it is necessary to conduct a large amount of analysis on the internal characteristics of hanging strings, and then design and build corresponding The model is used to meet the defect identification requirements of the hanging string, so as to ensure the safe operation of the high-speed railway catenary.

This application is based on the FasterRCNN model and has been improved in three aspects to realize the defect detection of the suspension string under the occluded condition, and to improve the recall rate and accuracy of the detection of the suspension string defect, thus providing strong support for the safe and intelligent operation of the high-speed rail catenary. .

The existing FasterRCNN model architecture mainly includes an input layer, a backbone network (also known as a backbone network), a neck, a Head detection head, and an output layer. The backbone network is divided into four stages, and each stage contains a certain number of convolution modules.

The improved model of this application is shown in Figure 5, and it has mainly been improved in three aspects: 1) Based on the existing image enhancement method and the inherent characteristics of the hanging wire, a Limited cutout algorithm (namely L-cutout) is proposed to solve the problem of hanging string training. The problem of small sample size, 2) Aiming at the limitations of variable convolution on the identification of hanging string defects and the huge difference in the length-to-width ratio of hanging strings, a constraint-based variable convolution (C-DCV) is proposed to improve the identification of hanging strings performance; 3) Aiming at the dependence of image recognition on CNN's local perception and ViT's long range, a novel backbone network is constructed for hanging string defect detection. Inspired by the residual network, this application replaces the 3x3 convolution with C-DCV to construct a new type of residual network, which we call CNN block (abbreviated as CB in this application). This application also optimizes the efficiency of HMSA, and combines it with FFN to construct an efficient self-attention module, called transformer block (abbreviated as TB in this application).

Further, in one embodiment, the collecting the catenary suspension string image and performing image enhancement processing on the suspension string image to obtain the suspension string defect sample set specifically includes:

Collect the catenary hanging string image during the operation of the high-speed railway;

Based on the improved image enhancement algorithm, image enhancement processing is performed on the catenary hanging string image, N mask areas are randomly generated in any hanging string image, and the mask area that blocks the key features of the hanging string is filtered according to the number N of mask areas. Obtain a sample set of hanging string defects;

The hanging string defect sample set is divided into training samples and validation samples.

Wherein, according to the number N of mask regions, filter the mask region that blocks the key features of the hanging string, specifically including:

If N=1, filter out the single mask area that completely blocks the hanging string in the hanging string image;

If N=2, then filter out the two mask regions that block the upper and lower ends of the hanging string in the hanging string image;

If N≥3, filter out a single mask area that completely blocks the hanging string in the hanging string image, and filter out any two mask areas that block the upper and lower ends of the hanging string in the hanging string image.

In the specific practice process, the application first performs image enhancement on the collected hanging string image. Image enhancement is beneficial to recognition scenarios such as image classification, object detection, and semantic segmentation. Usually image enhancement adopts traditional image enhancement algorithms, such as flipping (Flip, Rotation, Scale, etc.). In recent years, new image enhancement algorithms such as cutout, mixup, and cutmix have emerged, which can further improve image recognition capabilities. However, if these new algorithms are directly applied to the detection of hanging string defects, the detection recall rate and precision rate will be seriously reduced. The reason is mainly because these algorithms use random masking, image overlay and other operations to cover up the key features of the dropper, that is, to cover up the dropper clamps at both ends, leaving the dropper wire, Causes the model to recognize the line as a hanging string during the subsequent recognition process.

The enhanced recognition results on the suspended string samples of the existing image enhancement technology are shown in Figure 6, wherein, the a4 part of the figure in Figure 6 represents the standard suspended string sample; the b4 and c4 part of the figure shows the image that retains the key suspended string information Enhancement results; Figures in sections d4, e4, and f4 show the results of image enhancements that occlude key hanging string information.

In view of the problems existing in the above-mentioned image enhancement algorithm, the application proposes an improved algorithm based on the cutout method, called the Limited cutout (L-cutout) algorithm, which randomly generates mask areas, but the mask areas (also called mask blocks) The distinctive feature of the suspension string cannot be covered, that is, at least one suspension string clip is left unobstructed (as shown in the b4 and c4 part of Figure 6). L-cutout has the following advantages in the identification of hanging string defects: 1) It improves the recognition ability of blocked hanging strings; 2) It increases the samples of hanging string defects, which is conducive to the sufficient training of the model.

The general idea of the L-cutout algorithm in this application is: randomly generate mask areas in the hanging string image, but need to filter the mask areas that block the key features of the hanging string (d4, e4 and f4 in Figure 6). The specific processing of the L-cutout algorithm is as follows:

First, let the position and size of the randomly generated mask area in the image be represented by (X _mask, Y _mask , W _mask , H _mask ), where X _mask and Y _mask represent the coordinates of the starting point, W _{mask and} H _mask represent the mask in the image in width and height. The marked hanging string sample Ground Truth is represented by (X _GT , Y _GT , W _GT , H _GT ) in the image, where X _GT , Y _GT represent the starting coordinates, W _GT , H _GT represent the height and width. The physical hanging string is perspectived into the image, L _dw indicates the length of the hanging string in the picture, and L _dc indicates the length of the hanging string clamp.

Secondly, the algorithm filters the mask area, and the filtering logic of the algorithm is as follows:

Among them, the specific implementation process of the filtering logic of the algorithm is as follows: According to the processing logic of the second line, randomly generate N masks blocks in the input sample; when the number of mask blocks N is 1, filter out the hanging string image that completely blocks the hanging string For a single mask block, obtain the hanging string samples according to the processing logic in lines 3 to 17; when the number of mask blocks N is 2, filter out the two mask blocks that block the upper and lower ends of the hanging string in the hanging string image, according to line 18 The processing logic to line 29 obtains the hanging string samples; when the number of mask blocks N≥3, each mask block enters the processing logic branch of N=1 for filtering, and at the same time, any two mask blocks enter the processing logic of N=2 branch to filter.

Further, in one embodiment, the convolution module is constructed using a constraint-based variable convolution network, and the self-attention module is constructed according to an improved high-efficiency multi-head self-attention mechanism, and the optimal module allocation ratio is used to divide the The above convolution module is deeply fused with the self-attention module to generate a multi-block cross hybrid network, including:

According to the aspect ratio of the height-width ratio of the hanging string, the constraint relationship is used to constrain the height-width coordinate ratio of the sampling point position in the variable convolutional network, and at the same time limit the height-width coordinates of the sampling point position to not exceed the height-width of the input feature map , obtain a constraint-based variable convolution network, and construct a convolution module through a constraint-based variable convolution network;

Space dimensionality reduction operations are performed on the key vector K and value vector V in the original Transformer self-attention mechanism to obtain an improved high-efficiency multi-head self-attention mechanism and build a self-attention module based on the improved high-efficiency multi-head self-attention mechanism;

Based on the backbone network architecture in the FasterRCNN model, the number of convolution modules and the number of self-attention modules in the backbone network architecture are allocated according to the optimal module allocation ratio, and the convolution modules and self-attention modules are fused in a new paradigm to generate multiple blocks cross-hybrid network.

Wherein, the optimal module allocation ratio is specifically: the number of convolution modules (CB): the number of self-attention modules (TB) = 7:2, and this ratio is the optimal allocation ratio of this application. Further, according to the optimal module allocation ratio, the number of convolution modules and the number of self-attention modules in the backbone network architecture are allocated specifically: according to the number of modules in each stage in the backbone network, the modules in the stage are divided into 7 CB blocks and 2 CB blocks. The TB block is divided into multiple groups of modules in a group arrangement, and the multiple groups of modules are connected in series and merged to form a stage in the backbone network. This application mainly allocates and integrates the number of modules of Stage2, Stage3 and Stage4 in the backbone network. In some other embodiments of the present application, the proportion of module allocation may also be adjusted according to the architecture of the network in the model, and the present application will not repeat them here.

In the specific practice process, the deformable convolution can be closer to the shape and size of the object when sampling, while the sampling of the convolutional neural network cannot follow the change of the shape of the object, which is limited by the fixed geometry of the CNN module. The neural network is more friendly to objects with equal height and width, and the deformable convolutional neural network is more beneficial to the recognition of objects with diverse shapes. Since the width and height ratio of the suspension string is not equal, usually the height is much larger than the width. The string identification is unfavorable, so a variable convolutional neural network is selected to identify the defects of the hanging string.

Referring to Figure 7, this application introduces three convolution sampling images based on the hanging string image, where Dropper represents the hanging string sample, SC represents the standard convolution, DCV2 represents the deformable convolution, and C-DCV represents the constraint-based deformable volume Part a5 of Figure 7 shows the actual hanging string image, part b5 shows the sample image of standard convolution, part c5 shows the sample image of deformable convolution, and part d5 shows Sampling graphs for constraint-based deformable convolutions. It can be seen that the C-DCV can be changed in the sampling process according to the aspect ratio of the suspension string.

However, when DCV1 and DCV2 are directly applied to the detection of hanging wire defects, the overall performance of the model is slightly improved, but the ability of variable convolution cannot be fully utilized. The main reason is that although the variable convolution performs acquisition and recognition based on the shape of the object, it does not specifically target the type of hanging string, that is, the specific feature that the hanging string is much higher than the width.

In-depth research on the applicable objects of variable convolution, combined with the analysis of the inherent characteristics of the hanging string, this application proposes a convolution network for the inherent characteristics of the special case of the hanging string, which is called Constraint-based deformable ConvNets (Constraint-based deformable ConvNets, C- DCV), not only can be applied to the detection of hanging string defects, but also can be applied to the recognition of other objects that are taller than wide.

Specifically, the constraint-based deformable convolution algorithm (referred to as the C-DCV algorithm in this application) is shown in the following formula (1):

Among them, H and W represent the height and width of the input feature map, p _k represents the original sampling point in the input feature map, h _pk and w _pk represent the height and width coordinates of p _k ; Δw _k and Δh _k represent the offset corresponding to p _k quantity; x represents the feature of p _k position; α represents the scalable parameter, the purpose is to ensure the proportional relationship between Δw _k and Δh _k is limited. p+p _k +offset _k (Δw _k , Δh _k ) is the coordinate of the sampling point of p _k , because DCV2 generates Δw _k and Δh _k without limitation through the convolution operation, resulting in exceeding the boundary. Therefore, the C-DCV of this application adopts a constraint relationship to constrain the proportional relationship between h _pk and w _pk , making it close to the height-width ratio of the hanging string, while restricting h _pk and w _pk to not exceed the range of the input feature map.

At the same time, offset _k (Δw _k , Δh _k ) generates Δw _k and Δh _k through convolution, which may be a decimal, so the coordinates of p+p _k +offset _k (Δw _k , Δh _k ) are not integers, so it needs to pass the double line Interpolation obtains the integer coordinate position in the image, and then obtains the corresponding feature value according to the integer coordinate, and decomposes the decimal coordinate into four adjacent integer coordinate points to calculate the result. The specific operation process is as follows:

The principle of bilinear interpolation is shown in Figure 8. The coordinates P _k (X, Y) obtained through training are not integers, and the four adjacent coordinate points corresponding to P _k in the figure are Q11(x ₁ ,y ₁ ), Q12 (x ₁ ,y ₂ ), Q21(x ₂ ,y ₁ ), Q22(x ₂ ,y ₂ ), their corresponding eigenvalues are f(Q ₁₁ ), f(Q ₁₂ ), f(Q ₂₁ ) , f(Q ₂₂ ). We will calculate P values by bilinear interpolation method. Wherein, the principle of bilinear interpolation can be implemented with reference to the prior art (http://www.cnblogs.com/yssongest/p/5303151.html).

First perform linear interpolation in the x direction to obtain formula (2):

Then perform linear interpolation in the y direction to obtain formula (3):

Taken together, it is the final result of bilinear interpolation, that is, formula (4):

Since the image bilinear interpolation only uses 4 adjacent points, the denominator of the above formula 4 is 1. Here, the calculation process of P _k (X, Y) is shown in the following formula (5):

Further, in one embodiment, the multi-block cross hybrid network FasterRCNN network is used for improvement, and the improved FasterRCNN network is trained and verified based on the hanging string defect sample set, and the trained FasterRCNN improvement is obtained. models, including:

The backbone network architecture of the FasterRCNN network is replaced by the network architecture of the multi-block cross hybrid network to obtain the improved FasterRCNN model;

The improved Faster RCNN model is trained based on the training samples in the hanging string defect sample set. After the training is completed, the model is verified by the verification samples, and the improved Faster RCNN model that has been trained and verified is obtained.

In the specific practice process, the identification of hanging string defects not only requires high recall rate and high precision rate, but also requires real-time image reasoning of not less than 12fps. Although the real-time performance of many existing algorithms can meet the requirements of suspension string defect identification, they are not enough in terms of recall rate and precision rate, which seriously affects the safe operation of high-speed rail. To solve this problem, this application proposes a new model combining CNN and transformer to solve the problem of hanging string defect recognition. Firstly, a high-performance and high-efficiency multi-head attention is proposed to improve the computational efficiency and less weight parameters of the transformer, and then a new paradigm of CNN and transformer fusion is constructed.

The specific process is as follows:

1. High-performance and high-efficiency multi-head self-attention mechanism (HE-MHSA):

The self-attention mechanism is an attention mechanism based on scaling dot products, which projects the input vector of the original sequence into three different spaces as query, key, and value, that is, the corresponding query vector Q, key vector K, and value vector V, the input in each sequence performs attention calculations on the entire sequence, including itself.

The self-attention of the original transformer, as the resolution of the input image increases, the computational cost and memory consumption increase geometrically. Many works reduce the computational cost of SA by reducing the spatial resolution of the input K and V vectors. However, these operations lose important feature information and even introduce new noise, which leads to the attenuation of the subsequent MHSA characterization ability.

In response to the above problems, we propose HE-MHSA, which not only reduces the calculation and memory overhead, but also does not affect the representation ability of the model. The specific implementation process is as follows: First, we construct a new type of spatial dimensionality reduction operation, as shown in formula (6). The purpose is to reduce the dimensions of the vectors K and V, and the spatial dimensionality reduction operation proposed in this application neither loses feature information nor introduces new noise.

Among them, SR(.), AVG(.), and DW(.) respectively represent the usual space dimensionality reduction operation, average pooling operation, and variable convolution operation. These three types of operations perform dimensionality reduction operations on the key vector K, and obtain vectors K _sr , K _avg , and K _dw after normal space dimensionality reduction operations, average pooling operations, and variable convolution operations, respectively. Perform the above three types of dimensionality reduction operations on the vector V to obtain the vectors V _sr , V _avg , and V _dw after the usual space dimensionality reduction operation, average pooling operation, and variable convolution operation. Then add the dimension-reduced vectors K _sr , K _avg , K _dw to obtain a low-resolution vector K*, that is, K _sr +K _avg +K _dw →K*, and at the same time reduce the dimensionality-reduced vectors V _sr , V _avg and V _dw are added together to obtain a low-resolution vector V*, that is, V _sr +V _avg +V _dw →V*.

Second, z, V* and K* are used as input and applied to MHSA to obtain feature Z, as shown in formula (7):

Among them, z represents the feature image, which is used as the query input of the transformer block, and the low-resolution vectors V* and K* represent the key and value input, respectively, and the output feature Z is obtained through the processing of the transformer block.

2. The new paradigm of CNN and ViT fusion

The backbone network of FasterRCNN consists of four stages, as shown in the figure a6 in Figure 9, and each stage consists of multiple overlapping CNN blocks. The traditional fusion network of CNN and ViT is shown in the drawings of b6, c6, d6, e6 and f6 in Figure 9, and the network shown in the drawings of b6, c6 and d6 in Figure 9 is the backbone network of FasterRCNN ( backbone) The last one or several stages are replaced by multiple overlapping transformer blocks to build a new detector. Although these methods have greatly improved the image recognition effect, they increase the computational complexity and weight parameters, resulting in a large reasoning delay, which cannot meet the real-time recognition requirements on high-speed trains. The network shown in part e6 and f6 in Figure 9 has made a trade-off trade-off in terms of latency and accuracy, but it cannot meet the high recall rate and high precision requirements of hanging string recognition.

Referring to the attached drawing of part g6 in Figure 9, based on the above problems, this application proposes a new high-precision and high-efficiency new fusion paradigm of CNN and ViT, which is called multi-block cross-fusion hybrid network, that is, multi-block Cross Convergence Hybrid Network (MCHN), the network uses (CBxN _C +TBxN _T )xL to replace the (CB xN+TB x1)xL of the existing next-ViT.

Among them, the main difference between the MHCN of this application and next-ViT is reflected in the following three points:

1) The number of transformer blocks in each stage of this application is variable, not fixed at 1, so that it can further capture global features and long-range dependencies;

2) The constrained variable convolution adopted by the CNN block of MHCN in this application replaces the standard variable convolution;

3) The transformer block of this application uses a high-performance and high-efficiency multi-head self-attention mechanism to replace the MHSA of the original transformer.

Based on the above improvements, the MHCN network model constructed in this application has two advantages: 1) It can not only meet the high precision and high recall rate, but also meet the real-time reasoning requirements at high speed. Furthermore, according to the requirements of hanging string detection, this method is more reasonable and makes a better compromise in terms of accuracy and efficiency to meet the requirements of actual scenarios. 2) The MHCN network is compatible with all traditional modes, and the traditional mode is a special case of the network structure shown in part g6 of Fig. 9 . For example, when NT=1, the MHCN network evolves into the network shown in part e6 in FIG. 9 . Moreover, in the MHCN of this application, in stage2 to stage4, the distribution ratio of CNN block and transformer block is more flexible, and the relevant module distribution ratio can be configured according to actual scene requirements.

This application studies the reasonable distribution ratio of CNN block and transformer block by fixing the total number of modules in stage1, stage2 and stage4 in Figure 9, so as to further optimize the performance of the model. For a fairer comparison, all models use approximately equal number of blocks in the same stage. Through a large number of experiments, MHCN has more advantages than next-ViT in terms of hanging string recognition and inference speed. In the MHCN model, the optimal configuration ratio of the CNN block to the transformer block is 7:2, the model performance is better and the delay is smaller. Among them, the network structure formed by the cross fusion of CB blocks and TB blocks according to the optimal allocation ratio is shown in Figure 10. The modules are arranged in a group of 7 CB blocks and 2 TB blocks, and multiple groups of modules They are sequentially connected in series to form a stage.

Specifically, as shown in FIG. 11 , the specific structure of the CB block provided by the present application includes a 3×3 C-DCV layer, a Batch Norm layer, and a Dy-Relu layer connected in sequence. Referring to FIG. 12 , the TB block provided by the present application includes an HE-MHSA layer and an FFN layer connected in sequence.

In order to further verify the method proposed in this application, the application verifies the performance of the method by setting up a corresponding experimental environment. The specific process is as follows:

(1) Experimental environment

About 14,300 pieces of sample data have been accumulated, including 7 types of samples, including normal hanging strings, broken hanging strings, dropped hanging strings, bent hanging strings, loose strands of hanging strings, and irregular installation of hanging strings. The number of samples in each category is shown in Table 1:

Table 1 Number of hanging string samples

Among them, Trainset occupies 80%, and Testset occupies 20%. The model loss algorithm uses the focal loss method to solve difficult and unbalanced samples.

Due to the unbalanced samples, the model is more inclined to the type with a large number of samples. Therefore, according to the different number of samples of each type, different new strategies are adopted: 20% samples are added for the type with a large number of samples, and the L-cutout algorithm is used for the enhancement method. It is to improve occlusion recognition; if the sample size is small, the traditional method (e.g., roration, filp, etc.) is combined with the L-cutout method. The traditional method uses a 1:1 ratio enhancement, and the L-cutout uses a 20% enhancement.

(2) Image enhancement experiments (cutout, mixup, cutmix, and L-cutout) are compared;

Experimental environment: We use hanging strings as the training set and verification set, and fasterRCNN as our experimental network. Replace the L-cutout enhancement algorithm with cutout, mixup, and cutmix respectively, and train the samples enhanced by cutout, mixup, and cutmix in the same model. The evaluation results are shown in Table 2:

Table 2, the effect comparison table of the enhanced algorithm

It is easy to see from Table 2 that different data enhancement methods have different effects. L-cutout is significantly better than cutout, mixup, cutmix. The main reason is that cutout, mixup and cutmix use random enhancement to block or cover the key features of the suspension string, resulting in a decrease in recognition accuracy, while the L-cutout algorithm avoids the loss of key suspension string key information when the data is enhanced.

(3) Comparison of DCV2 and C-DCV

We compared the effect of DCV2 and C-DCV on the identification of suspension string defects. Replace the 3x3 convolutional neural network in the backbone network stage2, stage3, and stage4 with DCV2 and C-DCV, respectively. The latter two networks will be replaced for training, and their respective recall and precision rates will be verified. The sampling effect comparison of DCV2 and C-DCV is shown in Table 3.

Table 3 Sampling effect comparison table of DCV2 and C-DCV

It can be seen from Table 3 that although C-DCV and DCV2 are equal in inference efficiency, C-DCV has higher precision and recall for hanging string recognition. The main reason is that C-DCV is more inclined to the inherent characteristics of the hanging string during the sampling process, that is, the height-to-width ratio constraint learned by offset, which improves the model's ability to identify the hanging string.

(4) Influence of the new paradigm on the model:

For a fair comparison, all models use an approximately equal number of blocks in the same stage. The number of blocks in the dedicated network stage1, stage2, and stage4 is fixed at 3, 4, and 3, respectively. We adjust the number of stage3 blocks and the CNN block and transformer The allocation ratio of blocks is used to compare MCHN with traditional hybrid networks. The experiment uses ImageNet-22K as the model pre-training pre-training, training 300epoach. Use dropper samples for fine-tuning training and evaluation.

Under the same test environment, MCHN is compared with the traditional hybrid network, where the blocks of stage1, stage2, and stage4 in the backbone network are equal.

Table 4 Comparison table of recognition effect between MCHN and traditional hybrid network

According to the experimental results presented in Table 4, compared with the traditional hybrid network, it is not difficult to see that monotonously replacing the cnn blocks of the entire stage3 with transformer blocks does not significantly improve the recognition effect, while increasing the reasoning delay. In the stage, the cross-fusion of CNN block and transformer block is used, which improves the performance and does not increase the reasoning delay significantly. However, the distribution ratio of CNN block and transformer block in stage3 is not optimal, and the fusion effect cannot be exerted. MCHN has optimized the allocation ratio of cnn block and transformer block. The ratio of 7:2 can take advantage of their fusion, the recognition effect is greatly improved, and the reasoning delay is small.

In this application, CNN and transformer are cross-deeply fused, and the generated multi-block cross-mixed network is also compatible with the existing fusion mode, and for the inherent characteristics of hanging strings, this application also proposes a constraint-based variable convolution based on And a restriction-based L-cutout data augmentation algorithm. A large number of experiments have proved that the multi-block cross-hybrid network constructed in this application can not only greatly improve the defect recall rate and precision rate of hanging string recognition in complex application scenarios, but also have a small recognition delay.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The scope of the claimed invention is defined by the appended claims and their equivalents.

Claims (6)

1. The utility model provides a catenary dropper defect detection method based on CNN and Transformer fusion, which is characterized by comprising the following steps:

collecting a hanger image of the overhead line system and carrying out image enhancement processing on the hanger image to obtain a hanger defect sample set;

constructing a convolution module by using a constraint-based variable convolution network, constructing a self-attention module according to an improved high-efficiency multi-head self-attention mechanism, and performing deep fusion on the convolution module and the self-attention module based on an optimal module allocation proportion to generate a multi-block cross-mixed network;

improving the FaterRCNN network by utilizing the multi-block cross-mixed network, and training and verifying the improved FaterRCNN network based on the dropper defect sample set to obtain a trained FaterRCNN improved model;

and deploying the FasterRCNN improved model to a dropper detection device, capturing a dropper image of the overhead line system of the high-speed railway in real time, inputting the dropper image into the dropper detection device for dropper defect detection, and identifying the dropper defect in the overhead line system.

2. The method for detecting the catenary dropper defects based on the fusion of the CNN and the Transformer according to claim 1, wherein the steps of collecting the catenary dropper images and performing image enhancement processing on the dropper images to obtain a dropper defect sample set comprise:

acquiring a catenary dropper image in the running process of a high-speed railway;

performing image enhancement processing on the contact network dropper images based on the improved image enhancement algorithm, randomly generating N mask areas in any dropper image, and filtering mask areas for shielding the key features of the droppers according to the number N of the mask areas to obtain a dropper defect sample set;

the set of dropper defect samples is divided into training samples and verification samples.

3. The method for detecting the defects of the hanging strings of the overhead line system based on the fusion of the CNN and the Transformer according to claim 2, wherein the mask area which shields the key features of the hanging strings is filtered according to the number N of the mask areas, and specifically comprises the following steps:

if N=1, filtering out a single mask area which completely shields the dropper in the dropper image;

if N=2, filtering two mask areas which cover the upper end and the lower end of the dropper in the dropper image;

if N is more than or equal to 3, filtering out a single mask area which completely shields the dropper in the dropper image, and filtering out any two mask areas which shield the upper end and the lower end of the dropper in the dropper image.

4. The method for detecting the catenary dropper defect based on the fusion of the CNN and the Transformer according to claim 1, wherein the constructing the convolution module by using the constraint-based variable convolution network, constructing the self-attention module according to the improved high-efficiency multi-head self-attention mechanism, and performing deep fusion on the convolution module and the self-attention module based on the optimal module allocation proportion to generate the multi-block cross-mixed network specifically comprises:

according to the height-width ratio of the height-width ratio hanging string, adopting a constraint relation to constrain the height-width coordinate ratio of the sampling point position in the variable convolution network, simultaneously limiting the height-width coordinate of the sampling point position not to exceed the height-width of the input feature map, obtaining a constraint-based variable convolution network, and constructing a convolution module through the constraint-based variable convolution network;

performing space dimension reduction operation on a key vector K and a value vector V in an original transducer self-attention mechanism respectively to obtain an improved high-efficiency multi-head self-attention mechanism and constructing a self-attention module according to the improved high-efficiency multi-head self-attention mechanism;

based on a backbone network architecture in a FasterRCNN model, the number of convolution modules and the number of self-attention modules in the backbone network architecture are distributed according to an optimal module distribution proportion, and the convolution modules and the self-attention modules are subjected to new pattern fusion to generate a multi-block cross-mixed network.

5. The method for detecting the catenary dropper defect based on the fusion of the CNN and the Transformer according to claim 4, wherein the optimal module distribution ratio is specifically as follows: number of convolution modules: number of self-care modules = 7:2.

6. The method for detecting the dropper defects of the catenary based on the fusion of the CNN and the Transformer according to claim 1, wherein the method for improving the FasterRCNN network by using the multi-block cross-mixed network, and training and verifying the improved FasterRCNN network based on the dropper defect sample set, so as to obtain a trained FasterRCNN improvement model, comprises the following steps:

replacing the backbone network architecture of the FaterRCNN network with the network architecture of the multi-block cross-hybrid network to obtain an improved FaterRCNN model;

training the improved FaterRCNN model based on the hanger defect sample set training samples, and verifying the model by using a verification sample after training is completed to obtain a FaterRCNN improved model after training and verification.