CN117830703A - Image recognition method based on multi-scale feature fusion, computer device and computer-readable storage medium - Google Patents

Abstract

The invention provides an image identification method based on multi-scale feature fusion, a computer device and a computer readable storage medium, wherein the method comprises the following steps: manufacturing a training sample set; constructing a bidirectional cascade neural network model, wherein the bidirectional cascade neural network model comprises a first identification neural network, the first identification neural network comprises a first encoder and a first decoder, the first decoder is used for acquiring a first current feature map and a second current feature map which are output by the first encoder, respectively convolving the first current feature map and the second current feature map to obtain a first current convolution feature map and a second current convolution feature map, cascading the first current convolution feature map, the second current convolution feature map and the up-sampled first feature map to form a first cascade map, and performing feature remapping on the first cascade map to obtain a first feature map; training the bidirectional cascade neural network model by using a training sample set; the image to be identified is input into the bidirectional cascade neural network model to obtain an identification result, and the identification performance is improved.

Description

Image recognition method based on multi-scale feature fusion, computer device and computer-readable storage medium

Technical Field

The present invention relates to the technical field of image processing, and in particular to an image recognition method based on multi-scale feature fusion, a computer device and a computer-readable storage medium.

Background technique

Nowadays, deep learning methods can identify a variety of objects when used for image processing. They have wide applicability and accuracy, and have become a new means to solve the challenges encountered in agricultural vision applications. However, due to the constraints of the natural growth laws of crops, the collection of agricultural image data is very expensive and time-consuming. An experimental field can often only obtain 1-2 sequences of images a year, so the training data available for the neural network model is very limited, which seriously affects the performance of the neural network model.

There is a Faster R-CNN neural network that outputs a single and small feature layer, which is prone to losing pixel information of small targets and cannot identify small target objects in the image, thus affecting the performance of the neural network.

Summary of the invention

The first object of the present invention is to provide an image recognition method based on multi-scale feature fusion to enhance the performance of a neural network.

The second objective of the present invention is to provide a computer device for implementing the above-mentioned image recognition method based on multi-scale feature fusion.

A third object of the present invention is to provide a computer-readable storage medium to which the above-mentioned image recognition method based on multi-scale feature fusion is applied.

In order to achieve the above-mentioned first purpose, the present invention provides an image recognition method based on multi-scale feature fusion, which includes preparing a training sample set; constructing a bidirectional cascade neural network model, wherein the bidirectional cascade neural network model includes a first recognition neural network, the first recognition neural network includes a first encoder and a first decoder, the first encoder is used to downsample the image to extract a multi-scale feature map; the first decoder is used to obtain a first current feature map output by the first encoder and a second current feature map of the previous level relative to the first current feature map, respectively convolving to obtain a first current convolution feature map and a second current convolution feature map, upsampling the previous feature map output by the decoder, cascading the first current convolution feature map, the second current convolution feature map and the upsampled first feature map to form a first cascade map, and remapping the first cascade map to obtain a first feature map; using the training sample set to train the bidirectional cascade neural network model; inputting the image to be recognized into the trained bidirectional cascade neural network model to obtain a recognition result.

It can be seen from the above scheme that when the first decoder extracts the feature map, the first encoder outputs the first current feature map and fuses it with the second current feature map of the previous level to retain a more accurate positioning signal, so that the neural network has a better ability to discriminate features for the target area. The first decoder also fuses the previous feature map to increase the spatial dimension between cascades and improve the recognition performance of small targets. In the first recognition neural network, the first current feature map and the second current feature map are convolved using SAConv respectively. SAConv can adaptively capture the relationship between different positions in the input feature map, adaptively adjust the weight of the convolution kernel according to the input feature map, and fuse and enhance the contextual information of the input feature map, so that the neural network learns more feature information and can maintain good performance when trained with a small number of training sets.

In a further scheme, the first decoder performs feature remapping on the first cascade graph, including: convolving the first cascade graph, splitting the convolved first cascade graph into a first split cascade graph and a second split cascade graph, performing deformable convolution on the first split cascade graph, outputting a first split convolution cascade graph, cascading and convolving the first split convolution cascade graph with the second split cascade graph to obtain a first feature graph.

It can be seen that feature remapping is implemented using a feature remapping module. The feature remapping module in the first decoder is combined with a deformable convolution module. The deformable convolution can introduce an offset to dynamically adjust the sampling position, thereby better adapting to the deformation of plant morphological characteristics.

In a further scheme, the bidirectional cascade neural network model also includes a second recognition neural network, and the second recognition neural network includes a second encoder and a second decoder. The second encoder is used to downsample the image and extract a multi-scale feature map; the second decoder is used to input the fourth target feature map output by the second encoder into the attention mechanism module, perform an upsampling operation and weighted fusion with the first target feature map to obtain a first fused feature map, upsample the first fused feature map, and weighted fusion with the second target feature map to form a second fused feature map, perform feature remapping on the second fused feature map to obtain a second feature map, obtain the second feature map, the first target feature map and the first fused feature map for weighted fusion to obtain a third fused feature map, and perform feature remapping on the third fused feature map through the second feature remapping module to obtain a third feature map.

It can be seen that in the second recognition neural network, the fourth target feature map is upsampled and weightedly fused with the first target feature map of the previous level until all feature maps of the second encoder are weightedly fused, and then feature remapping is performed to obtain the second feature map. The second feature map uses the first target feature map, the first fused feature map and the second feature map output by the second encoder as input, performs feature remapping, and obtains the third feature map. The decoding process of the second recognition neural network respects the diversity of gradients, integrates the feature maps of each stage in the network, and the decoding module adopts weight fusion technology. Through continuous iterative bottom-up and top-down processing, the second recognition neural network can maintain a high quality of feature information.

In a further solution, the image to be recognized is input into the trained bidirectional cascade neural network model, including: judging whether the processing capacity of the processor in the current unit time is greater than a preset value; if the processing capacity of the processor is greater than the preset value, the image to be recognized is input into the second recognition neural network for processing; if the processing capacity of the processor is less than the preset value, the second recognition neural network is input into the first recognition neural network for processing.

It can be seen that whether to use the first recognition neural network or the second recognition neural network is determined according to the current processing volume, thereby reducing the computational processing volume of the processor.

In a further scheme, before upsampling the previous feature map output by the decoder, it also includes: inputting the third current feature map output by the first encoder into the attention mechanism module to obtain the third current attention feature map, convolving the third current attention feature map, the first current feature map and the third current feature map respectively, cascading the convolved third current attention feature map, the sampled first current feature map and the sampled third current feature map to obtain a second cascade map, and performing feature remapping on the second cascade map to form the previous feature map.

It can be seen that the previous feature map is obtained by cascading and remapping the two feature maps and the current attention feature map output by the attention mechanism module, and the third current feature map is one level higher than the first current feature map. Fusion of an additional low-level feature in the first encoder retains a more accurate positioning signal. This connection method more effectively enhances our network's ability to learn the most discriminative features of the target area, and is also of great significance for the positioning of small targets.

In a further embodiment, the first recognition neural network also includes a first loss module, which processes the feature map output by the first decoder.

It can be seen that the first loss module includes classification loss and regression loss. Classification loss is used to penalize the difference between the predicted category and the true category, and regression loss is used to constrain the model's learning of the predicted box position and shape. The total loss of the first recognition neural network is obtained by combining the classification loss and regression loss. This total loss is continuously minimized through iteration, driving the model to continuously improve its classification and positioning accuracy, and ultimately achieving accurate detection and positioning of crops in agricultural production activities.

In a further embodiment, the second recognition neural network also includes a second loss module, which processes the feature map output by the second decoder.

It can be seen that the second loss module includes classification loss and regression loss. The classification loss is used to penalize the difference between the predicted category and the true category, and the regression loss is used to constrain the model's learning of the predicted box position and shape. The total loss of the second recognition neural network is obtained by combining the classification loss and regression loss. This total loss is continuously minimized through iteration, driving the model to continuously improve its classification and positioning accuracy, and ultimately achieving accurate detection and positioning of crops in agricultural production activities.

In order to achieve the above-mentioned second purpose, the computer device provided by the present invention is characterized in that the computer device includes a processor and a memory, the memory stores a computer program, and when the computer program is executed by the processor, it implements the image recognition method based on multi-scale feature fusion as mentioned above.

In order to achieve the third objective mentioned above, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the above-mentioned image recognition method based on multi-scale feature fusion is implemented.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an embodiment of an image recognition method based on multi-scale feature fusion according to the present invention.

FIG2 is a structural system block diagram of a bidirectional cascade neural network model of an embodiment of an image recognition method based on multi-scale feature fusion according to the present invention.

FIG3 is a schematic diagram of a feature remapping module in a first decoder of an embodiment of an image recognition method based on multi-scale feature fusion according to the present invention.

The present invention is further described below in conjunction with the accompanying drawings and embodiments.

Detailed ways

Embodiment of the image recognition method based on multi-scale feature fusion:

The image recognition method based on multi-scale feature fusion of the present invention utilizes the multi-layer feature map of the encoder and utilizes the information of the multi-layer feature map in the decoding process to enhance the positioning signal.

Refer to Figure 1, which is a flow chart of an embodiment of an image recognition method based on multi-scale feature fusion of the present invention. First, step S1 is executed to prepare a training sample set. The training sample set is obtained by photographing crops in a farmland, and the crops in the training picture are marked. In order to reduce the amount of calculation of the processor, the longest side of the input picture is set to 608 pixels, and the other side length is scaled proportionally. While ensuring the calculation efficiency, the original information of the image is retained as much as possible.

After making the training sample set, execute step S2 to construct a bidirectional cascade neural network model PlantBiCNet. Referring to Figure 2, the bidirectional cascade neural network model includes a first recognition neural network model and a second recognition neural network model. Among them, the first recognition neural network model includes a first encoder and a first decoder. The structure adopted by the first encoder is CSPDarknet, and the image is continuously downsampled by a convolutional layer with a step size of 2 to generate a multi-scale feature map. In this embodiment, three feature maps are generated by the first encoder, among which the second current feature map is a feature map of 1/8 of the original image, the first current feature map is a feature map of 1/16 of the original image, and the third current feature map is a feature map of 1/32 of the original image.

In the first decoder, the third current feature map output by the first encoder is input into the attention mechanism module, wherein the attention mechanism module is an Mlt-ECA (Multi-Efficient Channel Attention) module, which is used to respond to the weighted adjustment of the feature. After the third current feature map is input into the attention mechanism module, the third current attention feature map is obtained, and the third current attention feature map, the first current feature map and the third current feature map are convolved respectively. Among them, in the first decoder, the SAConv (Switchable Atrous Convolution) module is used for convolution. The SAConv module is mainly responsible for dimensional transformation and connection operations. Unlike ordinary convolution, the SAConv module can adaptively capture the relationship between different positions in the input feature map. Specifically, it introduces a switch mechanism to adaptively adjust the weight of the convolution kernel according to the input feature map, and there is a context module to fuse and enhance the context information of the input feature map.

The convolved third current attention feature map, the sampled first current feature map, and the sampled third current feature map are cascaded to obtain a second cascade map, and the second cascade map is feature remapped to form the previous feature map. Referring to Figure 3, in the first decoder, feature remap is performed through the first feature remap module (CSPDCN). In the first feature remap module, the bottleneck convolution layer (CSPBottleneck) and the deformable convolution (DCN (Deformable Convolutional Networks)) are combined to change the two convolution layers in CSPLayer into two 3×3 deformable convolution layers. Compared with the fixed geometric structure and finite geometric transformation in traditional convolution, deformable convolution introduces an offset to dynamically adjust the sampling position, thereby better adapting to the deformation of plant morphological characteristics.

After forming the previous feature map, the first current feature map output by the first encoder and the second current feature map of the previous level of the first current feature map are convolved using the SAConv module respectively to obtain the first current convolution feature map and the second current convolution feature map. The previous feature map output by the decoder is upsampled, and the first current convolution feature map, the second current convolution feature map and the upsampled first feature map are cascaded to form a first cascade map, and the first cascade map is feature remapped by the first feature remapping module to obtain the first feature map. In the feature remapping of the first cascade map, the first feature remapping module is used to perform feature remapping, which includes: convolving the first cascade map, splitting the first cascade map after convolution into a first split cascade map and a second split cascade map, performing double deformable convolution on the first split cascade map, outputting the first split convolution cascade map, cascading and convolving the first split convolution cascade map with the second split cascade map, and obtaining the first feature map. The first cascade map is divided into two parts, and they are merged through a cross-stage hierarchy to achieve richer gradient combinations. This design allows the gradient flow to propagate on different network paths, which helps improve the network's learning ability and reduce the amount of computation.

After the first decoder outputs the feature map, the feature map is subjected to the Dropout strategy. Using the Dropout strategy on the feature map means randomly applying Dropout at certain locations of the feature map.

Whether it is the previous feature map or the first feature map, the two current feature maps are used for fusion to retain more accurate positioning signals, so that the neural network has better recognition ability of discriminative features for the target area. The first feature map also fuses the previous feature map to increase the spatial dimension between cascades and improve the recognition performance of small target objects.

The first recognition neural network model also includes a first loss module, which processes the feature map output by the first decoder. The first loss module includes classification loss and regression loss. The classification loss is calculated based on the BCE (Binary Cross-Entropy) function to penalize the difference between the predicted category and the true category. The regression loss is used to constrain the model's learning of the predicted box position and morphology, guided by the SCYLLA-IoU (SIoU) and Distribution FocalLoss (DFL) functions. It can be regarded as the sum of the box matching degree loss calculated by SIoU and the distance matching loss calculated by DFL. SIoU guides the model to learn the matching degree of the bounding box by considering the scale and angle information of the bounding box, and DFL optimizes the bounding box position through smooth L1 loss. The total loss of the second recognition neural network is obtained by combining the classification loss and the regression loss. This total loss is continuously minimized through iteration, driving the model to continuously improve its classification and positioning accuracy, and ultimately achieving accurate detection and positioning of crops in agricultural production activities.

The first neural network model also includes an Anchor Free algorithm, which performs prediction on the feature map output by the first decoder. The Anchor Free algorithm generates a prediction box based on the information on the feature map.

The second recognition neural network includes a second encoder and a second decoder, and the second encoder is used to downsample the image to extract a multi-scale feature map. The structure adopted by the second encoder is CSPDarknet, which continuously downsamples the image through a convolution layer with a step size of 2 to generate a multi-scale feature map. In this embodiment, three feature maps are generated by the second encoder, wherein the second target feature map is a feature map of 1/8 of the original image, the first target feature map is a feature map of 1/16 of the original image, and the fourth target feature map is a feature map of 1/32 of the original image. The second decoder is different from the first decoder, and DWConv (DepthwiseConvolution) is used for convolution.

The second decoder first inputs the fourth target feature map into the attention mechanism module, performs feature weighting adjustment, and then uses DWConv for convolution to adjust the number of channels and dimensions. Then, it performs upsampling operation through bilinear interpolation to form a feature map equivalent to 1/16 of the original image, and then uses DWConv for convolution to adjust the number of channels and dimensions. Then, it uses the weight fusion module to perform weighted fusion with the first target feature map to obtain the first fused feature map, and then upsamples the first fused feature map to form a feature map equivalent to 1/8 of the original image, and performs weighted fusion with the second target feature map to form a second fused feature map. The second feature remapping module (CSPLayer) is used to perform feature remapping on the second fused feature map to obtain the second feature map.

The second decoder obtains the second feature map, the first target feature map and the first fused feature map for weighted fusion to obtain the third fused feature map, and uses the second feature remapping module (CSPLayer) to remap the third fused feature map to obtain the third feature map. Among them, the second feature map is convolved using DWConv and then weighted fused. The decoding process of the second decoder respects the diversity of gradients and integrates the feature maps of each stage in the network. At the same time, these decoding modules use weight fusion technology. Through continuous iterative bottom-up and top-down processing, the second recognition neural network can maintain a high quality of feature information. To be precise, the weight fusion module uses a learnable weight coefficient structure, which is mainly used to control the degree of influence of feature maps between different levels on target object detection.

The third feature map is fused with the second feature map, which shows obvious advantages in the refinement of feature representation and the diversity of feature fusion, thereby enhancing the model's ability to perceive and capture deep information.

After the second decoder outputs the feature map, the feature map uses the Dropout strategy. Using the Dropout strategy on the feature map means randomly applying Dropout to certain positions of the feature map. Specifically, Dropout can be applied independently on each channel of the feature map, or certain positions of the entire feature map can be set to 0 with a certain probability. This strategy can help the model better learn the spatial information in the feature map and reduce the risk of overfitting. By randomly discarding some features, the model is forced to learn a more robust feature representation instead of relying on certain specific features.

The second recognition neural network also includes a second loss module, which processes the feature map output by the second decoder. The second loss module includes classification loss and regression loss. The classification loss is calculated based on the BCE (Binary Cross-Entropy) function to penalize the difference between the predicted category and the true category. The regression loss is used to constrain the model's learning of the predicted box position and morphology, guided by the SCYLLA-IoU (SIoU) and Distribution Focal Loss (DFL) functions. It can be regarded as the sum of the box matching degree loss calculated by SIoU and the distance field matching loss calculated by DFL. SIoU guides the model to learn the matching degree of the bounding box by considering the scale and angle information of the bounding box, and DFL optimizes the bounding box position through smooth L1 loss. The total loss of the second recognition neural network is obtained by combining the classification loss and the regression loss. This total loss is continuously minimized through iteration, driving the model to continuously improve its classification and positioning accuracy, and ultimately achieving accurate detection and positioning of crops in agricultural production activities.

The second neural network model also includes an Anchor Free algorithm, which performs prediction on the feature map output by the second decoder. The Anchor Free algorithm generates a prediction box based on the information on the feature map.

Since DWConv is used in the second recognition neural network to reduce computational complexity, and the deformable convolution is integrated in the first feature remapping module, the computational processing volume is greater than that of the second feature remapping module.

After building the bidirectional cascade neural network model, execute step S3 to train the bidirectional cascade neural network model using the training sample set. During training, SGD (Stochastic Gradient Descent) was used as the optimizer, the momentum factor was set to 0.937, and the initial learning rate was 0.01. In addition, an early stopping strategy was set. If the loss value does not decrease for 50 consecutive iterations, the training is stopped to avoid overfitting.

After the bidirectional cascade neural network model is trained using the training sample set, step S4 is executed to input the image to be identified into the trained bidirectional cascade neural network model to obtain the recognition result. Wherein, when the image to be identified is input into the trained bidirectional cascade neural network model, it also includes: judging whether the processing capacity of the processor in the current unit time is greater than the preset value; if the processing capacity of the processor is greater than the preset value, the image to be identified is input into the second recognition neural network for processing; if the processing capacity of the processor is less than the preset value, the second recognition neural network is input into the first recognition neural network for processing. Determine whether to use the first recognition neural network or the second recognition neural network according to the current processing capacity to reduce the computational processing capacity of the processor.

By making full use of the feature map output by the encoder and the feature map output by the decoder, the positioning signal and the spatial dimension are enhanced, and the present invention improves the recognition performance of small target objects by better utilizing spatial information, and has higher efficiency. In the case of fewer training pictures, small target objects can also be recognized.

Computer device embodiment:

The computer device of this embodiment includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the above-mentioned image transformation method is implemented.

The computer device may include but is not limited to a processor and a memory. Those skilled in the art will appreciate that the computer device may include more or fewer components, or a combination of certain components, or different components, for example, the computer device may also include input and output devices, network access devices, buses, etc.

Computer readable storage medium embodiment:

The image recognition method based on multi-scale feature fusion in the computer device described in the above embodiment can be stored in a computer-readable storage medium in the form of a computer program. When the computer program is executed by the processor, the steps of the above-mentioned image recognition method based on multi-scale feature fusion in the computer device can be completed. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

The above are only preferred embodiments of the present invention, but the design concept of the invention is not limited thereto. Without departing from the concept of the present invention, more other equivalent embodiments may be included. It is possible for those skilled in the art to make various obvious changes, readjustments and substitutions without departing from the protection scope of the present invention.

Claims (9)

1. The image identification method based on multi-scale feature fusion is characterized by comprising the following steps of:

manufacturing a training sample set;

constructing a bidirectional cascade neural network model, wherein the bidirectional cascade neural network model comprises a first identification neural network, the first identification neural network comprises a first encoder and a first decoder, and the first encoder is used for downsampling an image to extract a multi-scale feature map;

the first decoder is configured to obtain a first current feature map output by the first encoder and a second current feature map corresponding to a previous stage of the first current feature map, respectively convolve the first current feature map and the second current feature map to obtain a first current convolution feature map and a second current convolution feature map, upsample the previous feature map output by the decoder, cascade the first current convolution feature map, the second current convolution feature map and the upsampled first feature map to form a first cascade map, and remap the features of the first cascade map to obtain a first feature map;

training the bidirectional cascade neural network model by using the training sample set;

and inputting the image to be identified into the trained bidirectional cascade neural network model to obtain an identification result.

2. The method for identifying images based on multi-scale feature fusion according to claim 1, wherein:

the first decoder performs feature remapping on the first cascade graph to obtain a first feature graph, where the first feature graph includes: the first cascade diagram is convolved, the convolved first cascade diagram is divided into a first split cascade diagram and a second split cascade diagram, deformable convolution is carried out on the first split cascade diagram, the first split convolution cascade diagram is output, and the first split convolution cascade diagram and the second split cascade diagram are cascaded and convolved to obtain the first feature diagram.

3. The method for identifying images based on multi-scale feature fusion according to claim 1 or 2, characterized in that:

the bidirectional cascade neural network model further comprises a second identification neural network, wherein the second identification neural network comprises a second encoder and a second decoder, and the second encoder is used for downsampling an image to extract a multi-scale feature map;

the second decoder is configured to input a fourth target feature map output by the second encoder into the attention mechanism module, perform weighted fusion with the first target feature map after performing an upsampling operation to obtain a first fused feature map, perform weighted fusion with the first fused feature map and a second target feature map to form a second fused feature map, perform feature remapping on the second fused feature map to obtain a second feature map, obtain the second feature map, perform weighted fusion between the first target feature map and the first fused feature map, obtain a third fused feature map, and perform feature remapping on the third fused feature map through the second feature remapping module to obtain the third feature map.

4. A method of identifying images based on multi-scale feature fusion according to claim 3, characterized in that:

inputting the image to be identified into a trained bidirectional cascade neural network model comprises the following steps:

judging whether the processing capacity of the processor in the current unit time is larger than a preset value or not;

if the processing amount of the processor is larger than the preset value, inputting the image to be identified into the second identification neural network for processing;

and if the processing amount of the processor is smaller than the preset value, inputting the second identification neural network into the first identification neural network for processing.

5. The method for identifying images based on multi-scale feature fusion according to claim 1 or 2, characterized in that:

before upsampling the last feature map of the output of the decoder, the method further comprises: and inputting a third current feature map output by the first encoder into an attention mechanism module to obtain a third current attention feature map, respectively convolving the third current attention feature map, the first current feature map and the third current feature map, cascading the convolved third current attention feature map, the sampled first current feature map and the sampled third current feature map to obtain a second cascading map, and performing feature remapping on the second cascading map to form the previous feature map.

6. The method for identifying images based on multi-scale feature fusion of claim 5, wherein:

the first recognition neural network further comprises a first loss module, and the first loss module processes the characteristic diagram output by the first decoder.

7. The method for identifying images based on multi-scale feature fusion of claim 4, wherein:

the second recognition neural network further comprises a second loss module, and the second loss module processes the characteristic diagram output by the second decoder.

8. Computer device, characterized in that it comprises a processor and a memory, the memory storing a computer program which, when executed by the processor, implements a method for recognition of images based on multi-scale feature fusion according to any of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program characterized by:

the computer program when executed implements a method of identifying images based on multi-scale feature fusion as claimed in any one of claims 1 to 7.