WO2025050369A1 - Method for detecting object in image and related device - Google Patents

Abstract

A method for detecting an object in an image, a computer-readable storage medium, and an electronic device. The method comprises: acquiring an image to be detected (102); on the basis of said image, generating a plurality of feature maps for detecting objects having different sizes (104); fusing the plurality of feature maps to obtain a fused feature map (106); and on the basis of the fused feature map, detecting the objects (108). Objects of difference sizes can be detected, thereby improving the detection speed and saving computing resources.

Description

Method and related device for detecting objects in an image Technical Field

The present disclosure relates to the technical field of image processing, and more particularly, to a method for detecting an object in an image, a computer-readable storage medium, and an electronic device.

Background Art

In recent years, target detection technology, as one of the research tasks in the field of image processing technology, has been widely used in industrial production and intelligent manufacturing. Image processing technology uses a camera to obtain images of industrial products to be detected and transmits the images to an image processing system, which detects the characteristics of the target based on the distribution, brightness, color and other information of the pixels in the image, such as defects in industrial products such as scratches, holes and dents.

Summary of the invention

Embodiments of the present disclosure provide a method, a computer-readable storage medium, and an electronic device for detecting an object in an image.

In a first aspect of the present disclosure, a method for detecting an object in an image is provided. The method comprises: acquiring an image to be detected; based on the image, generating a plurality of feature maps for detecting objects of different sizes; fusing the plurality of feature maps to obtain a fused feature map; and detecting the object based on the fused feature map.

In an embodiment of the present disclosure, the multiple feature maps include: a first feature map for detecting an object having a first size, a second feature map for detecting an object having a second size, and a third feature map for detecting an object having a third size, wherein the first size is larger than the second size, and the second size is larger than the third size.

In an embodiment of the present disclosure, generating the first feature map based on the image includes: downsampling the image to obtain first image data; and performing a first convolution process on the first image data to generate the first feature map.

In an embodiment of the present disclosure, the downsampling ratio is 6 times, and in the first convolution process, the size of the convolution kernel is 3 and the step size is 2.

In an embodiment of the present disclosure, generating the second feature map based on the image includes: performing a second convolution process on the image to obtain second image data; and performing a third convolution process on the second image data to generate the second feature map.

In an embodiment of the present disclosure, in the second convolution processing, the size of the convolution kernel is 3 and the step size is 3, and wherein, in the third convolution processing, the size of the convolution kernel is 4 and the step size is 4.

In an embodiment of the present disclosure, generating the third feature map based on the image includes: performing a fourth convolution process on the image to obtain third image data; performing a fifth convolution process on the third image data to obtain fourth image data; performing a first pooling process and a second pooling process on the fourth image data, respectively, to obtain fifth image data and sixth image data; splicing the fifth image data and the sixth image data to obtain spliced image data; and performing a sixth convolution process on the spliced image data to generate the third feature map.

In an embodiment of the present disclosure, the first pooling process includes a first maximum pooling process, and wherein the second pooling process includes a second maximum pooling process.

In an embodiment of the present disclosure, in the fourth convolution processing, the size of the convolution kernel is 3 and the step size is 3, in the fifth convolution processing, the size of the convolution kernel is 3 and the step size is 1, and wherein, in the first pooling processing, the size of the pooling kernel is 3 and the step size is 2, in the second pooling processing, the size of the pooling kernel is 5 and the step size is 2, and wherein, in the sixth convolution processing, the size of the convolution kernel is 3 and the step size is 3.

In an embodiment of the present disclosure, a trained neural network is used to detect the object based on the fused feature map.

In an embodiment of the present disclosure, the trained neural network is a Faster-RCNN network, wherein the Faster RCNN network includes a backbone network, a region proposal network, and a regression classification network.

In an embodiment of the present disclosure, the SimOTA algorithm is used to train the region proposal network.

In an embodiment of the present disclosure, the SimOTA algorithm is used to train the region proposal network, including: taking feature points falling in the real box or near the real box as candidate positive samples; calculating the cost matrix of the predicted box of the candidate positive sample and the real box; for each real box, ranking the intersection and union (IoU) values of the real box and the predicted box from large to small, and selecting the first m predicted boxes; summing the IoU values corresponding to the first m predicted boxes, and the sum is n; and ranking the cost matrix from small to large, selecting the first n predicted boxes as positive samples, and the others as negative samples.

In an embodiment of the present disclosure, when using the SimOTA algorithm, if the size of the real frame is smaller than a predetermined threshold, the size of the real frame is adjusted to the predetermined threshold.

In an embodiment of the present disclosure, if there are M real boxes and N predicted boxes, the size of the cost matrix is M×N, and each element in the cost matrix is the value of the loss function of the real box and the predicted box.

In an embodiment of the present disclosure, the loss function includes a cross entropy loss for classification and an IOU loss for regression.

In an embodiment of the present disclosure, the cross entropy loss for classification is expressed as:

Where N _cls is the number of selected prediction boxes, _pi is the probability that the prediction box is the true box, _pi ^* = 0 is a positive sample, _pi ^* = 0 is a negative sample, and L _cls ( _pi , _pi ^* ) is expressed as:

In an embodiment of the present disclosure, the IOU loss for regression is expressed as:

Where _ti is the offset predicted by the predicted box, _ti ^* is the offset of the predicted box relative to the true box, and _Lreg ( _ti , _ti ^* ) is expressed as:

Where R is the smooth _L1 function, which is expressed as:

According to a second aspect of the present disclosure, a device for detecting an object in an image is provided. The device comprises: an acquisition module configured to acquire an image to be detected; a feature map generation module configured to generate multiple feature maps for detecting objects of different sizes based on the image; a fusion module configured to fuse the multiple feature maps to obtain a fused feature map; and an object detection module configured to detect the object based on the fused feature map.

According to a third aspect of the present disclosure, a computer-readable storage medium is provided. Computer program instructions are stored on the computer-readable storage medium, wherein when the computer program instructions are executed by a processor, the processor is caused to perform the method according to the first aspect of the present disclosure.

According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method according to the first aspect of the present disclosure.

Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for illustrative purposes and are not intended to limit the scope of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are for illustrative purposes only of selected embodiments, not all possible implementations, and are not intended to limit the scope of the present application, wherein:

FIG1 shows a schematic flow chart of a method for detecting an object in an image according to an embodiment of the present disclosure;

FIG2 shows an exemplary STEM network according to an embodiment of the present disclosure;

FIG3 shows a network architecture in which the method shown in FIG1 is implemented according to an embodiment of the present disclosure, the network architecture including the stem network and the Faster-RCNN network shown in FIG2 ;

FIG4 shows an example of a result detected by using a method according to an embodiment of the present disclosure;

FIG5 shows an example of a result detected by using a method according to an embodiment of the present disclosure;

FIG6 is a block diagram showing a schematic structure of an electronic device according to an embodiment of the present disclosure; and

FIG. 7 shows an exemplary structural block diagram of a device for detecting an object in an image according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. The embodiments of the present disclosure will be described in detail below with reference to the drawings and in conjunction with the embodiments. It should be noted that the features in the embodiments of the present disclosure can be combined with each other without conflict.

Faster-RCNN can realize target (also called object in this article) detection. The method based on Faster-RCNN can obtain a bounding box bbox (Bounding-box), which can define the target or object. In the process of performing target detection on industrial products, the image of the product to be detected may contain extremely small objects (for example, objects of 2 to 3 pixels) and larger objects (for example, objects of 300 to 8000 pixels). If Faster-RCNN is used directly for target detection, a large video memory is required due to the high resolution of the image. Therefore, target detection may not be completed under existing hardware conditions. If Faster-RCNN is used for detection after downsampling the image, small objects may not be detected due to the loss of information about small objects. Usually, if these objects need to be detected at the same time, an image pyramid structure is used to extract multi-scale features, and the image is processed using a sliding window method. However, this method is more complex and time-consuming.

Traditional Faster-RCNN mainly processes natural scenes. In order to process complex natural scenes, each layer of the Faster-RCNN backbone network is relatively heavyweight. Since natural scenes are complex and the objects to be detected are not too small, the resolution of natural scene images is usually within 1024×1024. However, the images of industrial products to be detected have a higher resolution, usually around 8000×5000. Therefore, if the heavyweight backbone network is used directly to process the image, there will be a huge waste of resources and it will take a long time.

FIG1 shows a schematic flow chart of a method for detecting an object in an image according to an embodiment of the present disclosure. As shown in FIG1 , in box 102, an image to be detected is acquired. In an embodiment of the present disclosure, the image may be an image of an industrial product to be detected, and the object may refer to defects present in the industrial product, such as burns, punctures, creases, etc. The image may be acquired in real time by an image acquisition device, or may be pre-stored in a storage device, which is not limited.

Continuing to refer to FIG. 1, in block 104, based on the acquired image, a plurality of feature maps for detecting objects of different sizes are generated. In an embodiment of the present disclosure, the plurality of feature maps may include a first feature map, a second feature map, and a third feature map. The first feature map may be used to detect an object of a first size, the second feature map may be used to detect an object of a second size, and the third feature map may be used to detect an object of a third size, wherein the first size is greater than the second size, and the second size is greater than the third size. In this embodiment, an object of the first size may be a larger defective object in an industrial product, an object of the second size may be a medium defective object, and an object of the third size may be a smaller defective object. In one embodiment, for an image with an image resolution of 8000, the first size may be in the range of 1 to 32 pixels, the second size may be in the range of 32 to 256 pixels, and the third size may be in the range of more than 256 pixels. In another embodiment, for an image with an image resolution of 15000, the first size may be in the range of 1 to 64 pixels, the second size may be in the range of 64 to 512 pixels, and the third size may be in the range of more than 512 pixels.

It should be noted that the embodiment of the present disclosure divides the objects in the image into three sizes, and those skilled in the art can divide the objects into more or fewer sizes according to actual needs. In addition, the following examples are described by taking the acquired image as a grayscale image as an example, and it should be understood that the image can also be a color image.

FIG2 shows an exemplary stem network according to an embodiment of the present disclosure. The following will describe in detail how to generate the first, second, and third feature maps in conjunction with the three branches of the stem network of FIG2. It should be noted that the image data and feature maps mentioned below are represented in the form of tensor shapes [B, C, H, W], where B represents batch, C represents channel (for example, the channel of a grayscale image is 1, the channel of an RGB image is 3), H represents the height of the image, and W represents the width of the image.

As shown in Figure 2, the first branch (labeled as "branch 1") is used to generate a first feature map. In this branch, first, the acquired image is downsampled to obtain first image data. In one embodiment, the downsampling process can use a bilinear interpolation function, the downsampling ratio can be 6 times, and the tensor shape of the obtained first image data is [B, 1, H/6, W/6]. Then, the first convolution process is performed on the first image data [B, 1, H/6, W/6] to generate a first feature map. In one embodiment, the size of the convolution kernel used in the first convolution process is 3 and the step size is 2. The tensor shape of the generated first feature map is [B, 16, H/12, W/12]. In the processing of the first branch, a large amount of computing resources can be saved by downsampling. The generated first feature map retains the image data of large-sized objects, but loses the image data of small-sized objects, and is mainly used to detect large-sized objects.

As shown in Figure 2, the second branch (labeled as "branch 2") is used to generate a second feature map. In this branch, first, a second convolution process is performed on the acquired image to obtain second image data. In one embodiment, the size of the convolution kernel used in the second convolution process is 3, and the step size is 3. The tensor shape of the obtained second image data is [B, 8, H/3, W/3]. Then, a third convolution process is performed on the second image data [B, 8, H/3, W/3] to generate a second feature map. In one embodiment, the size of the convolution kernel used in the third convolution process is 4, and the step size is 4. The tensor shape of the generated second feature map is [B, 16, H/12, W/12]. The second feature map generated by the processing of the second branch retains the image data of medium-sized objects and is mainly used to detect medium-sized objects.

Continuing to refer to FIG. 2 , the third branch (labeled as “branch 3”) is used to generate a third feature map. In this branch, a fourth convolution process is performed on the acquired image to obtain third image data. In one embodiment, the size of the convolution kernel used in the fourth convolution process is 3, and the step size is 3. The tensor shape of the obtained third image data is [B, 8, H/2, W/2]. Next, a fifth convolution process is performed on the third image data [B, 8, H/2, W/2] to obtain fourth image data. In one embodiment, the size of the convolution kernel used in the fifth convolution process is 3, and the step size is 1. The tensor shape of the obtained fourth image data is [B, 16, H/2, W/2]. Then, a first pooling process is performed on the fourth image data [B, 16, H/2, W/2], for example, a first maximum pooling process, to obtain fifth image data. In one embodiment, the size of the pooling kernel used in the first pooling process is 3, and the step size is 2. The tensor shape of the obtained fifth image data is [B, 8, H/4, W/4]. A second pooling process, for example, a second maximum pooling process, is performed on the fourth image data [B, 16, H/2, W/2] to obtain the sixth image data. In one embodiment, the size of the pooling kernel used in the second pooling process is 5, and the step size is 2. The tensor shape of the obtained sixth image data is [B, 8, H/4, W/4]. Next, the fifth image data [B, 8, H/4, W/4] and the sixth image data [B, 8, H/4, W/4] are spliced to obtain the spliced image data [B, 32, H/4, W/4]. In one embodiment, a fully connected layer can be used to splice the fifth image data and the sixth image data. Finally, a sixth convolution process is performed on the spliced image data [B, 32, H/4, W/4] to generate a third feature map. In one embodiment, the size of the convolution kernel used in the sixth convolution process is 3, and the step size is 3. The tensor shape of the generated third feature map is [B, 32, H/12, W/12]. The third feature map generated by the processing of the third branch retains image data of large-sized objects and is mainly used for detecting large-sized objects.

It should be noted that, in the embodiments of the present disclosure, the above three branches may be executed in parallel, and the first pooling process and the second pooling process may be executed in parallel.

In an embodiment of the present disclosure, each of the above-mentioned convolution processes may sequentially use a two-dimensional convolution layer, a normalization layer (for example, a batch normalization layer or a group normalization layer), and an activation function.

Continuing to refer to FIG. 1 , in block 106 , the first, second, and third feature maps are fused. In an embodiment of the present disclosure, a fully connected layer may be used to fuse the first, second, and third feature maps to obtain a fused feature map having a tensor shape of [B, 64, H/12, W/12]. In one embodiment, the fusion process may fuse the image data on a channel basis.

In block 108, objects are detected based on the fused feature map. In an embodiment of the present disclosure, a trained neural network may be used to detect objects. In one embodiment, the trained neural network may be a Faster-RCNN network, which will be described in detail below.

It should be noted that the flowchart shown in FIG. 1 is only for example, and those skilled in the art will appreciate that various modifications may be made to the flowchart shown or the steps described therein.

FIG3 shows a network architecture in which the method shown in FIG1 is implemented according to an embodiment of the present disclosure. As shown in FIG3, the network architecture includes the stem network and the Faster-RCNN network shown in FIG2. In an embodiment of the present disclosure, the Faster-RCNN network may include a backbone network, a region proposal network, and a regression classification network. It should be noted that although FIG3 combines the stem network and the The Faster-RCNN network is shown separately, but it can be understood that the stem network can also be embedded between appropriate layers of the backbone network of the Faster-RCNN network.

In an embodiment of the present disclosure, the backbone network may be a residual network ResNet50. ResNet50 receives the fused feature map output by the stem network. The fused feature map is processed via the five layers of ResNet50. In an embodiment of the present disclosure, the tensor shape of the image data output by the 0th layer is [B, 256, H/24, W/24], the tensor shape of the image data output by the 1st layer is [B, 512, H/48, W/48], the tensor shape of the image data output by the 2nd layer is [B, 1024, H/96, W/96], the tensor shape of the image data output by the 3rd layer is [B, 2048, H/192, W/192], and the tensor shape of the image data output by the 4th layer is [B, 2048, H/384, W/384]. Then, the image data processed by the five layers is provided to the feature pyramid network FPN to generate unified image data.

In an embodiment of the present disclosure, the FPN processing process is as follows: the 4th layer features are processed by a layer of convolutional network, the convolutional network kernel size is 1, the number of output channels is 256, and the generated tensor shape is [B, 256, H/384, W/384]; the 3rd layer features are processed by a layer of convolutional network, the convolutional network kernel size is 1, the number of output channels is 256, and the generated tensor shape is [B, 256, H/192, W/192]; the 4th layer features are upsampled twice and added to the 3rd layer features as the third layer candidate features, and the third layer candidate features are processed by a layer of 3x3 convolutional network to become new third layer features; the 2nd layer features are processed by a layer of convolutional network, the convolutional network kernel size is 1, the number of output channels is 256, and the generated tensor shape is [B, 256, H/96, W/96]; the 3rd layer candidate features are upsampled twice and added to the 2nd layer features As the second-layer candidate features, the second-layer candidate features are processed by a 3x3 convolutional network to become the new second-layer features; the first-layer features are processed by a convolutional network with a kernel size of 1 and an output channel number of 256, and the generated tensor shape is [B, 256, H/48, W/48]; the second-layer candidate features are upsampled twice and added to the first-layer features as the first-layer candidate features, and the first-layer candidate features are processed by a 3x3 convolutional network to become the new first-layer features; the 0th-layer features are processed by a convolutional network with a kernel size of 1 and an output channel number of 256, and the generated tensor shape is [B, 256, H/24, W/24]; the first-layer candidate features are upsampled twice and added to the 0th-layer features as the 0th-layer candidate features, and the 0th-layer candidate features are processed by a 3x3 convolutional network to become the new 0th-layer features; finally, The new multi-layer feature tensor data is provided to the region proposal network (RPN) and regression classification network (RCNN) for subsequent processing.

In an embodiment of the present disclosure, multiple layers of feature tensor data are input into a region proposal network. In the region proposal network, first, based on each layer of feature tensor data, different multi-layer convolutional networks are used to process and generate a proposed feature tensor, which is then processed using two independent branches. In an embodiment of the present disclosure, the first branch uses convolution to generate a proposal box; the other branch uses a convolutional network for processing and uses sigmoid for activation to classify the corresponding proposal box as foreground or background. The corresponding foreground proposal box (also referred to as a preliminary bounding box in this application) will be used as the input of the next stage (that is, the RCNN stage).

Next, the preliminary bounding box and the multi-layer feature tensor data are provided to the regression classification network. In the regression classification network, based on the multi-layer feature tensor data and the preliminary bounding box, the region of interest pooling layer cuts the multi-layer feature tensor data based on the preliminary bounding box, and the cut features are the corresponding candidate target features. The candidate target features are then processed using two independent branches. In the embodiment of the present disclosure, the first branch is processed using a 4-layer convolutional network to ultimately generate the corresponding regression parameters from the preliminary bounding box to the final target; the other branch is processed using a two-layer fully connected network to generate the final classification result; the final bounding box is obtained through the cooperation of the classification branch and the regression branch, that is, the category of each object and the position of the final bounding box of the object are obtained.

In the embodiments of the present disclosure, non-maximum suppression (NMS) can also be used to process the preliminary bounding boxes output by the region proposal network, and the top 512 preliminary bounding boxes with the highest confidence levels are selected as candidate bounding boxes; then ROIAlignPool is used to clip the multi-layer feature tensors based on the candidate boxes to obtain clipped data, and a 4-layer 3x3 convolutional network and a layer of full connection are used to process the clipped data to obtain the final regression parameters; and two layers of full connection are used to obtain the final classification result.

It should be noted that in the field of object recognition technology, the location of an object is determined by drawing a bounding box around one or more objects. Classification refers to the process of assigning a label to the object (i.e., determining whether the object belongs to the background or a defect target such as a burn, puncture, or tear).

FIG4 and FIG5 show examples of the results detected by the method according to an embodiment of the present disclosure. The circular frame of FIG4 shows the detected small-sized objects, such as small defects in industrial products. The square frame of FIG5 shows the detected medium-sized objects, such as medium-sized defects in industrial products. It can be seen that the method of the present disclosure can accurately detect objects of various sizes in an image, especially small-sized objects.

In addition, before using the network architecture shown in FIG3 to detect objects, the region proposal network in Faster-RCNN needs to be trained. Traditionally, in the process of training the region proposal network, a binary classification label (e.g., 0 or 1) is assigned to each prediction box, where 0 represents a negative sample and 1 represents a positive sample. The full name of IoU is Intersection over Union, which is a concept used in object detection. In the traditional region proposal network, it is necessary to predefine prediction boxes of different sizes and proportions in advance, and to determine whether the corresponding prediction box is a positive sample or a negative sample during training by calculating the overlap rate of the prediction box and the "real box", that is, the ratio of their intersection and union. In an embodiment of the present disclosure, if the intersection of the prediction box and the real box (IoU) is greater than 0.7, it is called a positive sample. If the intersection of the prediction box and the real box IoU is less than 0.3, it is called a negative sample. Other prediction boxes are neither positive samples nor negative samples and are not used for final training. After the region proposal network is trained using this method, objects in industrial products are detected by Faster-RCNN. Since the size range of objects varies greatly, for example, from 2 pixels to 5000 pixels, the size difference is more than 2000 times, which is far greater than the size range of objects in natural scenes. In the implementation process, it is difficult to define appropriate prediction boxes so that each target has enough prediction boxes corresponding to it. In the end, the sensitivity of IoU to objects of different sizes varies greatly, especially the low sensitivity of IoU to small objects, which makes most prediction boxes become negative samples due to small position deviations, resulting in inaccurate label assignment. Therefore, the traditional Faster-RCNN neural network cannot meet the requirements for defect detection of industrial products.

To this end, in an embodiment of the present disclosure, the SimOTA algorithm can be used to train the region proposal network in Faster-RCNN. First, the feature points that fall in the true box or near the true box are taken as candidate positive samples. Next, the cost matrix of the predicted box and the true box of the candidate positive sample is calculated. If there are M true boxes and N predicted boxes, the size of the cost matrix is M×N, and each element in the cost matrix is the value of the loss function of the true box and the predicted box, where the loss function includes the cross entropy loss for classification and the IOU loss for regression. Then, for each true box, the IoU value is ranked from large to small, the first m predicted boxes are selected, and the IoU values corresponding to the m predicted boxes are summed (the sum value is n). Finally, the cost corresponding to each true box in the cost matrix is ranked from small to large, and the first n predicted boxes are selected as positive samples, and the others are used as negative samples. In this process, in order to better handle small targets, a small target threshold τ (typical value is 64) is set. If the scale of the true box is less than τ, its size is expanded to τ for the above processing.

In an embodiment of the present disclosure, the cross entropy loss for classification can be expressed as:

Where N _cls is the number of selected prediction boxes, _pi is the probability that the predicted box is the true box, _pi ^* = 0 is a positive sample, _pi ^* = 0 is a negative sample, and L _cls ( _pi , _pi ^* ) is the logarithmic loss of the two categories:

In the embodiments of the present disclosure, the IOU loss for regression can be expressed as:

Where _ti is the offset predicted by the predicted box, _ti ^* is the offset of the predicted box relative to the real box, and _Lreg ( _ti , _ti ^* ) is expressed as:

Among them, R is the smooth _L1 function, which is expressed as:

In an embodiment of the present disclosure, when the SimOTA algorithm is used to train a region proposal network, if the size of the true box is smaller than a predetermined threshold, the size of the true box is adjusted to the predetermined threshold.

Table 1 shows the comparison of detecting objects in an image using the method according to an embodiment of the present disclosure and using the sliding window method based on Faster-RCNN. Referring to Table 1, the detected objects include burns, punctures and creases in the product, and the larger the corresponding value, the better the method used.

Table 1

It can be seen from Table 1 that the time used by the method of the present invention is only about 1/100 of that of the conventional sliding window method. In addition, the method of the present invention also shows good performance.

It can be seen from the above description that by adopting the method according to an embodiment of the present disclosure, by generating multiple feature maps for detecting objects of different sizes, objects of various sizes in an image, especially small-sized objects, can be detected, thereby improving the detection speed and saving computing resources.

FIG6 shows a schematic block diagram of an electronic device 60 for detecting an object in an image according to an embodiment of the present disclosure. The electronic device 60 includes one or more processors 602 and a memory 604. The memory 604 is coupled to the processor 602 via a bus and an I/O interface 606, and stores instructions that can be executed by the processor 602. When the instructions are executed by the processor 602, the electronic device 60 can perform the steps of the method for detecting an object in an image in any of the above embodiments.

The memory 604 may include readable media in the form of volatile memory, such as random access memory (RAM) and/or cache memory, and may further include read-only memory (ROM).

Memory 604 may also include a program/utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 60 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 60, and/or may communicate with any device that enables the electronic device 60 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input/output (I/O) interface 606. Furthermore, the electronic device 60 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) via a network adapter. The network adapter may communicate with other modules of the electronic device 60 via a bus. It should be understood that, although not shown in the figure, other hardware and/or software modules may be used in conjunction with the electronic device 60, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

7 shows an exemplary structural block diagram of a device 700 for detecting an object in an image according to an embodiment of the present disclosure. The device 700 may include an acquisition module 710, a feature map generation module 720, a fusion module 730, and an object detection module 740.

In an embodiment of the present disclosure, the acquisition module 710 is configured to acquire an image to be inspected, which may be an image of an industrial product to be inspected, and the object may refer to defects existing in the industrial product, such as burns, punctures, creases, etc.

In an embodiment of the present disclosure, the feature map generation module 720 is configured to generate a plurality of feature maps for detecting objects of different sizes based on the acquired image. The plurality of feature maps may include a first feature map, a second feature map, and a third feature map. The first feature map may be used to detect an object of a first size, the second feature map may be used to detect an object of a second size, and the third feature map may be used to detect an object of a third size, wherein the first size is larger than the second size, and the second size is larger than the third size.

In an embodiment of the present disclosure, the fusion module 730 is configured to fuse multiple feature maps to obtain a fused feature map. The fusion module 730 will further explain in detail how to generate the first, second and third feature maps in conjunction with the three branches of the stem network introduced in FIG. 2. In the processing of the first branch shown in FIG. 2, the image is downsampled to obtain first image data; and the first convolution processing is performed on the first image data to generate the first feature map. In the processing of the second branch shown in FIG. 2, the image is subjected to a second convolution processing to obtain second image data; and the second image data is subjected to a third convolution processing to generate the second feature map. In the processing of the third branch shown in FIG. 2, the image is subjected to a fourth convolution processing to obtain third image data; the third image data is subjected to a fifth convolution processing to obtain fourth image data; the fourth image data is subjected to a first pooling processing and a second pooling processing respectively to obtain fifth image data and sixth image data; the fifth image data and the sixth image data are spliced to obtain spliced image data; and the spliced image data is subjected to a sixth convolution processing to generate the third feature map.

In an embodiment of the present disclosure, the object detection module 740 is configured to detect an object based on the fused feature map. In an embodiment of the present disclosure, a trained neural network may be used to detect an object.

Each unit of the device 700 can further implement the functions and methods described in FIGS. 1-3 , which will not be repeated here.

In an embodiment of the present disclosure, a computer-readable storage medium is also provided, on which computer program instructions are stored, and when the computer program instructions are executed by, for example, a processor, the steps of the method for detecting an object in an image described in any of the above embodiments can be implemented. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps of various exemplary embodiments of the present application described in the method for detecting an object in an image in this specification.

The program product for implementing the above method according to the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims (21)

A method for detecting an object in an image, comprising: Acquire an image to be detected; Based on the image, generating a plurality of feature maps for detecting objects having different sizes; fusing the multiple feature maps to obtain a fused feature map; and Based on the fused feature map, the object is detected. The method of claim 1, wherein the plurality of feature maps comprises: a first feature map for detecting an object having a first size, a second feature map for detecting an object having a second size, and a third feature map for detecting an object having a third size, wherein the first size is larger than the second size, and the second size is larger than the third size. The method according to claim 2, wherein generating the first feature map based on the image comprises: downsampling the image to obtain first image data; and A first convolution process is performed on the first image data to generate the first feature map. The method according to claim 3, wherein the downsampling ratio is 6 times, and in the first convolution process, the size of the convolution kernel is 3 and the step size is 2. The method according to claim 2, wherein generating the second feature map based on the image comprises: performing a second convolution process on the image to obtain second image data; and A third convolution process is performed on the second image data to generate the second feature map. The method according to claim 5, wherein, in the second convolution process, the size of the convolution kernel is 3 and the step size is 3, and wherein, in the third convolution process, the size of the convolution kernel is 4 and the step size is 4. The method according to claim 2, wherein generating the third feature map based on the image comprises: performing a fourth convolution process on the image to obtain third image data; performing a fifth convolution process on the third image data to obtain fourth image data; Performing a first pooling process and a second pooling process on the fourth image data respectively to obtain fifth image data and sixth image data; splicing the fifth image data and the sixth image data to obtain spliced image data; and A sixth convolution process is performed on the spliced image data to generate the third feature map. The method of claim 7, wherein the first pooling process comprises a first maximum pooling process, and wherein the second pooling process comprises a second maximum pooling process. According to the method according to claim 7 or 8, wherein, in the fourth convolution processing, the size of the convolution kernel is 3 and the step size is 3, in the fifth convolution processing, the size of the convolution kernel is 3 and the step size is 1, and wherein, in the first pooling processing, the size of the pooling kernel is 3 and the step size is 2, in the second pooling processing, the size of the pooling kernel is 5 and the step size is 2, and wherein, in the sixth convolution processing, the size of the convolution kernel is 3 and the step size is 3. The method according to any one of claims 1 to 9, wherein the object is detected using a trained neural network based on the fused feature map. The method according to claim 10, wherein the trained neural network is a Faster-RCNN network, wherein the Faster RCNN network includes a backbone network, a region proposal network, and a regression classification network. The method of claim 11, wherein the region proposal network is trained using a SimOTA algorithm. The method of claim 11, wherein using the SimOTA algorithm to train the region proposal network comprises: The feature points that fall within or near the true frame are taken as candidate positive samples; Calculate the cost matrix of the predicted box and the true box of the candidate positive sample; For each true box, rank the intersection over union (IoU) values of the true box and the predicted box from large to small, and select the top m predicted boxes; Sum the IoU values corresponding to the first m prediction boxes, where the sum is n; and The cost matrix is ranked from small to large, and the first n prediction boxes are selected as positive samples, and the others are selected as negative samples. The method according to claim 13, wherein, when using the SimOTA algorithm, if the size of the true frame is smaller than a predetermined threshold, the size of the true frame is adjusted to the predetermined threshold. The method according to claim 13 or 14, wherein, if there are M true boxes and N predicted boxes, the size of the cost matrix is M×N, and each element in the cost matrix is a value of a loss function between a true box and a predicted box. The method according to claim 15, wherein the loss function includes a cross entropy loss for classification and an IOU loss for regression. The method according to claim 16, wherein the cross entropy loss for classification is expressed as:
Where N _cls is the number of selected prediction boxes, _pi is the probability that the prediction box is the true box, _pi ^* = 0 is a positive sample, _pi ^* = 0 is a negative sample, and L _cls ( _pi , _pi ^* ) is expressed as:
The method of claim 16, wherein the IOU loss for regression is expressed as:
Where _ti is the offset predicted by the predicted box, _ti ^* is the offset of the predicted box relative to the true box, and _Lreg ( _ti , _ti ^* ) is expressed as:
Where R is the smooth _L1 function, which is expressed as:
A device for detecting an object in an image, comprising: An acquisition module is configured to acquire an image to be detected; a feature map generation module, configured to generate a plurality of feature maps for detecting objects with different sizes based on the image; a fusion module configured to fuse the multiple feature maps to obtain a fused feature map; and The object detection module is configured to detect the object based on the fused feature map. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 18. An electronic device, comprising: Processor; and a memory storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 18.