WO2021143865A1 - Positioning method and apparatus, electronic device, and computer readable storage medium - Google Patents

Abstract

The present invention provides a positioning method and apparatus, an electronic device, and a computer readable storage medium. According to the present invention, an anchor box, i.e., an object box corresponding to object box information, is determined for each feature point in an image feature map on the basis of the image feature map of a target image. The anchor box and the corresponding predicted anchor box are mutually exclusive, which reduces the number of anchor boxes used in the object positioning process, and reduces the computing amount. At the same time, object type information of an object to which each feature point in the image feature map belongs, the confidence score of the object box information thereof, and the confidence score of the object type information can also be determined on the basis of the image feature map of the target image, and thus the final confidence score of the object box information is determined on the basis of the determined two confidence scores, which effectively improves the information expression capability of an object box, and facilitates improving the accuracy of performing object positioning on the basis of the object box.

Description

Positioning method and device, electronic equipment, computer readable storage medium

Cross-references to related applications

This disclosure claims the priority of the Chinese patent application filed on January 18, 2020, the application number is 202010058788.7, and the invention title is "positioning method and device, electronic equipment, and computer-readable storage medium." All content is incorporated into this article by reference.

Technical field

The present disclosure relates to the fields of computer technology and image processing, and in particular to a positioning method and device, electronic equipment, and computer-readable storage media.

Background technique

Object detection or object positioning is an important basic technology in computer vision, which can be applied to scenes such as instance segmentation, object tracking, person recognition, and face recognition.

Object detection or object positioning usually uses anchor frames. However, if the number of anchor frames used is large and the expression ability of anchor frames is weak, it will lead to defects such as a large amount of calculation for object positioning and inaccurate positioning.

Summary of the invention

In view of this, the present disclosure provides at least one positioning method and device.

In the first aspect, the present disclosure provides a positioning method, including:

Acquiring a target image, where the target image includes at least one object to be positioned;

Based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs in the image feature map, the object frame information of the object to which each feature point belongs, the first confidence level of the object type information, and The second confidence level of the object frame information;

Based on the first confidence level and the second confidence level, respectively determine the target confidence level of the object frame information of the object to which each feature point belongs;

Based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information, the positioning information of the object in the target image is determined.

In the above embodiment, the image feature map based on the target image can determine only one anchor frame for each feature point in the image feature map, that is, the object frame corresponding to the object frame information, which reduces the number of anchor frames used in the object positioning process. , Which reduces the amount of calculation and improves the efficiency of object positioning. At the same time, the image feature map based on the target image can also determine the object type information of the object to which each feature point in the image feature map belongs, the confidence level of the object frame information, and the confidence level of the object type information, and then determine based on the determined two confidence levels The final confidence level of the object frame information effectively enhances the information expression ability of the object frame or object frame information. It can not only express the positioning information and object type information of the object frame corresponding to the object frame information, but also express the confidence level of the object frame information. Information, which helps to improve the accuracy of object positioning based on the object frame.

In a possible implementation manner, the image feature map includes a classification feature map used to classify the object to which the feature points in the image feature map belong and a classification feature map used to classify the feature points in the image feature map. The positioning feature map for positioning the object.

Based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs in the image feature map, the object frame information of the object to which each feature point belongs, the first confidence level of the object type information, and The second confidence level of the object frame information includes:

For each feature point in the image feature map, based on the classification feature map, determine the object type information of the object to which the feature point belongs, and the first confidence level of the object type information;

Based on the positioning feature map, the object frame information of the object to which the feature point belongs and the second confidence level of the object frame information are determined.

In the above embodiment, based on the classification feature map and the positioning feature map of the target image, not only the object frame information of the object to which each feature point belongs in the image feature map is determined, but also the object type of the object to which each feature point belongs in the image feature map is determined Information, as well as the respective confidence levels of the object type information and the object frame information, improve the information expression ability of the object frame, thereby helping to improve the accuracy of object positioning based on the object frame.

In a possible implementation manner, for each feature point in the image feature map, based on the positioning feature map, determining the object frame information of the object to which the feature point belongs includes:

For each feature point in the image feature map, based on the positioning feature map, respectively determine the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Based on the target distance range and the positioning feature map, respectively determine the target distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Based on the position information of the feature point in the image feature map and the target distance between the feature point and each boundary, the object frame information of the object to which the feature point belongs is determined.

In the foregoing embodiment, the target distance range in which the distance between the characteristic point and each boundary in the object frame of the object to which the characteristic point belongs is first determined, and then, based on the determined target distance range, the target distance between the characteristic point and each boundary is determined After the two-step processing, the accuracy of the determined target distance can be improved. After that, based on the determined precise target distance, an accurately positioned object frame can be determined for the feature point, which improves the accuracy of the determined object frame.

In a possible implementation manner, determining the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs includes:

For each boundary in the object frame of the object to which the feature point belongs, determine the maximum distance between the feature point and the boundary based on the positioning feature map;

Perform segmentation processing on the maximum distance to obtain multiple distance ranges;

Based on the positioning feature map, determine the first probability value at which the distance between the feature point and the boundary is within each distance range;

Based on the determined first probability value, select the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges.

In the above embodiment, the distance range corresponding to the maximum probability value can be selected as the target distance range in which the distance between the feature point and a certain frame is located, which improves the accuracy of the determined target distance range, thereby helping to improve the determination based on the target distance range The accuracy of the distance between the characteristic point and a certain boundary.

In a possible implementation manner, based on the determined first probability value, selecting the target distance range in which the distance between the characteristic point and the boundary is located from the multiple distance ranges includes:

The distance range corresponding to the largest first probability value is used as the target distance range.

In a possible implementation manner, based on the determined first probability value, selecting the target distance range in which the distance between the characteristic point and the boundary is located from the multiple distance ranges includes:

Based on the positioning feature map, determine the distance uncertainty parameter value of the distance between the feature point and the border;

Based on the distance uncertainty parameter value and each first probability value, determine the target probability value where the distance between the characteristic point and the boundary is within each distance range;

The distance range corresponding to the maximum target probability value is taken as the target distance range in which the distance between the characteristic point and the boundary is located.

In the foregoing embodiment, while determining the first probability value at which the distance between the feature point and a certain border is within each distance range, an uncertain parameter value is also determined, and the first probability can be corrected based on the uncertain parameter value Or correction to obtain the target probability value where the distance between the feature point and a certain frame is within each distance range, which improves the accuracy of the probability value of the determined feature point and the distance between a certain frame within each distance range, so that there is It is beneficial to improve the accuracy of the target distance range determined based on the probability value.

In a possible implementation manner, determining the second confidence level of the object frame information includes:

Based on the first probability value corresponding to the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs is determined, the second confidence level of the object frame information of the object to which the feature point belongs is determined.

In a possible implementation manner, determining the second confidence level of the object frame information of the object to which the characteristic point belongs includes:

Acquiring the average value of the first probability value corresponding to the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Determine the mean value as the second confidence level.

In the foregoing embodiment, the first probability value corresponding to the distance range in which the distance between the feature point and each boundary is located can be used to determine the confidence level of the object frame information of the object to which the feature point belongs, and enhance the information expressing ability of the object frame.

In a possible implementation manner, for each feature point in the image feature map, determining the object type information of the object to which the feature point belongs based on the classification feature map includes:

For each feature point in the image feature map, based on the classification feature map, determine that the object to which the feature point belongs is the second probability value of each preset object type;

Based on the preset object type corresponding to the largest second probability value, the object type information of the object to which the feature point belongs is determined.

In the foregoing embodiment, the preset object type corresponding to the largest second probability value is selected as the object type information of the object to which the feature point belongs, which improves the accuracy of the determined object type information.

In a possible implementation manner, determining the positioning information of the object in the target image based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information includes:

Multiple target feature points are filtered from the image feature map, wherein the distance between the multiple target feature points is less than a preset threshold, and the object type information of the object to which each target feature point belongs is the same;

From the object frame information of the object to which each target feature point belongs, select the object frame information with the highest target confidence as the target frame information;

Based on the selected target frame information and the target confidence of the target frame information, the positioning information of the object in the target image is determined.

In the above embodiment, the object frame information with the highest target confidence is selected from the feature points with the same object type information at a relatively close distance to locate the object, which can effectively reduce the amount of object frame information used for object positioning. It is helpful to improve the timeliness of object positioning.

In a second aspect, the present disclosure provides a positioning device, including:

An image acquisition module for acquiring a target image, wherein the target image includes at least one object to be located;

An image processing module for determining the object type information of the object to which each feature point belongs, the object frame information of the object to which each feature point belongs, and the object type information in the image feature map based on the image feature map of the target image The first confidence level of and the second confidence level of the object frame information;

A confidence processing module, configured to determine the target confidence of the object frame information of the object to which each feature point belongs based on the first confidence and the second confidence;

The positioning module is configured to determine the positioning information of the object in the target image based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information.

In a possible implementation manner, the image feature map includes a classification feature map used to classify the object to which the feature points in the image feature map belong and a classification feature map used to classify the feature points in the image feature map. The positioning feature map for positioning the object.

The image processing module is used for:

For each feature point in the image feature map, based on the classification feature map, determine the object type information of the object to which the feature point belongs, and the first confidence level of the object type information;

Based on the positioning feature map, the object frame information of the object to which the feature point belongs and the second confidence level of the object frame information are determined.

In a possible implementation manner, for each feature point in the image feature map, when the image processing module determines the object frame information of the object to which the feature point belongs based on the positioning feature map, it is used to:

For each feature point in the image feature map, based on the positioning feature map, respectively determine the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Based on the target distance range and the positioning feature map, respectively determine the target distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Based on the position information of the feature point in the image feature map and the target distance between the feature point and each boundary, the object frame information of the object to which the feature point belongs is determined.

In a possible implementation, the image processing module is used to determine the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs:

For each boundary in the object frame of the object to which the feature point belongs, determine the maximum distance between the feature point and the boundary based on the positioning feature map;

Perform segmentation processing on the maximum distance to obtain multiple distance ranges;

Based on the positioning feature map, determine the first probability value at which the distance between the feature point and the boundary is within each distance range;

Based on the determined first probability value, select the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges.

In a possible implementation manner, the image processing module selects the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges based on the determined first probability value When used for:

The distance range corresponding to the largest first probability value is used as the target distance range.

In a possible implementation manner, the image processing module selects the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges based on the determined first probability value When used for:

Based on the positioning feature map, determine the distance uncertainty parameter value of the distance between the feature point and the border;

Based on the distance uncertainty parameter value and each first probability value, determine the target probability value where the distance between the characteristic point and the boundary is within each distance range;

The distance range corresponding to the maximum target probability value is taken as the target distance range where the distance between the characteristic point and the boundary is located.

In a possible implementation manner, when the image processing module determines the second confidence level of the object frame information, it is configured to:

Based on the first probability value corresponding to the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs is determined, the second confidence level of the object frame information of the object to which the feature point belongs is determined.

In a possible implementation manner, when the image processing module determines the second confidence level of the object frame information of the object to which the characteristic point belongs, it is used to:

Acquiring the average value of the first probability value corresponding to the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Determine the mean value as the second confidence level.

In a possible implementation manner, for each feature point in the image feature map, the image processing module is used to determine the object type information of the object to which the feature point belongs based on the classification feature map for each feature point:

For each feature point in the image feature map, based on the classification feature map, determine that the object to which the feature point belongs is the second probability value of each preset object type;

Based on the preset object type corresponding to the largest second probability value, the object type information of the object to which the feature point belongs is determined.

In a possible implementation manner, the positioning module is used to:

Multiple target feature points are filtered from the image feature map, wherein the distance between the multiple target feature points is less than a preset threshold, and the object type information of the object to which each target feature point belongs is the same;

From the object frame information of the object to which each target feature point belongs, select the object frame information with the highest target confidence as the target frame information;

Based on the selected target frame information and the target confidence of the target frame information, the positioning information of the object in the target image is determined.

In a third aspect, the present disclosure provides an electronic device including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor is connected to the The memories communicate through a bus, and when the machine-readable instructions are executed by the processor, the steps of the positioning method described above are executed.

In a fourth aspect, the present disclosure also provides a computer-readable storage medium with a computer program stored on the computer-readable storage medium, and the computer program executes the steps of the positioning method described above when the computer program is run by a processor.

The apparatus, electronic equipment, and computer-readable storage medium of the present disclosure contain at least technical features that are substantially the same as or similar to the technical features of any aspect of the method or any implementation of any aspect of the present disclosure. For the description of the effects of the device, electronic equipment, and computer-readable storage medium, please refer to the description of the effects of the content of the method, which is not repeated here.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show certain embodiments of the present disclosure, and therefore do not It should be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative work.

FIG. 1 shows a flowchart of a positioning method provided by an embodiment of the present disclosure;

Figure 2 shows a flowchart of a positioning method provided by an embodiment of the present disclosure;

FIG. 3 shows a flowchart of a positioning method provided by an embodiment of the present disclosure;

FIG. 4 shows a flowchart of a positioning method provided by an embodiment of the present disclosure;

FIG. 5 shows a flowchart of a positioning method provided by an embodiment of the present disclosure;

FIG. 6 shows a schematic structural diagram of a positioning device provided by an embodiment of the present disclosure;

Fig. 7 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose and advantages of the embodiments of the present disclosure clearer, the following will clearly describe the embodiments of the present disclosure in conjunction with the accompanying drawings in the embodiments of the present disclosure. It should be understood that the drawings in the present disclosure are only for the purpose of illustration and description. It is not used to limit the protection scope of the present disclosure. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowchart used in the present disclosure shows operations implemented according to some embodiments of the present disclosure. It should be understood that the operations of the flowchart may be implemented out of order, and steps without logical context may be reversed in order or implemented at the same time. In addition, under the guidance of the present disclosure, those skilled in the art can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

In addition, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure generally described and illustrated in the drawings herein may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided with reference to the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present disclosure.

It should be noted that the term "including" will be used in the embodiments of the present disclosure to indicate the existence of the features declared thereafter, but it does not exclude the addition of other features.

Aiming at how to reduce the number of anchor frames used for positioning and improve the information expression ability of the anchor frames in the process of object positioning by using anchor frames, so as to improve the accuracy of object positioning, the present disclosure provides a positioning method and device, and electronic equipment , Computer-readable storage medium. Among them, based on the image feature map of the target image in the present disclosure, only one anchor frame is determined for each feature point in the image feature map, that is, the object frame corresponding to the object frame information, which reduces the number of anchor frames used in the object positioning process. The amount of calculation. At the same time, the image feature map based on the target image can also determine the object type information of the object to which each feature point in the image feature map belongs, the confidence level of the object frame information, and the confidence level of the object type information, and then determine based on the determined two confidence levels The final confidence level of the object frame information effectively enhances the information expression ability of the object frame, and is beneficial to improve the accuracy of object positioning based on the object frame.

The positioning method and device, electronic equipment, and computer-readable storage medium of the present disclosure will be described below through specific embodiments.

The embodiments of the present disclosure provide a positioning method, which is applied to a terminal device for positioning an object in an image. The terminal device may be a camera, a mobile phone, a wearable device, a personal computer, etc., which is not limited in the embodiment of the present disclosure. Specifically, as shown in FIG. 1, the positioning method provided by an embodiment of the present disclosure includes steps S110 to S140.

S110. Obtain a target image.

Here, the target image may be an image including a target object captured during the object tracking process, or an image including a human face captured during face detection. The purpose of the target image is not limited in the present disclosure.

The target image includes at least one object to be positioned. The objects here can be objects, people, animals, etc.

The target image may be captured by the terminal device that executes the positioning method of this embodiment, or it may be captured by another device and transmitted to the terminal device that executes the positioning method of this embodiment. The method of obtaining the target image is not limited in the present disclosure.

S120. Based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs, the object frame information of the object to which each feature point belongs, and the first confidence of the object type information in the image feature map Degree and the second confidence degree of the object frame information.

Before performing this step, the target image needs to be processed first to obtain the image feature map of the target image. In specific implementation, a convolutional neural network can be used to extract image features of the target image to obtain an image feature map.

After the image feature map of the target image is determined, the image feature map is processed. Thus, for each feature point in the image feature map, the object type information of the object to which the feature point belongs, the object frame information of the object to which the feature point belongs, the first confidence level of the object type information, and the object frame can be determined The second degree of confidence in the information. In a specific implementation, a convolutional neural network may be used to perform further image feature extraction on the image feature map to obtain the object type information, the object frame information, the first confidence level and the second confidence level.

The object type information includes the object category of the object to which the feature point belongs. The object frame information includes the distance between the characteristic point and each boundary in the object frame corresponding to the object frame information. Wherein, the object frame may also be referred to as an anchor frame.

The first confidence is used to characterize the accuracy or credibility of the object type information determined based on the image feature map. The second confidence is used to characterize the accuracy or credibility of the object frame information determined based on the image feature map.

S130: Based on the first confidence and the second confidence, respectively determine the target confidence of the object frame information of the object to which each feature point belongs.

Here, the product of the first confidence level and the second confidence level may be used as the target confidence level of the target frame information. The target confidence is used to comprehensively characterize the positioning accuracy and classification accuracy of the object frame corresponding to the object frame information.

Of course, other methods can also be used to determine the target confidence. For example, the preset weight of the first confidence, the preset weight of the second confidence, the first confidence and the second confidence can be combined to determine the target confidence. The disclosure does not limit the specific implementation scheme for determining the target confidence based on the first confidence and the second confidence.

S140: Determine positioning information of the object in the target image based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information.

Here, the object frame information of the object to which the feature point belongs and the target confidence of the object frame information can be used as the location information of the object to which the feature point belongs in the target image. Then, based on the location information of the object to which each feature point belongs in the target image, Determine the location information of each object in the target image.

Here, not only the object frame information of the object to which the feature point belongs is determined, but also the target confidence level of the object frame information is determined, which effectively enhances the information expression ability of the object frame or object frame information, and not only can express the object frame information corresponding to the object frame information. The positioning information and object type information can also express the confidence information of the object frame information, thereby helping to improve the accuracy of object positioning based on the object frame.

In addition, the above embodiment can determine an anchor frame for each feature point in the image feature map based on the image feature map of the target image, that is, the object frame corresponding to the object frame information, which reduces the number of anchor frames used in the object positioning process. Reduce the amount of calculation and improve the efficiency of object positioning.

In some examples, as shown in FIG. 2, the image feature map includes a classification feature map used to classify the object to which the feature points in the image feature map belong and a classification feature map used to classify the features in the image feature map. The positioning feature map for positioning the object to which the point belongs.

In the specific implementation, as shown in Figure 2, the convolutional neural network can be used to extract the image features of the target image to obtain the initial feature map, and then use 4 3×3 convolutional layer pairs with input and output 256. The initial feature map is processed to obtain the classification feature map and the positioning feature map.

After obtaining the classification feature map and the positioning feature map, based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs in the image feature map, the object frame information of the object to which each feature point belongs, and The first confidence level of the object type information and the second confidence level of the object frame information can be implemented by using the following steps:

Based on the classification feature map, determine the object type information of the object to which each feature point in the image feature map belongs, and the first confidence level of the object type information; determine the image feature map based on the positioning feature map The object frame information of the object to which each feature point belongs, and the second confidence level of the object frame information.

In a specific implementation, a convolutional neural network or a convolutional layer may be used to perform image feature extraction on the classification feature map to obtain the object type information of the object to which each feature point belongs, and the first confidence level of the object type information. Using a convolutional neural network or a convolutional layer to perform image feature extraction on the positioning feature map, the object frame information of the object to which each feature point belongs and the second confidence level of the object frame information are obtained.

In the above embodiment, based on the classification feature map and the positioning feature map of the target image, not only the object frame information of the object to which each feature point belongs in the image feature map is determined, but also the object type of the object to which each feature point belongs in the image feature map is determined Information, and the respective confidence levels corresponding to the object type information and the object frame information, improve the information expression ability of the object frame, thereby helping to improve the accuracy of object positioning based on the object frame.

In some embodiments, as shown in FIG. 3, based on the positioning feature map, determining the object frame information of the object to which each feature point in the image feature map belongs can be implemented by using steps S310 to S330.

S310: For each feature point in the image feature map, based on the positioning feature map, respectively determine the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs is located. Wherein, each boundary in the object frame may be a boundary of the object frame in various directions, for example, the upper boundary, the lower boundary, the left boundary, and the right boundary in the object frame.

Here, a convolutional neural network or a convolutional layer may be used to perform image feature extraction on the positioning feature map to determine the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs.

In specific implementation, you can first determine the maximum distance between the feature point and a certain boundary based on the positioning feature map; then, perform segmentation processing on the maximum distance to obtain multiple distance ranges; and use convolutional neural networks or convolutional neural networks. The layered layer performs image feature extraction on the positioning feature map to determine the first probability value of the distance between the feature point and the border within each distance range; finally, based on the determined first probability value, from the multiple Among the distance ranges, select the target distance range where the distance between the characteristic point and the boundary is located. Specifically, the distance range corresponding to the largest first probability value may be used as the target distance range.

As shown in FIG. 2, the object frame may include, for example, an upper boundary, a lower boundary, a left boundary, and a right boundary. Based on the method, five first probability values a, b, c of the five distance ranges corresponding to the left boundary are determined. d, e, and select the distance range corresponding to the largest first probability value b as the target distance range.

As mentioned above, the distance range corresponding to the maximum probability value is selected as the target distance range where the distance between the feature point and the boundary is located, which improves the accuracy of the determined target distance range, thereby helping to improve the feature points determined based on the target distance range The accuracy of the distance to a certain boundary.

S320: Based on the target distance range and the positioning feature map, respectively determine the target distance between the feature point and each boundary in the object frame of the object to which the feature point belongs.

After determining the target distance range, select a regression network that matches the target distance range, such as a convolutional neural network, and perform image feature extraction on the location feature map to obtain the feature point and each of the object borders of the object to which the feature point belongs The target distance of the boundary.

Here, on the basis of determining the target distance range, a convolutional neural network is further used to determine an accurate distance, which can effectively improve the accuracy of the determined distance.

In addition, as shown in FIG. 2, after the target distance is determined, a preset or trained parameter or weight N can be used to correct the determined target distance to obtain the final target distance.

As shown in Figure 2, the precise target distance between the feature point and the left boundary is determined using this step. The target distance is marked in Figure 2 and denoted by f. As shown in Figure 2, the determined target distance is within the determined target distance range.

S330: Determine the object frame information of the object to which the feature point belongs based on the position information of the feature point in the image feature map and the target distance between the feature point and each boundary.

Here, the location information of the feature point in the image feature map and the target distance between the feature point and each boundary can be used to determine the location information of each boundary in the object frame corresponding to the object frame information in the image feature map. Finally, the position information of all boundaries in the object frame in the image feature map can be used as the object frame information of the object to which the feature point belongs.

In the above embodiment, the target distance range in which the distance between the characteristic point and each boundary in the object frame is first determined, and then, based on the determined target distance range, the target distance between the characteristic point and each boundary is determined, and the two-step processing is performed The accuracy of the determined target distance can be improved. After that, based on the determined precise target distance, an accurately positioned object frame can be determined for the feature point, which improves the accuracy of the determined object frame.

In some embodiments, as shown in FIG. 4, based on the determined first probability value, from the multiple distance ranges, select the target distance range in which the distance between the feature point and a certain boundary is located, and it can also use Steps S410 to S430 are implemented.

S410: Based on the positioning feature map, determine a distance uncertainty parameter (distance uncertainty parameter) value of the distance between the feature point and a certain boundary.

Here, a convolutional neural network can be used to determine the first probability value where the distance between the feature point and a certain boundary is within each distance range, and at the same time determine the distance uncertainty parameter value of the distance between the feature point and the boundary. The distance uncertainty parameter value here can be used to characterize the credibility of the determined first probabilities.

S420: Based on the distance uncertainty parameter value and each first probability value, determine the target probability value where the distance between the characteristic point and the boundary is within each distance range.

Here, each first probability value is corrected by using the distance uncertainty parameter value to obtain the corresponding target probability value.

In specific implementation, the following formula can be used to determine the target probability value:

In the formula, p _{x, n} represents the target probability value of the distance between the feature point and the boundary x within the nth distance range, N represents the number of the distance range, σ _x represents the distance uncertainty parameter value corresponding to the boundary x, s _{x , N} represents the first probability value that the distance between the feature point and the boundary x is within the n-th distance range; s _x,m represents the first probability value that the distance between the feature point and the boundary x is within the m-th distance range.

S430: Based on the determined target probability value, select a target distance range in which the distance between the characteristic point and the boundary is located from the multiple distance ranges.

Here, specifically, the distance range corresponding to the maximum target probability value can be selected as the target distance range.

In the foregoing embodiment, while determining the first probability value at which the distance between the characteristic point and a certain boundary is within each distance range, a distance uncertainty parameter value is also determined, and the first probability can be corrected or corrected based on the parameter value. Correction to obtain the target probability value where the distance between the feature point and a certain boundary is within each distance range, which improves the accuracy of the probability value of the determined feature point and a certain boundary within each distance range, which is beneficial to Improve the accuracy of the target distance range determined based on the probability value.

After determining the target distance between the feature point and each boundary in the corresponding object frame, the following steps can be used to determine the confidence of the corresponding object frame information, that is, the second confidence: based on the feature points in the image feature map The first probability value corresponding to the target distance range in which the distance of each boundary in the object frame of the object to which the feature point belongs determines the second confidence level of the object frame information of the object to which the feature point belongs.

In a specific implementation, the average value of the first probability value corresponding to the target distance range where the distance between the feature point and all boundaries in the object frame of the object to which the feature point belongs may be used as the second confidence.

Of course, other methods may also be used to determine the second confidence level, and the present disclosure does not limit the method for determining the second confidence level based on the first probability value corresponding to the target distance range.

In the foregoing embodiment, the first probability value corresponding to the distance range in which the distance between the feature point and each boundary is located can be used to determine the confidence level of the object frame information of the object to which the feature point belongs, that is, the second confidence level, which enhances The information expression ability of the object frame.

In some embodiments, based on the classification feature map, determining the object type information of the object to which each feature point in the image feature map belongs can be achieved by using the following steps: based on the classification feature map, determining the image feature map The object to which each feature point belongs is the second probability value of each preset object type; based on the preset object type corresponding to the largest second probability value, the object type information of the object to which the feature point belongs is determined.

In a specific implementation, a convolutional neural network or a convolutional layer may be used to perform image feature extraction on the classification feature map to obtain the second probability value of the object to which the feature point belongs is each preset object type. Then, the preset object type corresponding to the largest second probability value is selected to determine the object type information of the object to which the feature point belongs. As shown in FIG. 2, the second probability value corresponding to the preset object type "cat" determined by this embodiment is the largest, so it is determined that the object type information corresponds to a cat. It should be noted that in this article, different operations can use different parts of the same convolutional neural network.

In the foregoing embodiment, the preset object type corresponding to the largest second probability value is selected as the object type information of the object to which the feature point belongs, which improves the accuracy of the determined object type information.

In some embodiments, as shown in FIG. 5, based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information, to determine the location information of the object in the target image, steps S510 to S530 implementation.

S510. Filter out multiple target feature points from the image feature map, where the distance between the multiple target feature points is less than a preset threshold, and the object type information of the object to which each target feature point belongs is the same.

Here, the multiple target feature points obtained by screening are feature points belonging to the same object.

S520: From the object frame information of the object to which each target feature point belongs, select the object frame information with the highest target confidence as the target frame information.

For feature points belonging to the same object, the object frame information corresponding to the highest target confidence can be selected to locate the object, and other object frame information with lower target confidence can be eliminated to reduce the amount of calculation in the object positioning process.

S530: Determine positioning information of the object in the target image based on the selected target frame information and the target confidence of the target frame information.

In the above embodiment, the object frame information with the highest target confidence is selected from the object frame information corresponding to the feature points with the same object type information at a relatively close distance to locate the object, which can effectively reduce the object frame used for object positioning. The amount of information is conducive to improving the timeliness of object positioning.

Corresponding to the positioning method, the embodiments of the present disclosure also provide a positioning device, which locates an object in an image on a terminal device, and the device and its various modules can perform the same method as the positioning method. Steps, and can achieve the same or similar beneficial effects, so the repeated parts will not be repeated.

As shown in Figure 6, the positioning device provided by the present disclosure includes:

The image acquisition module 610 is configured to acquire a target image, where the target image includes at least one object to be located.

The image processing module 620 is configured to determine, based on the image feature map of the target image, the object type information of the object to which each feature point belongs, the object frame information of the object to which each feature point belongs, and the object type in the image feature map The first confidence level of the information and the second confidence level of the object frame information.

The confidence processing module 630 is configured to determine the target confidence of the object frame information of the object to which each feature point belongs based on the first confidence and the second confidence.

The positioning module 640 is configured to determine the positioning information of the object in the target image based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information.

In some embodiments, the image feature map includes a classification feature map for classifying objects to which feature points in the image feature map belong, and a classification feature map for classifying objects to which the feature points in the image feature map belong. Positioning feature map for positioning.

The image processing module 620 is used to:

Based on the classification feature map, determine the object type information of the object to which each feature point in the image feature map belongs, and the first confidence level of the object type information;

Based on the positioning feature map, determine the object frame information of the object to which each feature point in the image feature map belongs, and the second confidence level of the object frame information.

In some embodiments, when the image processing module 620 determines the object frame information of the object to which each feature point in the image feature map belongs based on the positioning feature map, it is used to:

For each feature point in the image feature map, based on the positioning feature map, respectively determine the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Based on the target distance range and the positioning feature map, respectively determine the target distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Based on the position information of the feature point in the image feature map and the target distance between the feature point and each boundary, the object frame information of the object to which the feature point belongs is determined.

In some embodiments, the image processing module 620 is used to determine the target distance range in which the distance between a feature point and each boundary in the object frame of the object to which the feature point belongs:

For each boundary in the object frame of the object to which the feature point belongs, determine the maximum distance between the feature point and the boundary based on the positioning feature map;

Perform segmentation processing on the maximum distance to obtain multiple distance ranges;

Based on the positioning feature map, determine the first probability value at which the distance between the feature point and the boundary is within each distance range;

Based on the determined first probability value, select the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges.

In some embodiments, when the image processing module selects the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges based on the determined first probability value, use At:

The distance range corresponding to the largest first probability value is used as the target distance range.

In some embodiments, when the image processing module 620 selects the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges based on the determined first probability value, Used for:

Based on the positioning feature map, determine the distance uncertainty parameter value of the distance between the feature point and the border;

Based on the distance uncertainty parameter value and each first probability value, determine the target probability value where the distance between the characteristic point and the boundary is within each distance range;

The distance range corresponding to the maximum target probability value is taken as the target distance range where the distance between the characteristic point and the boundary is located.

In some embodiments, when the image processing module 620 determines the second confidence level of the object frame information, it is configured to:

Determine the object frame information of the object to which the feature point belongs based on the first probability value corresponding to the distance between a feature point in the image feature map and each boundary in the object frame of the object to which the feature point belongs. The second degree of confidence.

In some embodiments, when the image processing module determines the second confidence level of the object frame information of the object to which the characteristic point belongs, it is used to:

Acquiring the average value of the first probability value corresponding to the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs;

Determine the mean value as the second confidence level.

In some embodiments, when the image processing module 620 determines the object type information of the object to which each feature point in the image feature map belongs based on the classification feature map, it is used to:

Based on the classification feature map, determining that the object to which each feature point belongs in the image feature map is a second probability value of each preset object type;

Based on the preset object type corresponding to the largest second probability value, the object type information of the object to which the feature point belongs is determined.

In some embodiments, the positioning module 640 is used to:

Multiple target feature points are filtered from the image feature map, wherein the distance between the multiple target feature points is less than a preset threshold, and the object type information of the object to which each target feature point belongs is the same;

From the object frame information of the object to which each target feature point belongs, select the object frame information with the highest target confidence as the target frame information;

Based on the selected target frame information and the target confidence of the target frame information, the positioning information of the object in the target image is determined.

The embodiment of the present disclosure discloses an electronic device, as shown in FIG. 7, comprising: a processor 701, a memory 702, and a bus 703. The memory 702 stores machine-readable instructions executable by the processor 701. When the device is running, the processor 701 and the memory 702 communicate through the bus 703.

When the machine-readable instruction is executed by the processor 701, the following positioning method steps are executed:

Acquiring a target image, where the target image includes at least one object to be positioned;

Based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs in the image feature map, the object frame information of the object to which each feature point belongs, the first confidence level of the object type information, and The second confidence level of the object frame information;

Based on the first confidence level and the second confidence level, respectively determine the target confidence level of the object frame information of the object to which each feature point belongs;

Based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information, the positioning information of the object in the target image is determined.

In addition, when the machine-readable instruction is executed by the processor 701, the method content in any of the implementation manners described in the method section can also be executed, which will not be repeated here.

The embodiment of the present disclosure also provides a computer program product corresponding to the method and device, which includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. For implementation, refer to the method embodiment, which will not be repeated here.

The above description of the various embodiments tends to emphasize the differences between the various embodiments, and the same or similarities can be referred to each other. For the sake of brevity, the details are not repeated herein.

Those skilled in the art can clearly understand that, for convenience and concise description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, which will not be repeated in this disclosure. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device, and method may be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation. For example, multiple modules or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some communication interfaces, devices or modules, and may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

The above are only specific implementations of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of changes or substitutions within the technical scope disclosed in the present disclosure, and they shall be covered Within the protection scope of this disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (22)

A positioning method, characterized in that it comprises: Acquiring a target image, where the target image includes at least one object to be positioned; Based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs in the image feature map, the object frame information of the object to which each feature point belongs, the first confidence level of the object type information, and The second confidence level of the object frame information; Based on the first confidence level and the second confidence level, respectively determine the target confidence level of the object frame information of the object to which each feature point belongs; Based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information, the positioning information of the object in the target image is determined. The positioning method according to claim 1, wherein the image feature map includes a classification feature map used to classify the object to which the feature points in the image feature map belong and a classification feature map used to classify the image feature map. The positioning feature map for positioning the object to which the feature points belong, Based on the image feature map of the target image, determine the object type information of the object to which each feature point belongs in the image feature map, the object frame information of the object to which each feature point belongs, the first confidence level of the object type information, and The second confidence level of the object frame information includes: For each feature point in the image feature map, Based on the classification feature map, determine the object type information of the object to which the feature point belongs, and the first confidence level of the object type information; Based on the positioning feature map, the object frame information of the object to which the feature point belongs and the second confidence level of the object frame information are determined. The positioning method according to claim 2, wherein for each feature point in the image feature map, based on the positioning feature map, determining the object frame information of the object to which the feature point belongs includes: For each feature point in the image feature map, Based on the positioning feature map, respectively determine the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs; Based on the target distance range and the positioning feature map, respectively determine the target distance between the feature point and each boundary in the object frame of the object to which the feature point belongs; Based on the position information of the feature point in the image feature map and the target distance between the feature point and each boundary, the object frame information of the object to which the feature point belongs is determined. The positioning method according to claim 3, wherein determining the target distance range in which the distance between the characteristic point and each boundary in the object frame of the object to which the characteristic point belongs comprises: For each boundary in the object frame of the object to which the feature point belongs, Based on the positioning feature map, determine the maximum distance between the feature point and the border; Perform segmentation processing on the maximum distance to obtain multiple distance ranges; Based on the positioning feature map, determine the first probability value at which the distance between the feature point and the boundary is within each distance range; Based on the determined first probability value, select the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges. The positioning method according to claim 4, wherein based on the determined first probability value, selecting a target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges, include: The distance range corresponding to the largest first probability value is used as the target distance range. The positioning method according to claim 4, wherein based on the determined first probability value, selecting a target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges, include: Based on the positioning feature map, determine the distance uncertainty parameter value of the distance between the feature point and the border; Based on the distance uncertainty parameter value and each first probability value, determine the target probability value where the distance between the characteristic point and the boundary is within each distance range; The distance range corresponding to the maximum target probability value is taken as the target distance range in which the distance between the characteristic point and the boundary is located. The positioning method according to claim 4, wherein determining the second confidence level of the object frame information comprises: Based on the first probability value corresponding to the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs is determined, the second confidence level of the object frame information of the object to which the feature point belongs is determined. The positioning method according to claim 7, wherein determining the second confidence level of the object frame information of the object to which the characteristic point belongs comprises: Acquiring the average value of the first probability value corresponding to the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs; Determine the mean value as the second confidence level. The positioning method according to any one of claims 2 to 8, wherein for each feature point in the image feature map, based on the classification feature map, the object type information of the object to which the feature point belongs is determined, including : For each feature point in the image feature map, Based on the classification feature map, determining that the object to which the feature point belongs is the second probability value of each preset object type; Based on the preset object type corresponding to the largest second probability value, the object type information of the object to which the feature point belongs is determined. The positioning method according to any one of claims 1 to 9, wherein the positioning of the object in the target image is determined based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information Information, including: Multiple target feature points are filtered from the image feature map, wherein the distance between the multiple target feature points is less than a preset threshold, and the object type information of the object to which each target feature point belongs is the same; From the object frame information of the object to which each target feature point belongs, select the object frame information with the highest target confidence as the target frame information; Based on the selected target frame information and the target confidence of the target frame information, the positioning information of the object in the target image is determined. A positioning device is characterized in that it comprises: An image acquisition module for acquiring a target image, wherein the target image includes at least one object to be located; An image processing module for determining the object type information of the object to which each feature point belongs, the object frame information of the object to which each feature point belongs, and the object type information in the image feature map based on the image feature map of the target image The first confidence level of and the second confidence level of the object frame information; A confidence processing module, configured to determine the target confidence of the object frame information of the object to which each feature point belongs based on the first confidence and the second confidence; The positioning module is configured to determine the positioning information of the object in the target image based on the object frame information of the object to which each feature point belongs and the target confidence of the object frame information. The positioning device according to claim 11, wherein the image feature map comprises a classification feature map used to classify the object to which the feature points in the image feature map belong and a classification feature map used to classify the image feature map. The positioning feature map for positioning the object to which the feature points belong, The image processing module is used for: For each feature point in the image feature map, Based on the classification feature map, determine the object type information of the object to which the feature point belongs, and the first confidence level of the object type information; Based on the positioning feature map, the object frame information of the object to which the feature point belongs and the second confidence level of the object frame information are determined. The positioning device according to claim 12, wherein the image processing module determines the object frame information of the object to which the feature point belongs based on the positioning feature map for each feature point in the image feature map For: For each feature point in the image feature map, Based on the positioning feature map, respectively determine the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs; Based on the target distance range and the positioning feature map, respectively determine the target distance between the feature point and each boundary in the object frame of the object to which the feature point belongs; Based on the position information of the feature point in the image feature map and the target distance between the feature point and each boundary, the object frame information of the object to which the feature point belongs is determined. The positioning device according to claim 13, wherein the image processing module uses the target distance range when determining the distance between the characteristic point and each boundary in the object frame of the object to which the characteristic point belongs. At: For each boundary in the object frame of the object to which the feature point belongs, determine the maximum distance between the feature point and the boundary based on the positioning feature map; Perform segmentation processing on the maximum distance to obtain multiple distance ranges; Based on the positioning feature map, determine the first probability value at which the distance between the feature point and the boundary is within each distance range; Based on the determined first probability value, select the target distance range in which the distance between the characteristic point and the boundary is located from the plurality of distance ranges. The positioning device according to claim 14, wherein the image processing module selects the distance between the characteristic point and the boundary from the plurality of distance ranges based on the determined first probability value. When in the target distance range, it is used to: The distance range corresponding to the largest first probability value is used as the target distance range. The positioning device according to claim 14, wherein the image processing module selects the distance between the characteristic point and the boundary from the plurality of distance ranges based on the determined first probability value. When in the target distance range, it is used to: Based on the positioning feature map, determine the distance uncertainty parameter value of the distance between the feature point and the border; Based on the distance uncertainty parameter value and each first probability value, determine the target probability value where the distance between the characteristic point and the boundary is within each distance range; The distance range corresponding to the maximum target probability value is taken as the target distance range where the distance between the characteristic point and the boundary is located. The positioning device according to claim 14, wherein the image processing module is configured to: when determining the second confidence level of the object frame information: Based on the first probability value corresponding to the target distance range where the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs is determined, the second confidence level of the object frame information of the object to which the feature point belongs is determined. The positioning device according to claim 17, wherein the image processing module is used to: Acquiring the average value of the first probability value corresponding to the target distance range in which the distance between the feature point and each boundary in the object frame of the object to which the feature point belongs; Determine the mean value as the second confidence level. The positioning device according to any one of claims 12 to 18, wherein, for each feature point in the image feature map, the image processing module determines the object to which the feature point belongs based on the classification feature map. When the object type information is used for: For each feature point in the image feature map, Based on the classification feature map, determining that the object to which the feature point belongs is the second probability value of each preset object type; Based on the preset object type corresponding to the largest second probability value, the object type information of the object to which the feature point belongs is determined. The positioning device according to any one of claims 11 to 19, wherein the positioning module is configured to: Multiple target feature points are filtered from the image feature map, wherein the distance between the multiple target feature points is less than a preset threshold, and the object type information of the object to which each target feature point belongs is the same; From the object frame information of the object to which each target feature point belongs, select the object frame information with the highest target confidence as the target frame information; Based on the selected target frame information and the target confidence of the target frame information, the positioning information of the object in the target image is determined. An electronic device, characterized by comprising: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the bus The storage media communicate through a bus, and the processor executes the machine-readable instructions to execute the positioning method according to any one of claims 1 to 10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the positioning method according to any one of claims 1 to 10 when the computer program is run by a processor.

Images