WO2023058999A1 - Object of interest detection device and method, and computer-readable program for same - Google Patents

Abstract

Disclosed are an object of interest detection device and method, and a computer-readable program for same. The object of interest detection device according to the present invention detects an object of interest in an input image on the basis of one or more anchor boxes. The object of interest detection device comprises: a multi-extension feature map extraction unit for extracting, from the input image, a plurality of multi-extension feature maps based on the shapes of the anchor boxes; a feature combination unit for generating a combined feature map by combining features extracted from each of the multi-extension feature maps; and an object detection unit for detecting the object of interest by predicting an offset from the combined feature map. According to the invention described above, the multi-extension feature maps based on the various shapes of the anchor boxes are extracted from the input image, and the object of interest is detected from the combined feature map obtained by mutually combining the features, and thus the object detection can be performed more robustly on the shape of the object.

Description

Object of interest detection device, method and computer readable program therefor

The present invention relates to an object-of-interest detection apparatus, method, and computer readable program therefor, and more particularly, to an object-of-interest detection apparatus, method, and computer-readable program for detecting an object of interest based on an anchor box on an input image. am.

Object detection is one of the most actively researched areas in the field of computer vision.

Object detection refers to automatically classifying the type of object to be found in an image and automatically expressing the location and area of the identified object as a bounding box.

Recently, the importance of such object detection technology has emerged as autonomous driving and security monitoring technology have grown.

In particular, object detection technology in autonomous driving automatically detects cars or pedestrians seen on the road while driving and provides information such as road conditions and traffic information to the driver. Because of these advantages, object detection in a driving environment is highly likely to be used in various applications such as self-driving cars and automatic sensing systems, and the related market size is rapidly expanding.

Accordingly, there is a need for an object detection technique capable of more accurately detecting an object.

[Prior art literature]

[Patent Literature]

(Patent Document 1) Korean Patent Registration No. 10-1480065

According to one aspect of the present invention, a multiple extension-feature map based on the shape of an anchor box is extracted from an input image, a combined feature map is generated by combining the extracted features, and an offset is predicted from the combined feature map to obtain an object of interest An object of the present invention is to provide an object of interest detection device, method, and computer readable program for detecting the object of interest.

An object of interest detection apparatus according to an embodiment of the present invention is an object of interest detection apparatus for detecting an object of interest based on at least one anchor box in an input image, wherein a plurality of multiple extensions are detected based on the shape of the anchor box from the input image. - A multi-extension-feature map extractor extracting a feature map, a feature combining unit generating a combined feature map by combining features extracted from each multi-extension-feature map, and predicting an offset from the combined feature map and an object detection unit that detects the object of interest.

Meanwhile, the multi-extension-feature map extraction unit may extract the multi-extension-feature map by performing an extended convolution operation having a ratio corresponding to the shape of the anchor box.

In addition, the multiple extension-feature map extraction unit may perform the extended convolution operation by applying a plurality of convolution kernels corresponding to the aspect ratio of the anchor box.

In addition, the feature combiner extracts output features by applying a 1x1 convolution operation to the plurality of multi-extension-feature maps, divides the output features, and combines the divided output features to generate a plurality of combined feature maps. it could be

In addition, the object detection unit may predict the offset by performing a convolution operation by applying a convolution kernel having a fixed size to each of the plurality of combined feature maps.

A method of detecting an object of interest according to another embodiment of the present invention is a method of detecting an object of interest based on at least one anchor box on an input image in an object of interest detection apparatus, wherein a plurality of objects of interest are detected based on the shape of the anchor box from the input image. extracting multi-extension-feature maps of, generating a combined feature map by combining features extracted from each multi-extension-feature map, and detecting the object of interest by estimating an offset from the combined feature map. It includes steps to

Meanwhile, the step of extracting the multi-extension-feature map may include extracting the multi-extension-feature map by performing an extended convolution operation having a ratio corresponding to the shape of the anchor box.

In addition, the step of extracting the multiple extension-feature map may include performing the extended convolution operation to which a plurality of convolution kernels corresponding to the aspect ratio of the anchor box are applied.

In addition, generating a combined feature map may include extracting output features by applying a 1x1 convolution operation to the plurality of multi-extension-feature maps, dividing the output features, and combining the divided output features to form a plurality of combinations. It may be to generate a feature map.

The detecting of the object of interest may include predicting the offset by performing a convolution operation by applying a convolution kernel having a fixed size to each of the plurality of combined feature maps.

Further, another embodiment of the present invention may include a computer readable program stored in a computer readable recording medium configured to execute the object of interest detection method.

According to the present invention described above, since multiple extension-feature maps based on various shapes of anchor boxes are extracted from an input image and an object of interest is detected from a combined feature map obtained by combining these features, the object can be more robustly matched to the shape of the object. can be detected.

Therefore, since it is possible to detect an object not detected by the existing object-of-interest detection apparatus, the performance of the object-of-interest detection apparatus can be greatly improved.

1 and 2 are diagrams for explaining a method of detecting an object of interest in a conventional device for detecting an object of interest.

3 is a diagram showing the configuration of an object of interest detection apparatus according to an embodiment of the present invention.

4 is a diagram schematically illustrating an object of interest detection mechanism in the apparatus for detecting an object of interest shown in FIG. 3 .

FIG. 5 is a diagram schematically illustrating a process of extracting a multi-extension-feature map in the multi-extension-feature map extractor shown in FIG. 3 .

FIG. 6 is a diagram schematically illustrating a process of generating a combined feature map in the feature combiner shown in FIG. 3 .

FIG. 7 is a diagram schematically illustrating a process of predicting an offset in the object detection unit shown in FIG. 3 .

8 is a flowchart illustrating a method for detecting an object of interest according to another embodiment of the present invention.

9A and 9B are diagrams illustrating an example of a result of detecting an object of interest according to the method of detecting an object of interest according to FIG. 8 and a conventional method.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in another embodiment without departing from the spirit and scope of the invention in connection with one embodiment. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 and 2 are diagrams for explaining a method of detecting an object of interest in an apparatus for detecting an object of interest.

The object-of-interest detection device is a device for detecting an object of interest on an input image in real time, and may be, for example, a single shot multi-box detector (SSD) or a refine detector.

As shown in FIG. 1, the apparatus for detecting an object of interest includes a plurality of detection heads, and each detection head is based on the difference between the size of an anchor box and the size of a real object (ground truth box). to detect the object of interest. To this end, each detection head predicts a final offset by applying a 3x3 convolution operation collectively to the input feature map, as shown in FIG. 2 .

Meanwhile, anchor boxes used in the object-of-interest detection apparatus are pre-existing boxes in an input image and have various shapes, that is, various sizes and aspect ratios. For example, in the case of an SSD having an input size of 300x300, 8700 anchor boxes may exist in advance. Therefore, there is a need to detect an object of interest in consideration of various shapes of anchor boxes.

Hereinafter, an apparatus for detecting an object of interest based on various shapes of the anchor box will be described.

3 is a diagram showing the configuration of a device for detecting an object of interest according to an embodiment of the present invention, FIG. 4 is a schematic diagram of an object of interest detection mechanism in the device for detecting an object of interest shown in FIG. 3, and FIG. It is a diagram schematically illustrating the process of extracting the multi-extension-feature map in the multi-extension-feature map extraction unit shown in FIG. 3, and FIG. 6 is the process of generating the combined feature map in the feature combining unit shown in FIG. , and FIG. 7 is a diagram schematically illustrating a process of predicting an offset in the object detection unit shown in FIG. 3 .

The object of interest detection apparatus 100 according to the present embodiment is an object of interest detection apparatus 100 that detects an object of interest based on at least one anchor box on an input image, and includes a multiple extension-feature map extractor 110, a feature It includes a coupling unit 120 and an object detection unit 130.

The multi-dilated feature map extractor 110 extracts a plurality of multi-dilated feature maps based on the shape of an anchor box from an input image.

The multi-extension-feature map extractor 110 may extract the multi-extension-feature map by performing an extended convolution operation having a ratio corresponding to the shape of the anchor box.

More specifically, the multiple extension-feature map extractor 110 may perform an extended convolution operation by applying a plurality of convolution kernels corresponding to the aspect ratio of the anchor box.

)을 생성할 수 있다. 먼저, 다중확장-특징맵 추출부(110)는 1:1 의 종횡비를 갖는 앵커박스(빨간색 앵커박스)의 형상을 고려한 컨볼루션 커널(d=(1,1)) 과 입력 특징맵을 3x3 컨볼루션 연산하여 제 1 다중확장-특징맵(

)을 추출한다. 또한, 다중확장-특징맵 추출부는 1:2 의 종횡비를 갖는 앵커박스(노란색 앵커박스)의 형상을 고려한 컨볼루션 커널(d=(1,2)) 과 입력 특징맵을 3x3 컨볼루션 연산하여 제 2 다중확장-특징맵(

)을 추출하며, 2:1 의 종횡비를 갖는 앵커박스(보라색 앵커박스)의 형상을 고려한 컨볼루션 커널(d=(2,1)) 과 입력 특징맵을 3x3 컨볼루션 연산하여 제 3 다중확장-특징맵(

)을 추출한다. 또한, 다중확장-특징맵 추출부는 와이드-스케일(wide-scale) 객체의 특징을 보완하기 위해 컨볼루션 커널(d=(2,2)) 과 입력 특징맵을 3x3 컨볼루션 연산하여 제 4 다중확장-특징맵(

)을 추출한다.Referring to FIG. 5, the multi-expansion-feature map extractor 110 selects a convolution kernel corresponding to an anchor box having an aspect ratio of 1:1, 1:2, or 2:1, and the selected convolution kernel and By performing an extended convolution operation, a plurality of multiple extended-feature maps (

) can be created. First, the multiple extension-feature map extractor 110 performs a 3x3 convolution of a convolution kernel (d=(1,1)) considering the shape of an anchor box (red anchor box) having an aspect ratio of 1:1 and an input feature map. The first multi-extension-feature map (

) is extracted. In addition, the multi-expansion-feature map extractor calculates a convolution kernel (d=(1,2)) considering the shape of an anchor box (yellow anchor box) having an aspect ratio of 1:2 and an input feature map by performing a 3x3 convolution operation. 2 multi-extension-feature map (

) is extracted, and a convolution kernel (d=(2,1)) considering the shape of an anchor box (purple anchor box) having an aspect ratio of 2:1 is calculated by 3x3 convolution of the input feature map, and the third multiple expansion- feature map (

) is extracted. In addition, the multi-expansion-feature map extractor performs a 3x3 convolution operation on the convolution kernel (d = (2,2)) and the input feature map to compensate for the wide-scale object features, and obtains a fourth multiple extension. -Feature map (

) is extracted.

)을 추출하였으나, 이는 일 예일 뿐, 다양한 종횡비를 갖는 앵커박스의 형상을 고려하여 다중확장-특징맵을 추출할 수 있음은 물론이다. On the other hand, in FIG. 5, multiple expansion-feature maps (

) was extracted, but this is only an example, and it is of course possible to extract multiple extension-feature maps in consideration of the shape of anchor boxes having various aspect ratios.

The feature combiner 120 combines features extracted from each multi-extension-feature map to create a combined feature map.

More specifically, the feature combiner 120 extracts output features of the reduced channel by applying a 1x1 convolution operation to a plurality of multiple extension-feature maps, divides the output features, and converts the output features into a combined channel having a preset size. It may be to generate a plurality of combined feature maps by combining them.

)에 1x1 컨볼루션 연산을 적용하여 출력 특징들(

,

,

,

)을 추출한다. 보다 구체적으로, 특징결합부(120)는 제 1 다중확장-특징맵(

)에 1x1 컨볼루션 연산을 적용하여 256 채널의 제 1 출력특징(

)을 추출하고, 제 2 다중확장-특징맵(

)에 1x1 컨볼루션 연산을 적용하여 128 채널의 제 2출력특징(

)을 추출한다. 또한, 특징결합부(120)는 제 3 다중확장-특징맵(

)에 1x1 컨볼루션 연산을 적용하여 128 채널의 제 3 출력특징(

)을 추출하고, 제 4 다중확장-특징맵(

)에 1x1 컨볼루션 연산을 적용하여 256 채널의 제 4 출력특징(

)을 추출한다.Referring to FIG. 6, first, the feature combiner 120 each multiple extension-feature map (

) by applying a 1x1 convolution operation to the output features (

,

,

,

) is extracted. More specifically, the feature combining unit 120 is a first multi-extension-feature map (

) by applying a 1x1 convolution operation to the first output characteristic of 256 channels (

) is extracted, and the second multi-extension-feature map (

) by applying a 1x1 convolution operation to the second output characteristic of 128 channels (

) is extracted. In addition, the feature combining unit 120 is a third multi-extension-feature map (

) by applying a 1x1 convolution operation to the third output characteristic of 128 channels (

) is extracted, and the fourth multi-extension-feature map (

) by applying a 1x1 convolution operation to the fourth output characteristic of 256 channels (

) is extracted.

,

,

,

)을 분할하고 일정크기의 사이즈를 갖는 결합채널로 결합하여 복수의 결합 특징맵(

)을 생성한다. 즉, 특징결합부(120)는 서로 다른 형상을 갖는 앵커박스로부터의 출력특징들이 상호 결합된 결합 특징맵(

)을 생성하는 것이다. 이러한 결합 특징맵(

) 은 아래 수학식 1 과 같이 정의될 수 있다.In addition, the feature combining unit 120 output features extracted in this way (

,

,

,

) are divided and combined into a combination channel having a certain size to form a plurality of combined feature maps (

) to create That is, the feature combining unit 120 is a combined feature map in which output features from anchor boxes having different shapes are mutually combined (

) to generate. These combined feature maps (

) can be defined as in Equation 1 below.

[Equation 1]

Here, N is the number of anchor boxes. Meanwhile, in this embodiment, N is 3 considering three anchor boxes, but is not limited thereto.

) 과 제 4 출력특징들(

)은 분할되어 제 1 결합특징맵(

), 제 2 결합특징맵(

) 및 제 3 결합특징맵(

)의 구성성분이 되고, 제 2 출력특징들(

) 과 제 3 출력특징들(

)은 분할된 제 1 출력특징들(

) 과 제 4 출력특징들(

) 중 적어도 하나 이상과 결합하여 각각 제 2 결합특징맵(

) 및 제 3 결합특징맵(

)으로 생성되는 것이다. That is, the first output features (

) and the fourth output features (

) is divided into a first joint feature map (

), the second joint feature map (

) And the third joint feature map (

), and the second output features (

) and the third output features (

) is the divided first output features (

) and the fourth output features (

) In combination with at least one or more of the second combined feature map (

) And the third joint feature map (

) is created.

The object detection unit 130 detects an object of interest by predicting an offset from the combined feature map.

To this end, the object detector 130 may predict the offset by performing a 3x3 convolution operation by applying a convolution kernel having a fixed size to each of a plurality of combined feature maps. That is, the object detector 130 predicts the final offset by applying the same convolution kernel to each combined feature map in which output features based on anchor boxes of different shapes are mutually combined, and detects the object of interest through this.

8 is a flowchart illustrating a method for detecting an object of interest according to another embodiment of the present invention.

A method of detecting an object of interest according to the present embodiment is a method of detecting an object of interest based on at least one anchor box on an input image in an object of interest detection apparatus, wherein the multi-expansion-feature map extractor detects the anchor box from the input image. Extracting a plurality of multiple extension-feature maps based on the shape of (S10), generating a combined feature map by combining features extracted from each multiple extension-feature map in a feature combining unit (S20), and an object detection unit and detecting the object of interest by estimating an offset from the combined feature map in (S30).

Meanwhile, in the step of extracting the multi-extension-feature map (S10), the multi-extension-feature map may be extracted by performing an extended convolution operation having a ratio corresponding to the shape of the anchor box. . In addition, the step of extracting the multiple extension-feature map (S10) may be to perform the extended convolution operation to which a plurality of convolution kernels corresponding to the aspect ratio of the anchor box are applied.

In addition, in the step of generating a combined feature map (S20), output features are extracted by applying a 1x1 convolution operation to the plurality of multiple extension-feature maps, the output features are divided, the output features are divided, and the divided A plurality of combined feature maps may be generated by combining the output features with each other.

In the step of detecting the object of interest (S30), the offset value may be calculated by performing a convolution operation by applying a convolution kernel having a fixed size to each of the plurality of combined feature maps.

Since other descriptions are the same as those of the object of interest detection device, descriptions thereof are replaced with those of FIGS. 3 to 7 .

9A and 9B are diagrams illustrating an example of a result of detecting an object of interest according to the method of detecting an object of interest according to FIG. 8 and a conventional method.

Specifically, FIG. 9A is a result of detecting an object of interest according to a conventional method, and FIG. 9B is a diagram showing a result of detecting an object of interest according to the method of detecting an object of interest according to the present invention. Referring to FIGS. 9A and 9B, When an object of interest is detected according to the present invention, it is possible to confirm that objects not detected in the existing technique are detected in various ways.

That is, according to the present invention described above, multiple extension-feature maps based on various shapes of anchor boxes are extracted from an input image, and an object of interest is detected from a combined feature map obtained by combining these features, thereby making it more robust to the shape of the object. Object detection may be performed. Therefore, since it is possible to detect an object not detected by the existing object-of-interest detection apparatus, the performance of the object-of-interest detection apparatus can be greatly improved.

An operation by the method of detecting an object of interest according to the above-described embodiments may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. A computer-readable recording medium on which a program for implementing an object-of-interest detection operation according to embodiments is recorded includes all types of recording devices storing data readable by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, computer-readable recording media may be distributed in computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

Although the above has been described with reference to embodiments, it will be understood that those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will be able to.

[Description of code]

110: multi-extension-feature map extraction unit

120: feature coupling unit

130: object detection unit

Claims (11)

An object of interest detection apparatus for detecting an object of interest based on at least one anchor box on an input image, comprising: a multiple extension-feature map extractor extracting a plurality of multiple extension-feature maps based on the shape of the anchor box from the input image; a feature combining unit generating a combined feature map by combining features extracted from each multi-extension-feature map; and and an object detection unit configured to detect the object of interest by estimating an offset from the combined feature map. According to claim 1, The multi-extension-feature map extraction unit, and extracting the multi-extension-feature map by performing an extended convolution operation having a ratio corresponding to the shape of the anchor box. According to claim 2, The multi-extension-feature map extraction unit, and performing the extended convolution operation by applying a plurality of convolution kernels corresponding to the aspect ratio of the anchor box. According to claim 3, The feature combining unit, Extracting output features by applying a 1x1 convolution operation to the plurality of multi-extension-feature maps; and dividing the output features and combining the divided output features to generate a plurality of combined feature maps. According to claim 4, The object detection unit, and predicting the offset by performing a convolution operation by applying a convolution kernel having a fixed size to each of the plurality of combined feature maps. A method of detecting an object of interest based on at least one anchor box on an input image in an object of interest detection apparatus, the method comprising: extracting a plurality of multiple expansion-feature maps based on the shape of the anchor box from the input image; generating a combined feature map by combining features extracted from each multi-extension-feature map; and and detecting the object of interest by estimating an offset from the combined feature map. According to claim 6, The step of extracting the multi-extension-feature map, and extracting the multiple extension-feature map by performing an extended convolution operation having a ratio corresponding to the shape of the anchor box. According to claim 7, The step of extracting the multi-extension-feature map, and performing the extended convolution operation by applying a plurality of convolution kernels corresponding to the aspect ratio of the anchor box. According to claim 8, In the step of generating the combined feature map, Extracting output features by applying a 1x1 convolution operation to the plurality of multi-extension-feature maps; and generating a plurality of combined feature maps by dividing the output features and combining the divided output features. According to claim 9, The step of detecting the object of interest is, and predicting the offset by performing a convolution operation by applying a convolution kernel having a fixed size to each of the plurality of combined feature maps. A computer readable program stored in a computer readable recording medium, configured to execute the object of interest detection method according to claim 6 .

Images