WO2025146712A1 - Object detection device and method - Google Patents

Abstract

In the present invention: an estimation unit (22) estimates a density map of a target image serving as a target of object detection, by using a density estimation model (30) which is generated in advance by machine learning to estimate a density map indicating the density of the probability that an object is present at positions in an image; an extraction unit (24) extracts a region serving as a target of object detection, on the basis of the probability density in the estimated density map (30); a detection unit (26), by applying an object detection model (32) which is generated in advance by machine learning in order to detect an object from an image, detects an object in the region of the target image corresponding to the extracted region; and a processing unit (28) performs processing for setting the size of the region to be extracted according to the density map (30) or the object detection result.

Description

Object detection device and method

The disclosed technology relates to an object detection device and an object detection method.

An object detection device is a device that estimates the class of objects, such as people and cars, contained in an input image, the coordinate information of the rectangular bounding box that surrounds the object area in the image, and the reliability of the detection result. In recent years, several object detection models using deep learning have been proposed. As object detection models based on deep learning, YOLO (You Only Look Once) and RetinaNet, which infer the bounding box and object class at the same time, have been proposed (Non-Patent Documents 1 and 2). In addition, R-CNN, which detects candidate object areas and classifies the object classes separately, and an improved version of this, Faster R-CNN, have been proposed (Non-Patent Documents 3 and 4).

In addition, when performing object detection from high-definition images or videos such as 4K (3840 x 2160 pixels) or 8K (7680 x 4320 pixels), the input image size to the object detection model is limited to a maximum of 1536 x 1536 pixels in YOLO v5, for example. Therefore, high-definition images cannot be input to the object detection model at their original size. Therefore, a method has been proposed in which the input high-definition image is divided to match the input image size of the object detection model, the results of object detection from each divided image are aggregated, and the result of object detection from the entire image is output. As this method, several methods have been proposed depending on the image division method. For example, a method has been proposed in which a group of images divided evenly to match the input image size of the object detection model and an image reduced in size as a whole are input to the object detection model, respectively. In this method, the coordinate information of the obtained bounding box is scaled, and the detection results of each divided image and the reduced image are synthesized, and the final result is output (Non-Patent Document 5).

　Also proposed are methods that use probability density estimation or cluster detection to estimate the distribution of areas in an image where objects exist, and then cut out only the areas where objects are predicted to exist and apply an object detection model (Non-Patent Documents 6 and 7). For example, in the method of Non-Patent Document 6, a high-definition image that is the input image is first reduced, and a density map that estimates the distribution of areas in the image where objects exist is output. Next, based on the output density map, a rectangular area in the image to which object detection is applied is extracted. At this time, a single threshold value for the probability density of an object that is deemed to exist is set in advance, and the area where the probability density is equal to or greater than the threshold value is extracted as a rectangular area. Finally, a portion corresponding to the extracted rectangular area is cut out from the input image, and an object detection model is applied to obtain a detection result.

J. Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788. T. -Y. Lin et al., “Focal Loss for Dense Object Detection,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999-3007. R. Girshick et al., “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation,” 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580-587. S. Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017. H. Uzawa et al., "High-definition object detection technology based on AI inference scheme and its implementation", IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 20210323. C. Li et al., “Density Map Guided Object Detection in Aerial Images,” 2020 IEEE/CVF CVPRW, 2020, pp. 737-746. F. Yang et al., “Clustered Object Detection in Aerial Images,” 2019 IEEE/CVF ICCV, 2019, pp. 8310-8319.

In a method such as the one described in Non-Patent Document 5, which divides a high-resolution image evenly and performs object detection, it is not possible to selectively apply object detection to each divided image. Therefore, when the number of divided images increases as the resolution of the input image increases, the amount of calculations required for object detection increases. On the other hand, in methods based on cluster detection and probability density estimation such as the methods described in Non-Patent Documents 6 and 7, the area in the image to which object detection is applied is narrowed down, so that depending on the image, it may be possible to significantly reduce the amount of calculations.

However, when performing probability density estimation, a high-density distribution with low variance is generally obtained from areas on an image where small objects are densely packed, while a low-density distribution with high variance is obtained from areas where large objects are present. When extracting rectangular areas using a single threshold as in conventional methods, if the threshold is set high to match small objects, a rectangular area corresponding to a large object may not be extracted, or if it is extracted, a small rectangular area in which only a part of the large object is cut out may be extracted. On the other hand, if the threshold is set low to match large objects, the size of the rectangular area may be larger than the input size of the object detection model, and the cut-out image must be reduced when input to the object detection model, which may cause small objects to be crushed and not detectable. In this way, when extracting areas above a single threshold from the results of probability density estimation as areas to be input to the object detection model, it is not possible to extract an optimal size area, and the accuracy of object detection may deteriorate.

The disclosed technology has been developed in consideration of the above points, and aims to cut out an area from a target image of an optimal size according to the scene in the target image as an area to which an object detection model is applied.

A first aspect of the present disclosure is an object detection device that includes an estimation unit that estimates a density map of a target image that is the subject of object detection using a density estimation model previously generated by machine learning so as to estimate a density map indicating the probability density of an object being present at each position of the image, an extraction unit that extracts a region that is the subject of object detection based on the probability density in the density map estimated by the estimation unit, a detection unit that detects an object by applying an object detection model previously generated by machine learning to detect an object from an image to a region of the target image that corresponds to the region extracted by the extraction unit, and a processing unit that performs processing to set the size of the region extracted by the extraction unit according to the density map or the object detection result by the detection unit.

A second aspect of the present disclosure is an object detection method, in which an estimation unit estimates a density map of a target image to be subjected to object detection using a density estimation model previously generated by machine learning so as to estimate a density map indicating the probability density of the presence of an object at each position of the image, an extraction unit extracts a region to be subjected to object detection based on the probability density in the density map estimated by the estimation unit, a detection unit detects an object by applying an object detection model previously generated by machine learning to detect an object from an image to a region of the target image corresponding to the region extracted by the extraction unit, and a processing unit performs processing to set the size of the region to be extracted by the extraction unit according to the density map or the object detection result by the detection unit.

The disclosed technology makes it possible to extract an area of an optimal size from a target image according to the scene in the target image as an area to which an object detection model is applied.

FIG. 2 is a block diagram showing a hardware configuration of the object detection device. 1 is a block diagram showing a functional configuration of an object detection device according to a first embodiment. FIG. 13 is a diagram showing pairs of thresholds and sizes of rectangular areas. FIG. 13 is a diagram for explaining extraction of a rectangular area. FIG. 13 is a diagram for explaining merging of rectangular regions. FIG. 13 is a diagram illustrating an overview of updating a threshold value. FIG. 13 is a diagram illustrating an overview of updating a threshold value. 11A and 11B are diagrams for explaining adjustment of the size of a rectangular area. 5 is a flowchart illustrating an example of an object detection process according to the first embodiment. 13 is a flowchart illustrating an example of a threshold adjustment process. 13 is a flowchart illustrating an example of a size adjustment process. FIG. 11 is a block diagram showing a functional configuration of an object detection device according to a second embodiment. FIG. 13 is a diagram for explaining extraction of a rectangular region based on a maximum value. FIG. 13 is a diagram for explaining a search for a maximum value. 13 is a flowchart illustrating an example of an object detection process according to the second embodiment.

Below, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

First Embodiment
Fig. 1 is a block diagram showing a hardware configuration of an object detection device 10 according to the first embodiment. As shown in Fig. 1, the object detection device 10 has a central processing unit (CPU) 11, a read only memory (ROM) 12, a random access memory (RAM) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I/F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores an object detection program for executing the object detection process described below.

ROM 12 stores various programs and data. RAM 13 temporarily stores programs or data as a working area. Storage 14 is made up of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), and stores various programs including the operating system, and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs. The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system. The communication I/F 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

Next, the functional configuration of the object detection device 10 according to the first embodiment will be described. FIG. 2 is a block diagram showing an example of the functional configuration of the object detection device 10. As shown in FIG. 2, the object detection device 10 includes, as its functional configuration, an estimation unit 22, an extraction unit 24, a detection unit 26, and a processing unit 28. Furthermore, a density estimation model 30 and an object detection model 32 are stored in a predetermined storage area of the object detection device 10. Each functional configuration is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

The density estimation model 30 is a machine learning model that has been generated in advance by machine learning so as to estimate a density map that indicates the probability density of the presence of an object at each position in an image. Specifically, the density estimation model 30 is generated by using an image and ground-truth data of the density map for that image as training data, and updating the parameters of the density estimation model 30 so that the density map output from the density estimation model 30 approaches the ground-truth data.

The estimation unit 22 uses the density estimation model 30 to estimate a density map of a target image that is the subject of object detection. Specifically, the estimation unit 22 sequentially acquires each frame of the video input to the object detection device 10 as a target image. The estimation unit 22 inputs the acquired target image to the density estimation model 30 and acquires a density map output from the density estimation model 30. The estimation unit 22 passes the estimated density map to the extraction unit 24.

Based on the probability density in the density map passed from the estimation unit 22, the extraction unit 24 extracts a region to be subjected to object detection using a pair of a multi-stage threshold value and a rectangular region size supplied from the processing unit 28 described later. The threshold value is a value for identifying a portion in the density map where the probability density is equal to or greater than the threshold value, and the size of the rectangular region is a size determined so as to correspond one-to-one with the threshold value of each stage. In this embodiment, the region is a rectangle, and the length of one side of the rectangular region (number of pixels) is the size of the rectangular region. Also, as shown in FIG. 3, the number of pairs is N (any natural number), the threshold value is T _i , and the length of one side of the rectangular region corresponding to the threshold value T _i is S _i , where i=1, 2, ..., N, T ₁ > T ₂ > ... > T _N , and S ₁ < S ₂ < ... < S _N .

4, the extraction unit 24 extracts a rectangular region of size S _i as a target region for object detection, centered on a portion of a series of pixels whose probability density is equal to or greater than a threshold value T _i in the density map estimated by the estimation unit 22. The extraction unit 24 extracts a rectangular region for each threshold value T _i (i=1, 2, ..., N).

In addition, when the extraction unit 24 extracts multiple rectangular regions from the target image, it merges rectangular regions that have an overlapping degree equal to or greater than a predetermined value and for which the minimum size of a detectable object satisfies a predetermined condition relative to the size of the rectangular region after merging.

Specifically, as shown in the upper diagram of FIG. 5, a rectangular area i extracted based on a threshold T _i and a rectangular area j extracted based on a threshold T _j have an overlapping area (shaded area in FIG. 5). The extraction unit 24 calculates, for example, IoU (Intersection over Union) as the overlapping degree. Note that IoU=area of the overlapping area between rectangular area i and rectangular area j/area of the union area of rectangular area i and rectangular area j. When the overlapping degree is equal to or greater than a predetermined value T _iou , the extraction unit 24 sets a rectangular area including rectangular area i and rectangular area j as a merged rectangular area, as shown in the lower diagram of FIG. 5. By merging rectangular areas in this way, it is possible to reduce the area where object detection is performed overlappingly when the object detection model 32 is applied, and reduce the amount of calculation for object detection.

However, the extraction unit 24 does not merge rectangular regions when the following formula (1) is satisfied. Note that a _min ⁱ and a _min ^j are the minimum size of an object detected from an image corresponding to a rectangular region extracted based on each threshold value T _i and T _j in the immediately preceding frame, A _ij is the area of the rectangular region after merging, and A _min is the minimum size of an object detectable by the object detection model 32. This makes it possible to merge rectangular regions while preventing an object from being missed from object detection due to being smaller than the detectable size.

The extraction unit 24 assigns an ID, which is identification information, to each of the extracted rectangular areas, and passes rectangular information including the ID, the position of the rectangular area, and the size (length and width) of the rectangular area to the extraction unit 24 and the detection unit 26. The position of the rectangular area may be the coordinates of a specified point of the rectangular area, such as the center coordinates of the rectangular area or the coordinates of the upper left corner.

The object detection model 32 is a machine learning model that has been generated in advance by machine learning in order to detect objects from images. The object detection model 32 outputs metadata including the class of the object contained in the input image, a bounding box indicating the area of the object, and the confidence of the detection.

The detection unit 26 detects an object by applying the object detection model 32 to an area of the target image corresponding to the rectangular area extracted by the extraction unit 24. Specifically, the detection unit 26 identifies an area on the target image corresponding to the rectangular area extracted by the extraction unit 24 based on the rectangular information passed from the extraction unit 24. The detection unit 26 cuts out an image of the identified area from the target image, enlarges or reduces the cut-out image to match the input size of the object detection model 32, and inputs it to the object detection model 32. The detection unit 26 outputs the metadata output by the object detection model 32 as the detection result, and passes the detection result to the processing unit 28.

The processing unit 28 performs processing for setting the size of the rectangular area to be extracted by the extraction unit 24 according to the density map estimated by the estimation unit 22 or the object detection result by the detection unit 26. Specifically, the processing unit 28 acquires initial values of the pair of the above-mentioned multi-stage threshold value T _i and the rectangular area size S _i . The initial values may be set arbitrarily. When the target image is the first frame of the video, the processing unit 28 supplies the pair of initial values T _i and S _i to the extraction unit 24. When the target image is the second or subsequent frame, the processing unit 28 adjusts the threshold value T _i and the rectangular area size S _i based on the density map or the object detection result for the frame before the target image in the video. Then, the processing unit 28 supplies the adjusted pair of T _i and S _i to the extraction unit 24.

The adjustment of the threshold T _i will be described in more detail. When the area of a portion in the density map where pixels having a probability density equal to or greater than the threshold T _i are consecutive is smaller than the area (S _i ×S _i ) corresponding to the size S _i paired with the threshold T _i , the processing unit 28 lowers the threshold T _i by a predetermined value or a predetermined percentage. Also, when the area of a portion in the density map where pixels having a probability density equal to or greater than the threshold T _i are consecutive is larger than the area (S _i ×S _i ) corresponding to the size S _i , the processing unit 28 raises the threshold T _i by a predetermined value or a predetermined percentage.

For example, the processing unit 28 calculates an average area _ai of a portion in a certain frame that is identified by the extraction unit 24 as a portion whose probability density is equal to or greater than a threshold value T _i , and updates the threshold value T _i according to the following equation (2) using the calculated average area _ai and a constant α (0<α<1).

6 shows an overview of updating the threshold T _i when S _i ×S _i > a _i . The lower the threshold, the larger the area of the part where the probability density is equal to or greater than the threshold, and the higher the threshold, the smaller the area of the part where the probability density is equal to or greater than the threshold. Therefore, when S _i ×S _i > a _i , the processing unit 28 updates the threshold so as to expand the part where the probability density is equal to or greater than the threshold by lowering the threshold T _i .

7 shows an overview of updating the threshold T _i when S _i ×S _i ≦a _i . When S _i ×S _i ≦a _i , the processing unit 28 increases the threshold T _i . As a result, the processing unit 28 adjusts the threshold so as to extract an area with a high probability density where smaller objects are distributed using a pair of the threshold T _i and the size S _i . In addition, a pair of the threshold T _i+1 or a threshold lower than that and a size associated with that threshold becomes a threshold for extracting an area where objects larger than the object targeted by the threshold T _i are distributed.

When the adjacent thresholds T _k and T _k+1 become |T _k -T _k+1 |<ε (ε is any positive number) by adjusting the thresholds, the processing unit 28 integrates these thresholds. For example, when lowering T _i makes T _i approach T i+ ₁ and |T _i -T _i+1 |<ε, or when raising T _i makes T _i approach T _i-1 and |T _i-1 -T _i |<ε. Specifically, as the integration of the thresholds, the processing unit 28 makes a new pair of the threshold T _k and the size S _k+1 , and abolishes the threshold T _k+1 and the size S _k .

Next, the adjustment of the size of the rectangular region will be described in more detail. The processing unit 28 classifies the size of the object detected from the region of size S _i in the object detection result by the detection unit 26 for the previous frame into a plurality of categories with different sizes. The processing unit 28 enlarges the size S _i by a predetermined percentage when the number of objects included in the category of large object sizes is greater than the category of object sizes corresponding to the average. The processing unit 28 also reduces the size S _i by a predetermined percentage when the number of objects included in the category of small object sizes is greater than the category of object sizes corresponding to the average.

For example, the processing unit 28 classifies the size of the bounding box of an object detected in a certain frame into three categories, S (Small), M (Medium), and L (Large), based on the lower limit of the object size that can be detected by the object detection model 32. The processing unit 28 may classify the size based on the length of one side of the object's bounding box, or may classify the size based on the area of the bounding box.

Furthermore, the processing unit 28 adjusts the size S _i so that the object of the most numerous category among all objects detected from the image cut out from the target image based on the rectangular region of size S _i becomes the M category, which is an example of a category where the size of the object corresponds to the average. This is because, when the S category is the most numerous, there is a possibility that a smaller object that has not been detected by the object detection model 32 is present in the cut-out image. Also, when the L category is the most numerous, there is a possibility that the object is larger than size S _i and the whole object has not been detected as one object from the cut-out image. The processing unit 28 performs this size adjustment process for all sizes S _i (i = 1, 2, ..., N). At this time, the processing unit 28 does not adjust the size S _i when there is no image cut out at size S _i or when the category of the most numerous objects detected from the image cut out at size S _i is the M category.

A more detailed description will be given of the case where the area of the bounding box is used as the classification criterion. The processing unit 28 acquires an upper limit _Amax and a lower limit _Amin of the area of the bounding box of an object detectable by the object detection model 32. The processing unit 28 also determines an average area _AM of the bounding boxes of all objects that are expected based on the values of the upper limit _Amax and the lower limit _Amin of the area. For example, _AM may be set as ( _Amax + _Amin )/2.

Furthermore, the processing unit 28 classifies the detected object into S, M, or L using a predetermined criterion based on the detection result of the detection unit 26 in each frame. For example, if the area _aBbox of the bounding box is _Amin ≦ _aBbox <X, the object may be classified as S category, if X≦ _aBbox <2X, the object may be classified as M category, and if 2X≦ _aBbox ≦ _Amax , the object may be classified as L category. X may be, for example, ( _Amax − _Amin )/3.

The processing unit 28 defines the average areas of the bounding boxes of the objects belonging to each category as a _S , a _M , and a _L , respectively, and defines the maximum area of the detected object regardless of the category as a _max and the minimum area as a _min . Using these values, the size S _i is updated by the following formula (3).

However, as mentioned above, when the M category contains the most objects, the size of the rectangular region is not adjusted. Figure 8 shows an example of adjusting the size of the rectangular region using the above formula (3). As shown in Figure 8(a), when the L category contains the most objects, the size of the rectangular region is increased to capture the entire image of the object and make it easier to detect. On the other hand, as shown in Figure 8(b), when the S category contains the most objects, the size of the rectangular region is reduced to prevent small objects from being missed. This makes it possible to appropriately adjust the size of the rectangular region within a range in which the detected object does not exceed the upper or lower limits of the size of a detectable object.

Next, the operation of the object detection device 10 according to the first embodiment will be described. FIG. 9 is a flowchart showing the flow of the object detection process performed by the object detection device 10. The object detection process is performed by the CPU 11 reading out an object detection program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Note that the object detection process is an example of the object detection method of the present disclosure.

In step S10, the CPU 11, functioning as the estimation unit 22, acquires a frame of the video input to the object detection device 10 as a target image. Next, in step S12, the CPU 11, functioning as the estimation unit 22, estimates a density map of the target image using the density estimation model 30.

Next, in step S14, the CPU 11, functioning as the extraction unit 24, extracts a rectangular region to be subjected to object detection based on the probability density in the density map estimated in step S12 above, using a pair of multi-stage threshold value T _i and size S _i supplied from the processing unit 28. Note that the pair of multi-stage T _i and S _i supplied from the processing unit 28 here is a pair of initial values T _i and S _i when the target image is the first frame of a video. Also, when the target image is the second or subsequent frame, it is a pair of T _i and S _i after adjustment in a threshold adjustment process and a size adjustment process to be described later.

Next, in step S16, the CPU 11, as the extraction unit 24, merges rectangular areas in the target image that have an overlapping degree equal to or greater than a predetermined value and where the minimum size of a detectable object satisfies a predetermined condition relative to the size of the rectangular area after merging. The CPU 11, as the extraction unit 24, assigns an ID, which is identification information, to each of the extracted rectangular areas, and passes rectangular information including the ID, the position of the rectangular area, and the size of the rectangular area to the extraction unit 24 and the detection unit 26.

Next, in step S18, the CPU 11, as the detection unit 26, identifies an area on the target image that corresponds to the rectangular area extracted by the extraction unit 24, based on the rectangular information passed from the extraction unit 24. The CPU 11, as the detection unit 26, cuts out an image of the identified area from the target image, enlarges or reduces the cut-out image to match the input size of the object detection model 32, and inputs it to the object detection model 32. The CPU 11, as the detection unit 26, then obtains metadata that is the object detection result output by the object detection model 32.

Next, in step S20, the CPU 11, functioning as the detection unit 26, outputs the object detection result and passes the detection result to the processing unit 28. Next, in step S22, the CPU 11, functioning as the processing unit 28, determines whether the target image is the last frame of the video. If it is the last frame, the object detection process ends, and if there are subsequent frames, the process proceeds to step S30.

In step S30, the threshold adjustment process is executed. Here, the threshold adjustment process is explained with reference to FIG. 10.

In step S32, the CPU 11, functioning as the processing unit 28, sets the variable i to 1. Next, in step S34, the CPU 11, functioning as the processing unit 28, calculates the average area _ai of the portions of the target image that are identified by the extraction unit 24 in the density map as portions whose probability density is equal to or greater than the threshold value T _i . Then, the CPU 11, functioning as the processing unit 28, determines whether or not S _i ×S _i ≦a _i . If _{S i} ×S _i ≦a _i , the process proceeds to step S36, and if S _i ×S _i >a _i , the process proceeds to step S38.

In step S36, the CPU 11, functioning as the processing unit 28, increases the threshold T _i by a predetermined value or a predetermined ratio. On the other hand, in step S38, the CPU 11, functioning as the processing unit 28, decreases the threshold _{T i} by a predetermined value or a predetermined ratio. Next, in step S40, the CPU 11, functioning as the processing unit 28, determines whether or not |T _k -T _{k+1 |} <ε for the adjacent thresholds T k and T k+1. If |T _k -T _k+1 |<ε, the process proceeds to step S42, and if |T _k -T _k+1 |≧ε, the process proceeds to step S44.

In step S42, the CPU 11, as the processing unit 28, combines the threshold T _k and the size S _k+1 into a new pair, and abolishes the threshold T _k+1 and the size S _k , thereby integrating the threshold T _k and the threshold T _k+1 . Next, in step S44, the CPU 11, as the processing unit 28, determines whether the variable i has reached N, which is the number of pairs of thresholds and sizes of rectangular areas. If i<N, the process proceeds to step S46, and the CPU 11, as the processing unit 28, increments i by 1 and returns to step S34. On the other hand, if i=N, the threshold adjustment process ends, and the process returns to the object detection process (FIG. 9).

Next, in step S50, the size adjustment process is executed. Now, the size adjustment process will be described with reference to FIG. 11.

In step S52, the CPU 11, as the processing unit 28, classifies the size of the bounding box of the object detected from the target image into three categories, S, M, and L, based on the lower limit of the object size that can be detected by the object detection model 32. Next, in step S54, the CPU 11, as the processing unit 28, sets the variable i to 1.

Next, in step S56, the CPU 11, functioning as the processing unit 28, determines whether or not a rectangular area extracted with size S _i exists from the density map of the target image. If a rectangular area extracted with size S _i exists, the process proceeds to step S58, and if not, the process proceeds to step S66.

In step S58, the CPU 11, functioning as the processing unit 28, determines whether the most common category among all objects detected from the image cut out from the target image based on the rectangular region of size S _i is category L. If it is category L, the process proceeds to step S60, where the CPU 11, functioning as the processing unit 28, enlarges size S _i by a predetermined percentage. On the other hand, if the most common category is not category L, the process proceeds to step S62.

In step S62, the CPU 11, functioning as the processing unit 28, determines whether the most prevalent category of all objects detected from the image cut out from the target image based on the rectangular region of size S _i is the S category. If it is the S category, the process proceeds to step S64, and the CPU 11, functioning as the processing unit 28, reduces the size S _i by a predetermined ratio. On the other hand, if the most prevalent category is not the S category, i.e., if the most prevalent category is the M category, the process proceeds to step S66.

In step S66, the CPU 11, functioning as the processing unit 28, determines whether the variable i has reached N, which is the number of pairs of a threshold value and the size of a rectangular area. If i<N, the process proceeds to step S68, and the CPU 11, functioning as the processing unit 28, increments i by 1 and returns to step S56. On the other hand, if i=N, the size adjustment process ends and the process returns to the object detection process (Figure 9). Then, the process returns to step S10, and the next frame of the video is acquired as the target image.

In the above object detection process, the threshold adjustment process in step S30 and the size adjustment process in step S50 may be performed first. Also, only one of them may be performed.

As described above, the object detection device according to the first embodiment estimates a density map of a target image to be subjected to object detection using a density estimation model previously generated by machine learning so as to estimate a density map indicating the probability density of an object being present at each position of the image. The object detection device also extracts a region to be subjected to object detection based on the probability density in the estimated density map. The object detection device then detects an object by applying an object detection model previously generated by machine learning for detecting an object from an image to a region of the target image corresponding to the extracted region. Furthermore, the object detection device performs processing for setting the size of the region to be extracted according to the region extraction result or object detection result. Specifically, the object detection device adjusts at least one of the threshold for identifying a portion with a probability density equal to or greater than a threshold and the size of the region paired with the threshold, based on the extraction result or object detection result for a frame prior to the target image in the video.

As a result, the object detection device according to the first embodiment can extract from the input image an area of an optimal size according to the scene of the target image as the area to which the object detection model is applied. This makes it possible to detect objects from high-definition video while suppressing degradation of object detection accuracy due to cutting off of objects or image reduction caused by an inappropriate size of the extracted area.

In addition, in general, when there are multiple pairs of thresholds and the sizes of rectangular areas associated with them, it is difficult to manually adjust these in order to improve the accuracy of object detection. On the other hand, with the object detection device according to the first embodiment, these adjustments can be made automatically, which has the effect of reducing the effort required for adjustment.

In the first embodiment, the size of the bounding box of a detected object is classified into three categories: S, M, and L. This is because three is the minimum number that can accommodate cases where the object size is too large, appropriate, and too small, respectively, but the number of categories may be four or more.

Second Embodiment
Next, a second embodiment will be described. In the object detection device according to the second embodiment, the same components as those in the object detection device 10 according to the first embodiment are denoted by the same reference numerals and detailed description thereof will be omitted. In addition, the hardware configuration of the object detection device according to the second embodiment is similar to the hardware configuration of the object detection device 10 according to the first embodiment shown in FIG. 1, and therefore description thereof will be omitted.

The functional configuration of the object detection device according to the second embodiment will be described. As shown in FIG. 12, the object detection device 210 includes, as its functional configuration, an estimation unit 22, an extraction unit 224, a detection unit 26, and a processing unit 228. A density estimation model 30 and an object detection model 32 are stored in a predetermined storage area of the object detection device 210. Each functional configuration is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

The processing unit 228 searches for a pixel where the probability density is a maximum value in the density map estimated by the estimation unit 22. The processing unit 228 also assumes a normal distribution centered on the pixel where the probability density is a maximum value, and calculates a variance that serves as a threshold value based on the maximum value.

Specifically, as shown in the upper diagram of FIG. 13, the processing unit 228 searches the entire density map for pixels where the probability density is a maximum value. For example, the processing unit 228 may search for the maximum value using a sliding window as shown in FIG. 14. The example in FIG. 14 uses a sliding window of 3×3 pixels. Each square in FIG. 14 represents a pixel in a portion of the density map, and the numbers in each square roughly represent the probability density of each pixel in the density map.

The processing unit 228 compares the size of the central pixel of the sliding window with that of the surrounding eight pixels, and if the probability density of the central pixel is greater than the probability density of any of the other pixels, the probability density is set as the maximum value and the coordinates of that pixel are set as the maximum value coordinates. The processing unit 228 sets the K maximum values found by the search as M _i (i=1, 2, ..., K) and the maximum value coordinates as (x _i , y _i ). Note that the method of searching for the maximum values is not limited to the above example, and any method may be applied.

Furthermore, the processing unit 228 assumes that the objects are distributed in a normal distribution pattern centered on the maximum coordinates, as shown in the middle diagram of Fig. 13. Specifically, the processing unit 228 defines a normal distribution centered on a certain maximum coordinate (x _i , y _i ) as f _i (x, y). In general, assuming that the covariance of the normal distribution f _i (x, y) in the x-axis direction and the y-axis direction is 0 and the variance is σ _i , the normal distribution f i (x, y) is given by the following formula (4):

Since f _i (x _i , y _i )=M _i , the processing unit 228 calculates the variance σ _i of f _i (x, y) by the following formula (5).

The processing unit 228 passes the found maximum value M _i and its coordinates (x _i , y _i ) and the calculated variance σ _i to the extraction unit 224 .

The extraction unit 224 extracts a continuous portion of pixels with a probability density equal to or greater than a threshold value around the maximum coordinates searched for by the processing unit 228. Specifically, the extraction unit 224 extracts a portion that includes the range of variance calculated as the threshold value in the normal distribution assumed by the processing unit 228 as a region to be subjected to object detection.

Specifically, the extraction unit 224 uses the variance _σi and constant T passed from the processing unit 228 to search for a portion of continuous pixels with a probability density of f _i (x _i ± _Tσi , yi ± _Tσi ) (any compound sign) or more, i.e., M _i exp(-T ² ), in the vicinity of the maximum coordinate (x _i , y _i ). The constant T is a positive real number that represents the interval of the variance to be extracted, and is set to an arbitrary value from outside. Then, the extraction unit 224 extracts a rectangular area surrounding the searched portion, as shown in the lower diagram of FIG. 13.

The extraction unit 224 also merges overlapping rectangular areas using a method similar to that used by the extraction unit 24 in the first embodiment. The extraction unit 224 passes rectangular information about the extracted rectangular areas to the detection unit 26.

Next, the operation of the object detection device 210 according to the second embodiment will be described. FIG. 15 is a flowchart showing the flow of the object detection process by the object detection device 210. The object detection process is performed by the CPU 11 reading out an object detection program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. Note that in the object detection process of the second embodiment, the same steps as those in the object detection process of the first embodiment are given the same step numbers and detailed descriptions are omitted.

After steps S10 and S12, in step S200, the CPU 11, as the processing unit 228, searches for pixels whose probability density is a maximum value from the entire density map estimated in step S12. The CPU 11, as the processing unit 228, defines the K maximum values found by the search as M _i (i=1, 2, ..., K) and the maximum value coordinates as (x _i , y _i ).

Next, in step S202, the CPU 11, as the processing unit 228, assumes a normal distribution _f (x, y) centered on the maximum value coordinate, and calculates the variance _σi of _f (x, y) from the maximum value _M. The CPU 11, as the processing unit 228, passes the searched maximum value _M and maximum value coordinates ( _x , _y ), and the calculated variance _σi to the extraction unit 224.

Next, in step S204, the CPU 11, functioning as the extraction unit 224, searches for a portion of continuous pixels with a probability density equal to or greater than M _i exp(-T ² ) around the maximum coordinate (x _i , y _{i )} using the variance σ i and constant T passed from the processing unit 228. Then, the CPU 11, functioning as the extraction unit 224, extracts a rectangular area surrounding the portion found, and passes rectangular information about the extracted rectangular area to the detection unit 26.

Next, after steps S16 to S20, in the next step S222, the CPU 11, as the processing unit 228, determines whether the target image is the last frame of the video. If it is the last frame, the object detection process ends, and if there are subsequent frames, the process returns to step S10.

As described above, the object detection device according to the second embodiment searches for the maximum coordinate in the density map where the probability density is a maximum value, and assumes a normal distribution centered on this maximum coordinate. The object detection device also calculates the variance of the assumed normal distribution based on the maximum value, and extracts a rectangular area that encloses the part where the probability density is equal to or greater than a threshold determined based on the variance. This makes it possible to extract a rectangular area of a size according to the density map, and as with the first embodiment, it is possible to cut out an area of an optimal size according to the scene in the target image from the input image as an area to which the object detection model is applied.

In addition, in the second embodiment, by using the coordinates of the searched maximum values, it is possible to extract a rectangular area that matches the size of the object using only a constant that specifies the range of variance, without using a threshold value and the size of the rectangular area as in the first embodiment. Therefore, the burden of setting and tuning the constants is significantly reduced.

In the above embodiments, a rectangular area is extracted by the extraction unit, but areas of other shapes may be extracted. In this case, when cutting out the corresponding area from the target image, or when inputting the cut-out image to the object detection model, it is necessary to process the image into a shape and size that can be input to the object detection model.

In addition, the object detection process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacture, and dedicated electric circuits such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed specifically to execute specific processes. The object detection process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

In addition, in each of the above embodiments, the object detection processing program is described as being pre-stored (installed) in the ROM 12 or storage 14, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

The following notes are further provided with respect to the above embodiment.

(Additional note 1)
an estimation unit that estimates a density map of a target image that is a target for object detection, using a density estimation model that has been generated in advance by machine learning, so as to estimate a density map that indicates a probability density of an object being present at each position in the image;
an extraction unit that extracts a region to be subjected to object detection based on the probability density in the density map estimated by the estimation unit;
a detection unit that detects an object by applying an object detection model that has been generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit, in order to detect an object from an image;
a processing unit that performs processing for setting a size of an area to be extracted by the extraction unit according to the density map or a result of object detection by the detection unit;
An object detection device comprising:

(Additional note 2)
The object detection device described in Appendix 1, wherein the processing unit, when each frame of the input video is the target image, adjusts at least one of the threshold for identifying a portion where the probability density is equal to or greater than a threshold, and the size of the region paired with the threshold, based on the density map for a frame in the video prior to the target image, or the object detection result by the detection unit.

(Additional note 3)
The object detection device described in Appendix 2, wherein the processing unit lowers the threshold by a predetermined value or a predetermined percentage when the area of a portion of consecutive pixels whose probability density is equal to or greater than the threshold is smaller than an area corresponding to the size of the region paired with the threshold, and raises the threshold by a predetermined value or a predetermined percentage when the area of the portion is larger than an area corresponding to the size of the region paired with the threshold.

(Additional note 4)
The object detection device described in Appendix 2, wherein the processing unit classifies the sizes of objects detected from a first size area in the object detection result by the detection unit for the previous frame into multiple different categories based on size, and performs at least one of a process of enlarging the first size by a predetermined percentage when the number of objects included in the category where the object size is large is greater than the category where the object size corresponds to the average, and a process of reducing the first size by a predetermined percentage when the number of objects included in the category where the object size is small is greater than the category where the object size corresponds to the average.

(Additional note 5)
the processing unit searches for a pixel where the probability density has a maximum value in the density map estimated by the estimation unit, assumes a normal distribution centered on the pixel where the probability density has a maximum value, and calculates a variance of the normal distribution based on the maximum value;
The extraction unit extracts a portion including the range of variance in the normal distribution around the pixel searched by the processing unit as a region to be subjected to the object detection.
2. The object detection device according to claim 1.

(Additional note 6)
The object detection device according to any one of Supplementary Items 1 to 5, wherein when multiple regions are extracted, the extraction unit merges regions that have an overlap degree equal to or greater than a predetermined value and for which the minimum size of a detectable object satisfies a predetermined condition relative to the size of the region after merging.

(Supplementary Note 7)
The estimation unit estimates a density map of a target image to be subjected to object detection using a density estimation model generated in advance by machine learning so as to estimate a density map indicating a probability density of an object being present at each position of the image;
an extraction unit extracts a region to be subjected to object detection based on the probability density in the density map estimated by the estimation unit;
a detection unit detects an object by applying an object detection model generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit, in order to detect an object from an image;
a processing unit performs processing for setting a size of an area to be extracted by the extraction unit according to the density map or a result of object detection by the detection unit;
Object detection methods.

(Supplementary Note 8)
Computer,
an estimation unit that estimates a density map of a target image that is a target for object detection, using a density estimation model that has been generated in advance by machine learning, so as to estimate a density map indicating a probability density of an object being present at each position in the image;
an extraction unit that extracts a region to be subjected to object detection based on the probability density in the density map estimated by the estimation unit;
a detection unit that detects an object by applying an object detection model that has been generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit in order to detect an object from an image; and
a processing unit that performs processing for setting a size of an area to be extracted by the extraction unit according to the density map or a result of object detection by the detection unit;
An object detection program to act as a.

(Supplementary Note 9)
Memory,
at least one processor coupled to the memory;
Including,
The processor,
Estimating a density map of a target image to be subjected to object detection using a density estimation model previously generated by machine learning so as to estimate a density map indicating a probability density of an object being present at each position of the image;
Extracting a region to be subjected to object detection based on the probability density in the estimated density map;
Detecting an object by applying an object detection model that has been generated in advance by machine learning to detect an object from an image to a region of the target image corresponding to the extracted region;
performing a process for setting a size of an area to be extracted according to the density map or an object detection result;
The object detection device is configured as follows.

(Supplementary Note 10)
A non-transitory recording medium storing a program executable by a computer to perform an object detection process,
The object detection process includes:
Estimating a density map of a target image to be subjected to object detection using a density estimation model previously generated by machine learning so as to estimate a density map indicating a probability density of an object being present at each position of the image;
Extracting a region to be subjected to object detection based on the probability density in the estimated density map;
Detecting an object by applying an object detection model that has been generated in advance by machine learning to detect an object from an image to a region of the target image corresponding to the extracted region;
performing a process for setting a size of an area to be extracted according to the density map or an object detection result;
Non-transitory recording media, including

10, 210 Object detection device 11 CPU
12 ROM
13 RAM
14 Storage 15 Input unit 16 Display unit 17 Communication I/F
19 Bus 22 Estimation unit 24, 224 Extraction unit 26 Detection unit 28, 228 Processing unit 30 Density estimation model 32 Object detection model

Claims (4)

an estimation unit that estimates a density map of a target image that is a target for object detection, using a density estimation model that has been generated in advance by machine learning, so as to estimate a density map that indicates a probability density of an object being present at each position in the image;
an extraction unit that extracts a region to be subjected to object detection based on the probability density in the density map estimated by the estimation unit;
a detection unit that detects an object by applying an object detection model that has been generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit, in order to detect an object from an image;
a processing unit that performs processing for setting a size of an area to be extracted by the extraction unit according to the density map or a result of object detection by the detection unit;
An object detection device comprising: The object detection device according to claim 1, wherein the processing unit adjusts at least one of the threshold for identifying a portion where the probability density is equal to or greater than a threshold and the size of the region paired with the threshold, based on the density map for a frame in the input video prior to the target image or the object detection result by the detection unit, when each frame of the input video is the target image. the processing unit searches for a pixel where the probability density has a maximum value in the density map estimated by the estimation unit, assumes a normal distribution centered on the pixel where the probability density has a maximum value, and calculates a variance of the normal distribution based on the maximum value;
The extraction unit extracts a portion including the range of variance in the normal distribution around the pixel searched by the processing unit as a region to be subjected to the object detection.
The object detection device according to claim 1 . The estimation unit estimates a density map of a target image to be subjected to object detection using a density estimation model generated in advance by machine learning so as to estimate a density map indicating a probability density of an object being present at each position of the image;
an extraction unit extracts a region to be subjected to object detection based on the probability density in the density map estimated by the estimation unit;
a detection unit detects an object by applying an object detection model generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit, in order to detect an object from an image;
a processing unit performs processing for setting a size of an area to be extracted by the extraction unit according to the density map or a result of object detection by the detection unit;
Object detection methods.