JP7749437B2 - Image processing device, imaging device, and image processing method - Google Patents

Description

This invention relates to technology for re-detecting the detection area of a tracking target from a captured image.

Object detection processing, which detects objects from images, is being applied to the functions of imaging devices such as digital cameras. Until now, object detection processing has often been performed on objects of specific categories, such as human faces and facial organs (eyes, nose, mouth), or the entire human body. In recent years, with the development of deep learning, technology has been realized that can detect objects of unspecified categories (hereinafter referred to as unspecified objects), such as animals and vehicles, by learning object-likeness using information on objects of various categories.

In digital cameras, object detection processing is applied to autofocus (AF) technology, which automatically focuses on a detected object as the subject. One type of AF technology is a tracking function that continuously focuses on the same subject. The tracking function identifies the tracking target in successive images, but if the tracking target is obscured by another object and becomes invisible, the tracking target will disappear. When the tracking target disappears, it must be redetected. Patent Document 1 discloses a technique for redetecting a tracking target when it has disappeared, in which the size of the search range is expanded and a search is performed again based on the tracking feature quantities of the tracking target.

JP 2009-17271 A

However, when identification is performed using tracking features alone, if the tracking target is an animal or other object with a similar texture throughout, and the object size changes before and after redetection, the tracking target (tracked part) may change. Furthermore, if redetection is performed when the tracking target has disappeared due to an unspecified object, many unspecified objects will be detected, making it difficult to detect the original tracking target from among them as intended by the user. The present invention provides technology for redetecting the tracking target from captured images while taking into account the user's intentions.

One aspect of the present invention is characterized by comprising a registration means for registering an occupancy indicating the proportion of the image area of the tracking target in the image area of the captured image or the image area of the object to which the tracking target belongs, and a feature amount of the tracking target, and a redetection means for redetecting the image area of the tracking target from the captured image based on the occupancy amount and the feature amount when it is determined that tracking of the tracking target in the captured image has not been successful.

This invention makes it possible to redetect a tracking target from captured images while taking into account the user's intentions.

FIG. 1 is a block diagram showing an example of the configuration of an imaging apparatus 100. 1A is a block diagram showing an example of the hardware configuration of an imaging device 100, and FIG. 1B is a block diagram showing an example of the hardware configuration of an image analyzing device 200 and a learning device 700. FIG. FIG. 2 is a block diagram showing an example of the functional configuration of the imaging apparatus 100. 10 is a flowchart of processing performed by the imaging device 100. 10 is a flowchart showing details of the process in step S401. FIG. 1 is a diagram showing an example of the configuration of a neural network. FIG. 10 is a diagram illustrating the occupancy rate of a detection region. FIG. 10 is a diagram showing an example of a captured image. FIG. 10 is a diagram showing an example of occupancy levels and tracking features stored in a storage unit 218. FIG. 10 is a diagram showing an example of a tracking feature. FIG. 7 is a block diagram showing an example of the functional configuration of a learning device 700. 10 is a flowchart of a neural network learning process performed by the learning device 700. FIG. 1 is a diagram illustrating a method for creating learning data. FIG. 4A is a diagram showing an example of a first image, and FIG. 4B is a diagram showing an example of a second image. FIG. 2 is a block diagram showing an example of the functional configuration of the imaging apparatus 100. FIG. 10 is a diagram showing an example of a captured image.

The following describes the embodiments in detail with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the attached drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

[First embodiment]
In this embodiment, an imaging device will be described that acquires images of each frame in a moving image or still images captured periodically or irregularly as captured images, and performs a tracking process to track a tracking target (part or all of an object) detected/redetected from the captured images, and an AF process to automatically focus on the tracking target as a subject.

As shown in FIG. 1, the imaging device 100 according to this embodiment has an image analysis device 200 that performs various analytical processes on the captured image to detect/redetect the "image area of the tracking target." Also connected to the imaging device 100 is a learning device 700 that performs learning processes on the neural network used by the image analysis device 200 to perform the above operations.

First, an example of the hardware configuration of the imaging device 100 will be described using the block diagram in Figure 2(a). Note that Figure 2(a) shows the main components relevant to the following description, and does not exclude the imaging device 100 from including devices other than those shown in Figure 2(a).

The arithmetic processing device 101 has a processor such as a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and memory that has a work area for the processor. The arithmetic processing device 101 controls the operation of the imaging device 100 (including the image analysis device 200) and the learning device 700 by executing various processes using computer programs and data stored in the storage device 102.

The storage device 102 is a storage device such as a magnetic storage device or semiconductor memory. The storage device 102 stores computer programs and data that cause the arithmetic processing device 101 to control the operation of the imaging device 100 (including the image analysis device 200) and the learning device 700. The storage device 102 can also store captured images as files.

The imaging unit 105 includes a lens, an aperture, an imaging element such as a CCD or CMOS that converts light from the outside world into an analog signal, an A/D converter that converts the analog signal into a digital signal, and a generation circuit that generates a captured image based on the digital signal. The imaging unit 105 also includes an aperture control device, a focus control device, and the like. In the imaging unit 105, the imaging element photoelectrically converts light incident through the lens into an analog signal, the A/D converter converts the analog signal into a digital signal, and the generation circuit generates and outputs a captured image based on the digital signal. The imaging unit 105 also performs AF, AE, AWB, and other functions in response to instructions from the processor 101. Still images captured periodically or irregularly by the imaging unit 105, or images of each frame of a moving image captured by the imaging unit 105, are stored in the storage device 102 as captured images.

The image analysis device 200 detects/redetects the tracking target from the captured image generated by the imaging unit 105. The arithmetic processing device 101 controls the imaging unit 105 to perform tracking processing to track the tracking target detected/redetected by the image analysis device 200, and AF processing to automatically focus on the tracking target as a subject.

The input device 103 is a user interface such as a mouse, keyboard, touch panel device, or buttons, which can be operated by the user to input various instructions to the processing unit 101.

The output device 104 is a device having a display screen such as a liquid crystal panel, and displays the processing results of the arithmetic processing device 101 on the display screen in the form of images, text, etc. In this embodiment, a touch panel screen is constructed by overlaying the input device 103, which functions as a touch panel device, on the display screen of the output device 104, which functions as a liquid crystal panel. The touch panel screen displays the processing results of the arithmetic processing device 101 in the form of images, text, etc., and also accepts operational input from the user.

The arithmetic processing unit 101, storage device 102, imaging unit 105, image analysis device 200, input device 103, and output device 104 shown in FIG. 2(a) are all connected to a system bus 107. Note that the imaging device 100 may also have, for example, an I/O unit for communicating between various devices. The I/O unit may be, for example, an input/output unit such as a memory card or USB cable, or a wired or wireless transmission/reception unit.

Next, an example of the hardware configuration of the image analysis device 200 and the learning device 700 will be described using the block diagram in Figure 2(b). For simplicity's sake, in this embodiment, the image analysis device 200 and the learning device 700 will be described as having the same hardware configuration (Figure 2(b)). However, the hardware configurations of the image analysis device 200 and the learning device 700 may be different.

The arithmetic processing device 130 has a processor such as a CPU and/or GPU, and memory that has a work area for the processor. The arithmetic processing device 130 controls the overall operation of the image analysis device 200/learning device 700 by executing various processes using computer programs and data stored in the storage device 131.

The memory device 131 is a storage device such as a magnetic memory device or semiconductor memory. The memory device 131 stores computer programs and data that cause the arithmetic processing device 131 to control the operation of the image analysis device 200/learning device 700.

I/F 132 is a communication interface for performing data communication with an external device via a wired and/or wireless network. I/F 132 of the image analysis device 200 is a communication interface for performing data communication with the learning device 700. I/F 132 of the learning device 700 is a communication interface for performing data communication with the image analysis device 200. The arithmetic processing device 130, storage device 131, and I/F 132 are all connected to a system bus 133.

Next, an example of the functional configuration of the imaging device 100 will be described using the block diagram in FIG. 3. In the following, each functional unit shown in FIG. 3 may be described as the subject of processing. However, in reality, the functions of each functional unit shown in FIG. 3, except for the tracking unit 219, AF processing unit 220, and memory unit 218, are realized by the processing unit 130 executing a computer program that causes the processing unit 130 to realize the functions of the functional units. Similarly, the functions of the tracking unit 219 and AF processing unit 220, among the functional units shown in FIG. 3, are realized by the processing unit 101 executing a computer program that causes the processing unit 101 to realize the functions of the tracking unit 219 and AF processing unit 220.

The acquisition unit 210 acquires the captured image generated by the imaging unit 105. For example, the acquisition unit 210 acquires the full HD (1920 pixels x 1280 pixels) captured image generated by the imaging unit 105 in real time (60 frames per second).

The acquisition unit 230 acquires information related to objects and parts of those objects (head, arms, legs, etc.) in the captured images acquired by the acquisition unit 210. An example of the functional configuration of the acquisition unit 230 is shown in the block diagram of Figure 3(b).

The extraction unit 211a extracts features (object features) from the captured image acquired by the acquisition unit 210. The estimation unit 212a uses the object features extracted from the captured image by the extraction unit 211a to estimate (detect) the entire image area of the object in the captured image or a partial image area of the object as a detection area. As a result, for each detection area estimated from the captured image, the estimation unit 212a acquires the position of the detection area in the captured image (center position, upper left corner position, etc.), the size of the detection area (vertical size and horizontal size), and the likelihood that the target included in the detection area is an object. Hereinafter, "feature" is synonymous with "feature vector" or "image feature."

For each detection area estimated from the captured image by the estimation unit 212a, the estimation unit 213a uses the object feature amounts extracted from the captured image by the extraction unit 211a to estimate the occupancy rate, which is the proportion of the image area of the object in the captured image that the detection area occupies.

The extraction unit 214a extracts features (tracking features) corresponding to each detection area estimated from the captured image by the estimation unit 212a. The selection unit 240 selects a detection area of the tracking target from the detection areas estimated from the captured image by the estimation unit 212a. The selection unit 240 then stores (registers) in the memory unit 218 the occupancy rate estimated by the estimation unit 213a for the selected detection area and the tracking features extracted by the extraction unit 214a for the selected detection area. The selection unit 240 has a selection unit 215, an input unit 216, and an input unit 217.

The input unit 216 displays the captured image acquired by the acquisition unit 210 on the display screen of the output device 104 and accepts user operations to indicate the position of the tracking target in the captured image. When the user operates the input device 103 to indicate the position of the tracking target in the captured image, the input unit 216 acquires the image coordinates of that position.

The input unit 217 acquires the range of occupancy of the tracking target (occupancy range) input by the user operating the input device 103. The selection unit 215 selects the detection area of the tracking target from the detection areas estimated from the captured image by the estimation unit 212a, based on the image coordinates acquired by the input unit 216 and the occupancy range acquired by the input unit 217. The selection unit 215 then stores (registers) in the memory unit 218 the occupancy estimated by the estimation unit 213a for the detection area of the tracking target and the tracking feature amount extracted by the extraction unit 214a for the detection area of the tracking target.

The tracking unit 219 performs tracking processing to track the tracking target in the captured image acquired by the acquisition unit 210, using the tracking feature amounts of the tracking target detection area stored in the memory unit 218 and the tracking target detection area selected by the selection unit 215 or the tracking target image area re-detected by the re-detection unit 250. Since tracking processing is well known, a detailed description of the tracking processing will be omitted.

The AF processing unit 220 performs AF processing on the image area of the tracking target that is being tracked by the tracking unit 219 in the captured image acquired by the acquisition unit 210. AF processing is well known, so a detailed description of the AF processing will be omitted.

The determination unit 221 determines whether the tracking unit 219 has successfully tracked the tracking target. The re-detection unit 250 re-detects the tracking target from the captured image if the determination unit 221 determines that the tracking unit 219 has not successfully tracked the tracking target (failed). An example of the functional configuration of the re-detection unit 250 is shown in the block diagram of Figure 3(c).

In FIG. 3(c), the extraction unit 211b, estimation unit 212b, estimation unit 213b, and extraction unit 214b operate in the same manner as the extraction unit 211a, estimation unit 212a, estimation unit 213a, and extraction unit 214a, respectively. In other words, the re-detection unit 250, like the acquisition unit 230, acquires the occupancy rate and tracking feature amount for each detection area from the captured image.

The processing unit 222 compares the occupancy and tracking feature values acquired for each detection area with the occupancy and tracking feature values of the detection area of the tracking target stored in the memory unit 218, and identifies (redetects) the detection area of the tracking target among the respective detection areas.

Next, the processing performed by the imaging device 100 to perform tracking processing in such an imaging device 100 will be described with reference to the flowchart in FIG. 4. In step S401, processing is performed to store in the storage unit 218 the occupancy and tracking feature amounts for the detection area of the tracking target, which is the entire or part of the object included in the captured image. Details of the processing in step S401 will be described with reference to the flowchart in FIG. 5.

In step S501, the acquisition unit 210 acquires the captured image generated by the imaging unit 105. This captured image is, for example, bitmap data of an RGB color image in which the R (red), G (green), and B (blue) pixel values of each pixel are all expressed in 8 bits.

In step S502, the extraction unit 211a extracts object features from the captured image acquired by the acquisition unit 210 in step S501. Various methods can be applied to extract object features from the captured image, but in this embodiment, the extraction unit 211a extracts object features from the captured image using a neural network. An example configuration of a neural network used to extract object features from the captured image is shown in Figure 6.

The extraction unit 211a is a neural network that performs recognition tasks by repeatedly using convolutional layers and pooling layers. The extraction unit 211a has multiple convolutional layers 511, 513, and 515 and multiple pooling layers 512 and 514, and uses these layers to extract object features from the input image (captured image) 530.

In the convolutional layer, for example, a 3x3 filter is set on multiple channels for the input image or feature map, a convolution operation is performed centered on the pixel of interest, and multiple feature maps 551, 553, and 555 corresponding to the multiple channels are output.

The pooling layer generates reduced feature maps 552, 554 by reducing the feature maps output from the convolutional layer. When pooling within a 2x2 range, the feature maps are reduced by a factor of 4. Pooling can be performed using methods such as maximum value pooling or average value pooling.

Note that the configuration of the neural network applicable to the extraction unit 211a is not limited to the configuration shown in Figure 6; for example, the neural network may be made more multilayered than the one shown in Figure 6, or the number of channels may be changed.

In step S503, the estimation unit 212a uses the object features extracted from the captured image by the extraction unit 211a in step S502 to estimate the entire image area of the object in the captured image or a partial image area of the object as a detection area. In this estimation, the estimation unit 212a estimates the position of the detection area, the size of the detection area, and the likelihood that it is an object, for each detection area estimated from the captured image.

In this embodiment, the estimation unit 212a also uses a neural network to estimate a detection area from a captured image. Taking Figure 6 as an example, the feature map 555 is input to the fully connected layer 556, which outputs the position, size, and likelihood of the detection area (detection frame), thereby estimating the detection area.

In step S504, the estimation unit 213a estimates the occupancy rate for each detection region estimated from the captured image by the estimation unit 212a in step S503. In this embodiment, the estimation unit 213a also estimates the occupancy rate of each detection region using a neural network. Taking Figure 6 as an example, the feature map 555 is input to the fully connected layer 556, and the occupancy rate of the detection region is output.

Here, we will explain occupancy in more detail. The occupancy of a detection area is the degree to which the detection area captures the image area of an object in a captured image. We will explain the occupancy of a detection area using the specific example shown in Figure 7. In Figure 7, a dog is used as an example of an object.

In Figure 7, the captured image 600 includes an image area 601 of a dog, and the estimation unit 212a estimates a detection area 602 that includes the entire body of the dog, and a detection area 603 that includes the head, which is part of the dog.

The detection area 602 captures the entire image area 601, and the detection area 602 occupies 100% of the image area 601, so the estimation unit 213a estimates that the occupancy rate of the detection area 602 is "1.0".

The detection area 603 captures a portion of the image area 601, and if the detection area 603 occupies 20% of the image area 601, the estimation unit 213a estimates that the occupancy rate of the detection area 603 is "0.2".

In step S505, the extraction unit 214a extracts tracking features corresponding to each detection area estimated from the captured image by the estimation unit 212a in step S503. In this embodiment, the extraction unit 214a also uses a neural network to extract tracking features for each detection area from the captured image. Taking FIG. 6 as an example, the extraction unit 214a acquires a feature map in map format such as feature map 555 as tracking features. Here, the map size of the tracking features is set to 1 x 1 x C (C is an arbitrary natural number) in terms of width, height, and channels. In this embodiment, tracking features extracted by a neural network will be used for explanation; however, this is not limiting; luminance values, RGB values, their histograms, SIFT features, SURF features, etc. may also be used as tracking features.

In step S506, the selection unit 215 selects the detection area of the tracking target from the detection areas estimated from the captured image by the estimation unit 212a in step S503, based on the image coordinates acquired by the input unit 216 and the occupancy range acquired by the input unit 217.

The process of selecting a detection area for tracking by the selection unit 215 will be explained using Figure 8(a) as an example. In the captured image 610, detection areas 611 to 616 are each detection areas estimated from the captured image 610 by the estimation unit 212a.

Detection area 611 is the detection area for the dog, and detection area 612 is the detection area for the head, which is part of the dog. Detection area 613 is the detection area for the tree, detection area 614 is the detection area for the group of flowers, which is part of the tree, and detection areas 615 and 616 are the detection areas for the flowers in the group of flowers. Point 617 indicates the position indicated by the user using input device 103 as the position of the tracking target.

The input unit 216 acquires the image coordinates corresponding to point 617. In addition, the input unit 217 acquires, as the occupancy range, the "acceptable range for the occupancy of the tracking target" input by the user operating the input device 103.

The selection unit 215 selects, from among the detection areas 611 to 616, a detection area whose occupancy falls within the occupancy range and contains the image coordinates of point 617 as the detection area of the tracking target. If there are multiple "detection areas whose occupancy falls within the occupancy range and contain the image coordinates of point 617," the detection area closest to the image coordinates of point 617 among the multiple detection areas is selected as the detection area of the tracking target. Furthermore, if there is no detection area containing the image coordinates of point 617, the detection area whose occupancy falls within the occupancy range and is closest to the image coordinates of point 617 is selected as the detection area of the tracking target.

In the example of Figure 8(a), detection area 611 is the only detection area that contains point 617. Here, if the occupancy range is "0.01 to 0.6," detection area 611 contains point 617, but its occupancy is 1.0, so it is not included in the occupancy range and therefore detection area 611 is not selected as the detection area of the tracking target. In such a case, of the detection areas with occupancy levels included in the occupancy range, the detection area closest to point 617 is selected as the detection area of the tracking target. In the example of Figure 8(a), the occupancy level of detection area 612 (0.2) is included in the occupancy range, and of detection areas 612 to 616 excluding detection area 611, detection area 612 is the detection area closest to point 617. Therefore, detection area 612 is selected as the detection area of the tracking target.

In step S507, the selection unit 215 stores (registers) in the memory unit 218 the occupancy rate of the detection area of the tracking target estimated by the estimation unit 213a in step S504 and the tracking feature amount of the detection area of the tracking target extracted by the extraction unit 214a in step S505. An example of the occupancy rate and tracking feature amount stored in the memory unit 218 is shown in Figure 9.

Returning to FIG. 4, next, in step S402, the tracking unit 219 performs tracking processing to track the tracking target in the captured image acquired by the acquisition unit 210, using the tracking feature amounts of the detection area of the tracking target stored in the memory unit 218 and the detection area of the tracking target selected by the selection unit 215 or the image area of the tracking target re-detected by the re-detection unit 250.

In step S403, the determination unit 221 determines whether the tracking process in the tracking unit 219 has been successful. Various determination criteria are possible for determining whether the tracking process in the tracking unit 219 has been successful, and are not limited to specific determination criteria. In this embodiment, the determination unit 221 calculates the similarity between the tracking feature of each detection area acquired from the captured image by the acquisition unit 230 and the tracking feature of the tracking target stored in the memory unit 218. Then, if there is one or more tracking feature of the tracking feature acquired from the captured image by the acquisition unit 230 whose similarity with the tracking feature of the tracking target stored in the memory unit 218 is equal to or greater than a threshold, the determination unit 221 determines that "the tracking process in the tracking unit 219 has been successful." On the other hand, if there is no tracking feature acquired by the acquisition unit 230 from the captured image whose similarity with the tracking feature of the tracking target stored in the storage unit 218 is equal to or greater than a threshold, the determination unit 221 determines that "the tracking process in the tracking unit 219 has not been successful (failed)."

Figure 8(b) shows an example of a situation in which it is determined that "the tracking process in the tracking unit 219 was not successful (failed)." As shown in Figure 8(b), if the dog being tracked is hidden behind a tree, which is another object, there is no detection area in the captured image where the tracking feature values have a similarity to the dog's tracking feature values that is equal to or greater than the threshold. Therefore, in this case, it is determined that "the tracking process in the tracking unit 219 was not successful (failed)."

If, as a result of this determination, it is determined that the tracking process in the tracking unit 219 was successful, processing proceeds to step S404. On the other hand, if it is determined that the tracking process in the tracking unit 219 was not successful, processing proceeds to step S406.

In step S404, the arithmetic processing unit 101 determines whether the conditions for terminating the operation of the imaging device 100 have been met. For example, when the user operates the input device 103 to input an instruction to stop the operation of the imaging device 100 or to turn off the power to the imaging device 100, the arithmetic processing unit 101 determines that the conditions for terminating the operation of the imaging device 100 have been met.

If it is determined that the conditions for terminating the operation of the imaging device 100 have been met, the processing according to the flowchart in FIG. 4 ends. On the other hand, if it is determined that the conditions for terminating the operation of the imaging device 100 have not been met, the processing proceeds to step S405.

In step S405, the acquisition unit 210 acquires the captured image generated by the imaging unit 105. The process then proceeds to step S402, where the tracking unit 219 performs tracking processing on the captured image acquired by the acquisition unit 210 in step S405.

In step S406, the extraction unit 211b extracts object features from the captured image acquired by the acquisition unit 210, similar to step S502. In step S407, the estimation unit 212b uses the object features extracted from the captured image by the extraction unit 211b to estimate the entire image area of the object in the captured image or a partial image area of the object as a detection area, similar to step S503. As a result, the estimation unit 212b acquires, for each detection area estimated from the captured image, the position of the detection area, the size of the detection area, and a likelihood representing the object-likeness of the detection area.

In step S408, the estimation unit 213b estimates the occupancy rate for each detection area estimated from the captured image by the estimation unit 212b, similar to step S504. In step S409, the extraction unit 214b extracts tracking features corresponding to each detection area estimated from the captured image by the estimation unit 212b, similar to step S505.

Here, at the end of the processing of step S409, it is assumed that detection areas 621-626 have been estimated from the captured image by the estimation unit 212b, as shown in Figure 8(c). Detection area 625 is the detection area of the dog, and detection area 626 is the detection area of the head, which is part of the dog. Detection area 621 is the detection area of a tree, detection area 622 is the detection area of a group of flowers, which is part of the tree, and detection areas 623 and 624 are the detection areas of flowers in the group of flowers. An example of the occupancy and tracking features of each of detection areas 621-626 at this time is shown in Figure 10. "NO." is the reference number of each detection area, and "ID" is an identification number unique to each detection area. The tracking features of each detection area take the same map format as the tracking features of the tracked target, and the map size here is 1x1xC in width, height, and channel.

In step S410, the processing unit 222 re-detects the detection area of the tracking target from the captured image. First, the processing unit 222 acquires the occupancy OCC _T of the detection area of the tracking target from the storage unit 218. Then, as shown in the following formula, the processing unit 222 determines, as a candidate detection area, a detection area in the captured image whose occupancy is found to be included in a range based on the occupancy OCC _T.

OCC _T - α < OCC _ID < OCC _T + α
The OCC _ID is the occupancy of each detection area estimated from the captured image. α is a value related to the allowable range of fluctuation in the occupancy of the detection area of the tracking target, and is set to 0.05 here, for example. When _{OCC T} = 0.2, the detection areas corresponding to the OCC _IDs that satisfy 0.15 < OCC _ID < 0.25 are candidate detection areas. In FIG. 10 , the two candidate detection areas are the detection area 622 with an occupancy of 0.20 and the detection area 626 with an occupancy of 0.18.

Next, processing unit 222 determines, from among the candidate detection areas, the candidate detection area having the highest correlation value with the tracking feature of the tracking target acquired from storage unit 218 and equal to or greater than a threshold (≧0), as the detection area of the tracking target. In the example of Fig. 10 , processing unit 222 calculates a correlation value X1 between the tracking feature F _T (1, 1, C) of the tracking target and the tracking feature F ₂ (1, 1, C) of detection area 622. Processing unit 222 also calculates a correlation value X2 between the tracking feature F _T (1, 1, C) of the tracking target and the tracking feature F ₆ (1, 1, C) of detection area 626. Then, if correlation value X1 is higher than correlation value X2 and correlation value X1 is equal to or greater than the threshold, processing unit 222 determines detection area 622 as the detection area of the tracking target. On the other hand, if the correlation value X2 is higher than the correlation value X1 and is equal to or greater than the threshold, the processing unit 222 determines the detection area 626 as the detection area of the tracking target. Note that if both the correlation value X1 and the correlation value X2 are less than the threshold, the re-detection unit 250 does not determine the detection area of the tracking target, but instead performs the same process on the next captured image input to redetect the detection area of the tracking target. In this embodiment, the re-detection unit 250 redetects the detection area of the tracking target for each captured image input within a predetermined period after starting re-detection of the detection area of the tracking target. If the re-detection unit 250 is unable to determine the detection area of the tracking target even after re-detecting the detection area of the tracking target for each captured image input within a predetermined period after starting re-detection of the detection area of the tracking target, the re-detection unit 250 considers the re-detection to be a failure and terminates operation.

Therefore, if the redetection process in step S410 results in the detection area of the tracking target being determined, processing proceeds to step S402; if the detection area of the tracking target cannot be determined, processing proceeds to step S411.

In step S411, the redetection unit 250 determines whether a preset period (predetermined time) has elapsed since redetection of the detection area of the tracking target began. If this determination shows that the preset period (predetermined time) has elapsed since redetection of the detection area of the tracking target began, processing according to the flowchart in FIG. 4 ends. On the other hand, if the preset period (predetermined time) has not yet elapsed since redetection of the detection area of the tracking target began, processing proceeds to step S410.

In this way, in this embodiment, the occupancy rate, which indicates the proportion of the image area of the tracking target in the captured image or the image area of the object to which the tracking target belongs, and the feature values of the tracking target are registered. Then, if it is determined that tracking of the tracking target in the captured image has not been successful, the image area of the tracking target is re-detected from the captured image based on the registered occupancy rate and feature values.

As shown in FIG. 8(c), if the dog has moved further into the screen than the dog in the captured image of FIG. 8(a) (the captured image from which the occupancy and tracking features stored in the memory unit 218 were obtained), the size of the dog in FIG. 8(c) will be relatively smaller than the size of the dog in FIG. 8(a). Here, assume that the size of the entire dog in FIG. 8(c) is approximately the same as the size of the dog's head in FIG. 8(a). In this case, the tracking features of the dog's head and the entire dog will be relatively similar, so if the tracking target is detected using only the tracking features, there is a possibility that the entire dog in FIG. 8(c) will be mistakenly redetected as the dog's head. In that case, the tracking target will change from the dog's head to the entire dog before and after redetection. In this embodiment, the tracking target is redetected using the occupancy in addition to the tracking features. Therefore, the tracking target can be reliably redetected even if the size of the tracking target has changed since the captured image from which the occupancy and tracking features stored in the memory unit 218 were obtained.

Furthermore, when detecting an unspecified object in Figure 8(c), many detection areas are estimated, such as dogs, trees, and parts of them, making it difficult to identify the detection area of the tracking target from among these detection areas using only the tracking feature amounts. However, even if the tracking target is an unspecified object, redetecting the tracking target using the occupancy in addition to the tracking feature amounts makes it possible to stably redetect the tracking target.

Next, we will explain the learning device 700 that performs learning processes for the neural networks used in the extraction units 211a/211b, estimation units 212a/212b, estimation units 213a/213b, and extraction units 214a/214b.

In the following, when a common description is given for extraction unit 211a and extraction unit 211b, extraction unit 211a and extraction unit 211b will be collectively referred to as extraction unit 211. Similarly, when a common description is given for estimation unit 212a and estimation unit 212b, estimation unit 212a and estimation unit 212b will be collectively referred to as estimation unit 212. Similarly, when a common description is given for estimation unit 213a and estimation unit 213b, estimation unit 213a and estimation unit 213b will be collectively referred to as estimation unit 213. Similarly, when a common description is given for extraction unit 214a and extraction unit 214b, extraction unit 214a and extraction unit 214b will be collectively referred to as extraction unit 214.

An example of the functional configuration of the learning device 700 according to this embodiment will be described using the block diagram in Figure 11. Note that the extraction unit 211, estimation unit 212, and estimation unit 213 in Figure 11 are all shown as learning targets by the learning device 700, and do not indicate that the learning device 700 has these functional units.

In the following, the functional units shown in Figure 11 (excluding the extraction unit 211, estimation unit 212, estimation unit 213, and memory unit 701) may be described as the main processing units. However, in reality, the functions of these functional units are realized by the arithmetic processing unit 130 executing a computer program that causes the arithmetic processing unit 130 to execute the functions of the functional units.

The memory unit 701 stores training data used to train the neural networks used in the extraction unit 211, estimation unit 212, and estimation unit 213. The training data includes multiple sets of training images, area information indicating the position (center position, upper left corner position, etc.) and size (vertical size and horizontal size) of a reference area, which is the image area of the entire or partial object in the training image, and the occupancy rate of the reference area.

The acquisition unit 702 acquires training data from the storage unit 701. The acquisition unit 703 acquires training images included in the training data acquired by the acquisition unit 702. The extraction unit 211 extracts object features from the training images acquired by the acquisition unit 703.

The estimation unit 212 uses the object features extracted from the training image by the extraction unit 211 to estimate the entire image area of the object in the training image or a partial image area of the object as a detection area. As a result, for each detection area estimated from the training image, the estimation unit 212 calculates the position of the detection area in the training image (center position, upper left corner position, etc.), the size of the detection area (vertical size and horizontal size), and the likelihood that the target included in the detection area is an object.

For each detection area estimated from the training image by the extraction unit 212, the estimation unit 213 estimates the occupancy as the proportion of the image area that the detection area occupies in the image area that includes the entire image area of the object in the training image.

The calculation unit 707 calculates, as the region error, the error (error based on the position error and size error) between the position and size of the detection region estimated by the estimation unit 212 from the training image and the position and size of the reference region indicated by the region information paired with the training image. The calculation unit 707 calculates, for example, the distance between the position of the detection region and the position of the reference region as the "position error." The calculation unit 707 also calculates, for example, the sum of the difference between the vertical size of the detection region and the vertical size of the reference region and the difference between the horizontal size of the detection region and the horizontal size of the reference region as the "size error." Then, for example, the calculation unit 707 calculates the sum of the "position error" and the "size error" for all detection regions, and calculates the sum of the sums calculated for all detection regions as the region error.

The calculation unit 708 calculates the difference between the occupancy for each detection region estimated from the training image by the estimation unit 213 and the occupancy for each reference region as the occupancy error. The learning unit 709 updates the parameters of the neural networks used by the extraction unit 211, estimation unit 212, and estimation unit 213 so as to reduce the region error calculated by the calculation unit 707 and the occupancy error calculated by the calculation unit 708. The neural network parameters are, for example, the weight coefficients of the convolutional layer and fully connected layer in the neural network. This updating process realizes the neural network learning process.

The neural network learning process performed by the learning device 700 will be described with reference to the flowchart in Figure 12. In step S801, the acquisition unit 702 acquires learning data from the storage unit 701. The acquisition unit 703 acquires learning images included in the learning data acquired by the acquisition unit 702. The learning data is created in advance and stored in the storage unit 701. A method for creating the learning data will now be described with reference to Figure 13. The learning data may be created by the learning device 700 or by another device.

In Figure 13(a), a training image 1300 including a person 1350 has an image area 1310 of the entire body of the person 1350, an image area 1320 of the head of the person 1350, an image area 1330 of the torso of the person 1350, and an image area 1340 of the lower body of the person 1350 set.

In the training image 1400, which is a close-up image of the person 1450 in Figure 13(b), an image area 1402 of the head of the person 1450, an image area 1403 of the torso of the person 1450, and an image area 1404 including the entire person 1450 in the training image 1400 are set.

In the training image 1500 containing the automobile 1550 in Figure 13(c), an image region 1501 containing the entire automobile 1550, image regions 1502 and 1503 of the headlights, image regions 1504 and 1505 of the tires, and an image region 1506 of the windshield are set.

In Figure 13 (d), a training image 1600 containing a cat 1650 has an image area 1601 for the entire body of the cat 1650, an image area 1602 for the head, an image area 1603 for the right eye, an image area 1604 for the left eye, and an image area 1605 for the torso set.

The image area of an object or a portion of an object on a training image may be set manually by the user using a user interface such as the input device 103, or by setting the image area detected by the detector. The image area detected by the detector may also be manually corrected by the user.

In this way, area information indicating the position and size of the image area of an object or a part of the image area of an object set in the training image is registered in the training data together with the training image. Note that, depending on the object, rotation information indicating the rotation direction and angle of the image area may also be included in the training data.

In addition, the ratio of the area of the "entire image area of the object or a portion of the object's image area" to the area (number of pixels) of the entire image area of the object in the training image is calculated as the occupancy of that image area, and this occupancy is registered in the training data together with the training image.

In the example of Figure 13(a), the ratio of the area of image region 1310 to the area of image region 1310 of the entire body of person 1350 is calculated as the occupancy of image region 1310 (the occupancy in this case is 1.0). The ratio of the area of image region 1320 of the head to the area of image region 1310 of the entire body of person 1350 is calculated as the occupancy of image region 1320. The ratio of the area of image region 1330 of the torso to the area of image region 1310 is calculated as the occupancy of image region 1330. The ratio of the area of image region 1340 of the lower body to the area of image region 1310 is calculated as the occupancy of image region 1340. In the example of Figure 13(a), the area of image region 1340 of the lower body is about half the area of image region 1310, so the occupancy of image region 1340 is calculated as 0.5.

As shown in the example of Figure 13(b), in the case of learning image 1400 captured by capturing a close-up of person 1450, image area 1404 of a portion of person 1450 is present, but the image area of person 1450's entire body is not present. In such a case, the user visually estimates and inputs the occupancy of image area 1404 relative to the image area of person 1450's entire body. When calculating the occupancy of image area 1402 of the head, the occupancy of image area 1402 is calculated as the product of the ratio of the area of image area 1402 to the area of image area 1404 and the occupancy of image area 1404. When calculating the occupancy of image area 1403 of the torso, the occupancy of image area 1403 is calculated as the product of the ratio of the area of image area 1403 to the area of image area 1404 and the occupancy of image area 1404. The user may also visually estimate and input the occupancy of image area 1402 of the head and image area 1403.

In the example of Figure 13 (c), the ratio of the area of image area 1501 to the area of image area 1501, which includes the entire automobile 1550, is calculated as the occupancy of image area 1501. The ratio of the area of image area 1502 of the automobile 1550's headlights to the area of image area 1501 is calculated as the occupancy of image area 1502. The ratio of the area of image area 1503 of the automobile 1550's headlights to the area of image area 1501 is calculated as the occupancy of image area 1503. The ratio of the area of image area 1504 of the automobile 1550's tires to the area of image area 1501 is calculated as the occupancy of image area 1504. The ratio of the area of image area 1505 of the automobile 1550's tires to the area of image area 1501 is calculated as the occupancy of image area 1505. Additionally, the ratio of the area of the image area 1506 of the windshield of the automobile 1550 to the area of the image area 1501 is calculated as the occupancy of the image area 1506.

In the example of Figure 13(d), the ratio of the area of image region 1601 to the area of image region 1601 of the entire body of cat 1650 is calculated as the occupancy of image region 1601. The ratio of the area of image region 1602 of cat 1650's head to the area of image region 1601 is calculated as the occupancy of image region 1602. The ratio of the area of image region 1603 of cat 1650's right eye to the area of image region 1601 is calculated as the occupancy of image region 1603. The ratio of the area of image region 1604 of cat 1650's left eye to the area of image region 1601 is calculated as the occupancy of image region 1604. The ratio of the area of image region 1605 of cat 1650's to the area of image region 1601 is calculated as the occupancy of image region 1605.

If you want to estimate a detection area by limiting the type (category) of object to be detected, you can prepare training data for the limited type of object. For example, if you want to detect people, you can prepare training data for people, and if you want to detect cars, you can prepare training data for cars. If you want to estimate a detection area without limiting the type of object to be detected, you can prepare training data for various types of objects.

For example, in addition to the people, cars, and cats shown in Figure 13, training data for various types of objects such as trains, airplanes, insects, birds, and dogs can be prepared. If training data for various types of objects is prepared in this way and the training process is performed appropriately, it will be possible to detect object types that are not included in the training data. For example, even if there is no training data for fish, it will be possible to detect fish because it is possible to detect their fish-likeness.

Returning to FIG. 12, next, in step S802, the extraction unit 211 extracts object features from the training image acquired by the acquisition unit 703 in step S801 using the neural network currently being trained.

In step S803, the estimation unit 212 uses the object features extracted from the training image in step S802 and the neural network being trained to estimate the entire image area of the object in the training image or a partial image area of the object as the detection area.

In step S804, the estimation unit 213 uses the neural network being trained to estimate the occupancy rate for each detection area estimated from the training image by the extraction unit 212 in step S803.

In step S805, the calculation unit 707 calculates the region error based on the position and size of the detection region estimated by the estimation unit 212 from the training image and the position and size of the reference region indicated by the region information paired with the training image. For example, the calculation unit 707 calculates the region error by adding the deviation (e.g., squared error) of the center coordinates and size of the detection region estimated by the estimation unit 212 from the center coordinates and size of the corresponding reference region in the training data.

In step S806, the calculation unit 708 calculates the difference (e.g., squared error) between the occupancy of each detection region in the training image estimated by the estimation unit 213 and the occupancy of the corresponding region paired with the training image. The calculation unit 708 then calculates the sum of the differences calculated for each detection region in the training image as the occupancy error.

In step S807, the learning unit 709 updates the parameters of the neural networks used by the extraction unit 211, estimation unit 212, and estimation unit 213 so as to reduce the sum (loss value) of the area error calculated by the calculation unit 707 and the occupancy error calculated by the calculation unit 708. The parameters can be updated using, for example, backpropagation.

In step S808, the learning unit 709 determines whether the learning termination condition has been met. There are various conditions for terminating learning, and they are not limited to specific conditions. For example, the learning termination condition may be that the loss value is below a threshold, the rate of change of the loss value is below a threshold, or the number of parameter updates is above a threshold. Alternatively, for example, learning data for accuracy verification may be prepared separately from the learning data for parameter update, and the processing of steps S801 to S807 described above may be performed, and the learning termination condition may be determined to be met when the sum of the loss values is below a threshold.

If it is determined that the learning termination conditions have been met, processing according to the flowchart in Figure 12 ends; if it is determined that the learning termination conditions have not been met, processing proceeds to step S801.

Next, learning related to the extraction unit 214 will be described. The memory unit 701 stores data (learning data) used for learning by the extraction unit 214. The learning data of the extraction unit 214 includes multiple sets of two learning images containing the same object, area information indicating the position (center position, upper left corner position, etc.) and size (vertical size and horizontal size) of the image area to be tracked in the learning image, and the occupancy of the image area. The definition of occupancy is as described above. Hereinafter, one of the two learning images included in the learning data of the extraction unit 214 will be referred to as the first image, and the other as the second image.

Figure 14 shows examples of the first and second images. Figure 14(a) shows an example of the first image, and Figure 14(b) shows an example of the second image. The first and second images include both a tracking target object 1701 and a non-tracking target object 1702. The non-tracking target is an object that has an appearance similar to the tracking target. In this way, the learning data of the extraction unit 214 includes two learning images that include both the tracking target and a non-tracking target that has an appearance similar to the tracking target.

The positions and sizes of image region 1705 of tracked target 1701 in the first image and image region 1707 of tracked target 1701 in the second image are defined by region information included in the training data. The positions and sizes of image region 1706 of non-tracked target 1702 in the first image and image region 1708 of non-tracked target 1702 in the second image are obtained from the first image and second image, respectively, by extraction unit 211 and estimation unit 212.

The extraction unit 214 then acquires tracking features of image area 1705 of the tracking target 1701 and image area 1706 of the non-tracking target 1702 from the first image, and acquires tracking features of image area 1707 of the tracking target 1701 from the second image.

The learning unit 709 then updates the parameters of the neural network used by the extraction unit 214 so that the inter-feature distance between the tracking feature of the tracked target in the first image and the tracking feature of the tracked target in the second image is shortened, and so that the inter-feature distance between the tracking feature of the tracked target in the first image and the tracking feature of the non-tracked target in the first image is lengthened. The parameter updating can be performed using, for example, backpropagation. In the example of FIG. 14 , the learning unit 709 updates the parameters of the neural network used by the extraction unit 214 so that the inter-feature distance between the tracking feature of image region 1705 and the tracking feature of image region 1707 is shortened, and so that the inter-feature distance between the tracking feature of image region 1705 and the tracking feature of image region 1706 is lengthened. Note that by preparing images containing various types of objects as training data, tracking feature amounts applicable to tracking unspecified objects can be acquired.

As such, according to this embodiment, even if the tracking target, which is an unspecified object, temporarily disappears, it can be redetected when the tracking target returns to the image. In this case, by using the occupancy of the tracking target during redetection, even if there is another object in the image with similar tracking features, the tracking target can be accurately redetected as long as the occupancy is different. Furthermore, by using the occupancy, if the tracking target is the entire object, the entire object can be redetected, and if the tracking target is only a part of the object, that part can be redetected. Therefore, it is possible to track and focus on the tracking target intended by the user.

Second Embodiment
The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the first embodiment, the detection area of the tracking target is specified using the image coordinates and occupancy range input by the user, and the occupancy and tracking feature amount for the specified detection area are registered in the storage unit 218.

In this embodiment, the detection area of the tracking target is identified using the occupancy corresponding to the imaging parameters of the imaging device 100 and the image coordinates input by the user, and the occupancy and tracking features for the identified detection area are registered in the storage unit 218.

An example of the functional configuration of the imaging device 100 according to this embodiment is shown in the block diagram of FIG. 15. In FIG. 15, functional units similar to those shown in FIG. 3 are assigned the same reference numbers, and a description of these functional units will be omitted. In the following, each functional unit shown in FIG. 15 may be described as the subject of processing. However, in reality, the functions of each functional unit shown in FIG. 15, except for the tracking unit 219, AF processing unit 220, and memory unit 218, are realized by the arithmetic processing device 130 executing a computer program that causes the arithmetic processing device 130 to realize the functions of the functional units. Similarly, the functions of the tracking unit 219 and AF processing unit 220 are realized by the arithmetic processing device 101 executing a computer program that causes the arithmetic processing device 101 to realize the functions of the tracking unit 219 and AF processing unit 220, among the functional units shown in FIG. 15.

First, we will explain the imaging parameters of the imaging device 100. Examples of imaging parameters that can be used include aperture value, exposure time, AF frame size, ISO sensitivity, and Bv value. For the sake of specificity, the following will describe, as an example, a case in which the imaging parameter of the imaging device 100 is the "aperture value of the lens of the imaging unit 105." However, the following explanation is equally applicable even if the imaging parameter of the imaging device 100 is a parameter other than the "aperture value of the lens of the imaging unit 105." The aperture value is one of the setting values that controls the brightness and blur of an image and is expressed as F1.4, F2, F2.8, F4, F5.6, F8, F11, or F16. A smaller setting value results in a larger aperture size, resulting in a brighter captured image and a shallower depth of field. On the other hand, a larger setting value results in a smaller aperture size, resulting in a darker captured image and a deeper depth of field. By reducing the aperture value and shallowing the depth of field, and limiting the range in focus, the main subject will stand out from the background, resulting in a more impressive captured image. Conversely, by increasing the aperture value and deepening the depth of field, and widening the range in focus, you can fit many subjects into the captured image without them becoming blurred.

The selection unit 940 selects a detection area of the tracking target from the detection areas estimated by the estimation unit 212a from the captured image. The selection unit 940 then stores (registers) in the storage unit 218 the occupancy rate estimated by the estimation unit 213a for the selected detection area and the tracking feature amount extracted by the extraction unit 214a for the selected detection area. The selection unit 240 has a selection unit 915, an input unit 216, and an input unit 917.

The selection unit 915 selects a detection area of the tracking target from the detection areas estimated from the captured image by the estimation unit 212a, based on the image coordinates acquired by the input unit 216 and the aperture value as an imaging parameter acquired by the input unit 917. The selection unit 915 holds occupancies corresponding to various aperture values. For example, the selection unit 915 holds "0.1" as the occupancy corresponding to an aperture value of F1.4 or less, and "1.0" as the occupancy corresponding to an aperture value of F8 or more. The selection unit 915 also calculates the occupancy corresponding to an aperture value between F1.4 and F8 by linear interpolation using the occupancy "0.1" corresponding to the aperture value F1.4 and the occupancy "1.0" corresponding to the aperture value F8.

The input unit 917 inputs the aperture value input by the user using the input device 103. The input device 103 operated by the user at this time can be, for example, a hardware dial on the imaging device 100.

The process of selecting detection areas for tracking by the selection unit 940 will be explained using Figure 16(a) as an example. In the captured image 1800, detection areas 1802 to 1804 are each detection areas estimated from the captured image 1800 by the estimation unit 212a.

Detection area 1802 is a detection area that includes the entire horse 1801, detection area 1803 is a detection area for the head, which is part of the horse 1801, and detection area 1804 is a detection area for trees. Point 1805 indicates the indicated position indicated by the user operating the input device 103 as the position of the tracking target. Here, the occupancy degree calculated by the estimation unit 213a for detection area 1802 is "1.0", and the occupancy degree calculated for detection area 1803 is "0.3".

The input unit 216 acquires image coordinates corresponding to point 1805. Furthermore, the input unit 917 acquires an "aperture value" input by the user by operating the input device 103. The selection unit 915 identifies the occupancy OCC _T stored in association with the acquired aperture value by the input unit 917. Furthermore, the selection unit 915 identifies, as a candidate, a detection area from among the detection areas 1802 to 1804 that includes image coordinates corresponding to point 1805. In the case of FIG. 16( a), the detection area that includes the image coordinates corresponding to point 1805 is detection area 1802, and therefore detection area 1802 is identified as the candidate. Note that if there are multiple detection areas that include the image coordinates corresponding to point 1805, the detection area with the largest occupancy among the multiple detection areas is identified as the candidate. Then, the selection unit 915 selects, as the detection area of the tracking target, a detection area that has an occupancy OCC _T from among the detection areas identified as candidates and detection areas included in the detection areas identified as candidates. In addition, when there are multiple detection areas with an occupancy degree of OCC _T among the detection areas identified as candidates and the detection areas included in the detection areas identified as candidates, the selection unit 915 selects, from among the multiple detection areas, the detection area closest to the image coordinates acquired by the input unit 216 as the detection area to be tracked.

For example, if the aperture value acquired by the input unit 917 is F2.8, the occupancy level corresponding to F2.8 is 0.3. Of detection areas 1802 to 1804, only detection area 1802 contains the image coordinates corresponding to point 1805, and therefore detection area 1802 is identified as a candidate. Then, of detection area 1802 identified as a candidate and detection area 1803 contained in detection area 1802, detection area 1803 is the only detection area with an occupancy level of 0.3, and therefore the selection unit 915 selects detection area 1803 as the detection area of the tracking target.

For example, if the aperture value acquired by the input unit 917 is F8, the occupancy level corresponding to F8 is 1.0. Of the detection areas 1802 to 1804, only detection area 1802 contains the image coordinates corresponding to point 1805, and therefore detection area 1802 is identified as a candidate. Of the detection area 1802 identified as a candidate and detection area 1803 contained within detection area 1802, detection area 1802 is the only detection area with an occupancy level of 1.0, and therefore the selection unit 915 selects detection area 1802 as the detection area of the tracking target.

Note that, among the detection areas identified as candidates and the detection areas included in the detection areas identified as candidates, a detection area with an occupancy degree OCC _ID that satisfies OCC _T −α< OCC _ID < OCC _T + α may be selected as the detection area of the tracking target.

In this way, when the aperture value is large, a detection area with a large occupancy is selected as the detection area for the tracking target, and when the aperture value is small, a detection area with a small occupancy is selected. This corresponds to the user's intention to limit the focus range when the aperture value is small, and to widen the focus range when the aperture value is large.

Then, suppose that after the occupancy rate and tracking features of the tracking target are registered in the memory unit 218, the captured image acquired by the acquisition unit 210 is as shown in Figure 16(b), in which the horse 1801 in Figure 16(a) moves to the right and disappears from the captured image, hiding behind a tree located in the center of the captured image. The captured images shown in Figures 18(c) and (d) are captured images acquired by the acquisition unit 210 after the captured image in Figure 16(b), and are captured images in which the horse 1801 reappears from the right side of the tree.

When the aperture setting is F2.8, as shown in Figure 16(c), the detection area 1850 of the horse 1801's head is redetected as the detection area of the tracking target, and the focus is set on this redetected head detection area 1850. As a result, a captured image is obtained in which the body of the horse 1801 and trees in the background direction relative to the head are blurred. In such a captured image, the head of the horse 1801 stands out from the background, making for an impressive captured image.

On the other hand, when the aperture setting is F8, the entire detection area 1860 of the horse 1801 is redetected as the detection area of the tracking target, as shown in Figure 16(d). Therefore, in such a captured image, the redetected horse 1801 is clearly visible in its entirety, resulting in an image that conveys the dynamic movement of the horse 1801.

In this way, by linking the occupancy setting when selecting a tracking target with the aperture value, it is possible to reliably detect the tracking target and obtain a captured image that expresses the user's intended expression.

As mentioned above, in this embodiment, the imaging parameter may be something other than the aperture value. For example, it is possible to consider a method of changing the occupancy selection criteria depending on the exposure time. As the exposure time becomes longer, the amount of subject blur and camera shake increases, so when viewing a captured image microscopically, the effects of blur become more apparent. Therefore, when the exposure time is longer, it is better to prioritize selecting a detection area with a larger occupancy as the detection area for the tracking target.

Another possible method is to change the occupancy selection criteria depending on the AF frame size. When the AF frame size is small, it can be assumed that the user intends to focus on only a portion of the subject, so the smaller the AF frame size, the better it is to prioritize selecting a detection area with a smaller occupancy as the detection area for the tracking target. Conversely, when the AF frame size is large, it can be assumed that the user intends to focus on the entire subject, so the larger the AF frame size, the better it is to prioritize selecting a detection area with a larger occupancy as the detection area for the tracking target.

In this way, in this embodiment, even if an unspecified object that is the tracking target temporarily disappears, the tracking target can be re-detected when it returns to the image. Furthermore, by linking the occupancy setting with the imaging parameters, it is possible to set the object as the tracking target so that a captured image that expresses the user's intended expression can be obtained.

[Third embodiment]
In the first and second embodiments, cases where tracking processing and AF processing are performed on the detection area of the tracking target have been described. However, the processing performed on the detection area of the tracking target is not limited to tracking processing and AF processing, and other processing such as auto exposure processing (AE processing) that controls exposure to a proper level and auto white balance processing (AWB processing) that performs color correction for a light source may also be used. Furthermore, multiple processing may be applied to the detection area of the tracking target.

Furthermore, in the configuration of Figure 1, the learning device 700 is described as being a separate device from the imaging device 100, but the imaging device 100 and the learning device 700 may be integrated to form a single imaging device 100.

Furthermore, the operation of the imaging device described in the above embodiment can also be applied to "an image processing device that detects/redetects the detection area of a tracking target from an image captured by an external imaging device." Such an image processing device can, for example, notify the imaging device of the detected/redetected detection area of the tracking target, causing the imaging device to perform tracking processing, AF processing, etc. for the detection area of the tracking target. Furthermore, such an image processing device may store information related to the detected/redetected detection area of the tracking target in an external device.

Furthermore, in the above embodiment, the functional units shown in Figures 3, 11, and 15 (excluding the memory unit 218 and memory unit 701) were described as being implemented as software (computer programs). However, some or all of the functional units shown in Figures 3, 11, and 15 may also be implemented as hardware.

Furthermore, the numerical values, processing timing, processing order, processing subject, data (information) destination/source/storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limiting.

Furthermore, some or all of the embodiments described above may be used in appropriate combination. Furthermore, some or all of the embodiments described above may be used selectively.

(Other embodiments)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

210: Acquisition unit 215: Selection unit 216: Input unit 217: Input unit 218: Storage unit 219: Tracking unit 220: AF processing unit 221: Determination unit 230: Acquisition unit 240: Selection unit 250: Redetection unit

Claims (9)

a registration means for registering an occupancy indicating a ratio of an image area of the tracking target in an image captured by the image capture device to an image area of an object to which the tracking target belongs, and a feature amount of the tracking target;
and a re-detection means for re-detecting an image area of the tracking target from the captured image based on the occupancy rate and the feature amount when it is determined that tracking of the tracking target in the captured image has not been successful. The re-detection means
The image processing device described in claim 1, characterized in that the occupancy and feature amount are calculated for a detection area of an object or a part of the object detected from a captured image, and an image area of the detection area detected from the captured image whose occupancy amount is included in a range based on the occupancy amount registered by the registration means and whose feature amount has the highest correlation value with the feature amount registered by the registration means is re-detected as the image area to be tracked. The registration means
3. The image processing device according to claim 1, wherein the image area of the entire object or a part of the object is detected as a detection area from the captured image, and the occupancy rate and feature amount of a detection area selected from the detected detection areas in accordance with a user operation are registered. The image processing device described in claim 3, characterized in that the registration means registers the occupancy and feature values of a detection area whose occupancy is included in the occupancy range input by the user and that includes the image coordinates specified by the user. The image processing device described in claim 3, characterized in that the registration means registers the occupancy and feature values of a detection area identified based on image coordinates input by the user, the detection area having an occupancy level according to the imaging parameters input by the user, or a detection area having an occupancy level included in an occupancy level range based on the occupancy level. an imaging means for capturing an image;
An image processing device according to any one of claims 1 to 5;
and processing means for executing processing on an image area of a tracking target in the captured image. The imaging device of claim 6, wherein the processing includes tracking processing, AF processing, AE processing, and AWB processing. An image processing method performed by an image processing device,
a registration step in which a registration means of the image processing device registers an occupancy rate indicating a ratio of an image area of the tracking target in an image area of the tracking target in a captured image or an image area of an object to which the tracking target belongs, and a feature amount of the tracking target;
and a re-detection step of re-detecting an image area of the tracking target from the captured image based on the occupancy rate and the feature amount when the re-detection means of the image processing device determines that tracking of the tracking target in the captured image has not been successful. A computer program for causing a computer to function as each means of the image processing device described in any one of claims 1 to 5.