WO2015093231A1

WO2015093231A1 - Image processing device

Info

Publication number: WO2015093231A1
Application number: PCT/JP2014/081004
Authority: WO
Inventors: 西谷隆夫
Original assignee: 西谷隆夫
Priority date: 2013-12-16
Filing date: 2014-11-25
Publication date: 2015-06-25
Also published as: JPWO2015093231A1

Abstract

[Problem] There are motion video processing devices whereby action input of a wearable ITC terminal is carried out using an ordinary camera. Use of foreground segmentation which extracts moving objects from a video results in misidentification of gesture inputs from shadows or light reflections. No simple method exists for avoiding the shadows or reflections which frequently occur with indoor use. [Solution] To carry out foreground segmentation excluding shadows or reflections indoors or outdoors, foreground segmentation is carried out using a method having a plurality of typical probability distributions, and using average color feature values of each region of a subdivided image as feature values used in foreground/background determinations. Provided is a simple method for removal of the shadows or reflections by elimination based on the type of the typical probability distributions and the elements of the feature values.

Description

Image processing device

The present invention relates to an image processing apparatus. The present invention particularly relates to a gesture input technique related to an IT apparatus using a general camera.

In the video monitoring and gesture input system, it becomes an obstacle if a part that does not have an entity such as a shadow appears as a moving picture. In this field, Microsoft has already made Kinect, which uses an infrared camera as its gesture input system for its game consoles, famous. This extracts the person by sensing the temperature of the object by the infrared camera. Since the shadow area is projected on the background area, it is lower than the body temperature. For this reason, the shadow area does not react to the infrared camera. However, use in places with a lot of heat sources or infrared rays is not considered. For this reason, Kinect cannot be used outdoors. In addition, it is difficult to use indoors if there is a kitchen or pet animal with a heat source indoors. *

A technique for distinguishing and extracting a moving part by a video image from a general monocular camera from a non-moving part is called foreground separation and is used for a surveillance camera or the like. However, if there is a motion in the image, there is a shadow area projected on the background by the foreground object with the foreground to be detected originally, or the reflected light from outside the camera view enters the camera view and the reflected light is in the middle If it is sometimes interrupted and fluctuated, it is detected as the foreground. In particular, reflection is remarkable in an environment where a desk lamp is attached. For this reason, in order to perform accurate gesture input, it is necessary to exclude shadows and reflection areas from foreground separation. Such a method for removing the influence of shadows and reflections is often handled as an image processing technique independent of foreground separation. However, foreground separation, shadow removal, and reflection removal generally require a large amount of computation. For this reason, the calorific value of the apparatus is large due to a large amount of calculation, and it is difficult to utilize it for a wearable terminal or the like.

However, in recent years, a method has appeared that significantly reduces the amount of calculation for shadow removal by effectively using parameters used for foreground separation. Details of this method are described in Non-Patent Document 1. This method uses the intermediate result used for foreground separation for shadow removal. The foreground separation used here divides the frame image from the camera into small areas, and determines whether each divided small area is foreground or background. Subsequently, the foreground area including shadows and reflections is excluded. Hereinafter, an area that does not include the influence of shadows and reflections is referred to as a true foreground area, and normal foreground separation including shadows and reflection areas is distinguished as an intermediate foreground area.

The method of excluding shadow regions from the intermediate foreground of Non-Patent Document 1 performs intermediate foreground separation based on a spectrum obtained by Walsh orthogonal transform for each small region of an image. The variation of the combined amount of the spectrum is expressed using a plurality of Gaussian distributions. Such a model for foreground detection using a plurality of Gaussian distributions is called a mixed Gaussian model, and a feature quantity to be input to the model is composed of a plurality of feature quantity elements created by combining spectra. In addition, the average and variance of each Gaussian distribution, and the weighting factor (hereinafter referred to as the Gaussian distribution coefficient) indicating how often each Gaussian distribution contributes, is a small region feature quantity element for each frame. It is corrected adaptively according to the value of. A Gaussian distribution with a large weight coefficient is a background Gaussian distribution because there is no movement. On the other hand, the foreground Gaussian distribution is a foreground Gaussian distribution with a small weight because the foreground object immediately leaves the area. A feature element at the position of the same small area of a new frame is input. If the feature element is included in the background Gaussian distribution updated one frame before, the background is used. If the feature element is included in the foreground Gaussian distribution, the foreground is used. The foreground Gaussian distribution and the background Gaussian distribution are determined by arranging the weighting factors in descending order. Of course, there may be cases where the feature elements are not included in the existing Gaussian distribution. In this case, an initial Gaussian distribution with the average input feature quantity is generated, and a Gaussian distribution with the smallest weight coefficient among the existing Gaussian distributions is modeled. Exclude from In this case, this small area is the foreground. Here, the expression of being included in the Gaussian distribution means that when the Gaussian distribution has mean μ and variance σ ^ 2, the probability of an event following this Gaussian distribution occurring within ± 3σ around the mean μ is 99.8%. It is coming. The symbol σ ^ 2 used in the variance represents the square of the standard deviation σ. In general, a section of 2.5σ around the average is often set as a section included in the Gaussian distribution.

In the method of Non-Patent Document 1, the feature amount is composed of three elements. First, the luminance signal of the small area is converted into a two-dimensional Walsh spectrum coefficient by a two-dimensional Walsh function. The lowest spectral coefficient f (DC) in the vertical and horizontal directions, which is the average brightness of the small area, f (ACV) obtained by weighting and adding multiple low frequency spectral coefficients in the horizontal direction with the lowest order in the horizontal direction, and the lowest in the vertical direction F (ACH) obtained by weighting and adding the low frequency spectrum coefficient in the horizontal direction with the order is used as the feature quantity element. For each of these, a mixed Gaussian model is constructed to perform foreground separation work. If even one feature element becomes the foreground, the small area is set as the foreground small area. Walsh conversion is performed because outdoor images and the like enter the field of view of the camera up to a distant view, and contain many high spatial frequencies due to forests and buildings.

Non-Patent Document 1 uses multi-resolution processing in which the vertical and horizontal sizes of the divided small areas are sequentially doubled and foreground separation is performed again. In the intermediate foreground separation using the mixed Gaussian model, independent processing is performed for each small area, so that the situation where the adjacent small area performs foreground determination with little noise regardless of the adjacent area is prevented. In the middle foreground separation in the multi-resolution division, even if the smallest divided small area is the foreground determination, the intermediate foreground area portion is used only when all the large divided small areas including the small area become the foreground. Stability can be ensured.

The above-described specific circuit until one frame image is divided into multiple resolutions and the above-described feature quantity elements f (DC), f (ACH), f (ACV) are generated for each region, and the features of all regions A circuit for performing the intermediate foreground separation by quantity is detailed in Non-Patent Document 2. There, both circuits are built into a single FPGA (Field Programmable Gate Array), and even when HDTV video is handled, power consumption is 30mW. A low power consumption indicates a foreground separation method with a small amount of calculation. In this Non-Patent Document 2, WPP (Walsh transform based Parameter processor), which is a feature processor that sequentially calculates feature amount calculation of the region size from 4x4 pixels to 64x64 pixels covering the frame image from the input video frame, and features generated by WPP The functional processor that performs the process from quantity to intermediate foreground separation is discussed as GTP (Gaussian mixture model Thread Processor).

The principle of the shadow removal method of Non-Patent Document 1 is to use the background Gaussian distribution used for intermediate foreground separation for shadow removal, and succeeds in reducing the computational complexity by one digit or less compared to the shadow removal of the conventional method. . In this method, the background region becomes the foreground due to the shadow is considered to be a result of the feature amount element deviating from the inclusion interval of the background Gaussian distribution. In addition, since each element of the feature quantity is obtained by a linear operation, the value of each feature quantity element is uniformly reduced by the shadow. For this reason, when the attenuation amount due to the shadow of a certain feature amount element can be estimated, the multiplied feature amount element returns to the original inclusion range of the background Gaussian distribution by multiplying the other feature element by the inverse number. If the small area can be determined to be a shadow by this method, true foreground separation is realized by removing the small area from the intermediate foreground area.

In order to estimate the attenuation, it is obtained from the average of the background Gaussian distribution of f (DC), which is the lowest conversion spectrum, as the feature quantity element. Since f (DC) represents the average brightness of the region, it always has a large value, and is therefore used as a reference feature amount element from the viewpoint of calculation accuracy. When there are a plurality of background Gaussian distributions, the processing is repeated considering all such possibilities.

The method for finding the shadow area after the attenuation amount is obtained can be explained with reference to FIG. 4 showing the principle of shadow area detection drawn based on Non-Patent Document 1. FIG. 4 includes sub graphs of an f (DC) element graph 401 in the foreground, another feature element graph 402, and a shadow verification graph 403. In these three subgraphs, the horizontal axis indicates the size within the dynamic range of the feature quantity element, and the value increases toward the right. The vertical axis indicates the probability of occurrence. The mark raised above the curve is the background Gaussian distribution that should contain f (DC). Also, a downward solid line arrow written in the subgraph indicates the value of the feature quantity element along the horizontal axis. Further, f (AC) described in the graph 402 and the graph 403 is applicable to both f (ACH) and f (ACV), and thus is represented by f (AC) as a representative of both.

The f (DC) feature quantity element graph 401 in the foreground shows a case where the foreground is caused by a shadow, and shows a background Gaussian distribution that should include the feature quantity f (DC) element and f (DC) originally. f (DC) deviates from this background Gaussian distribution due to the influence of shadows. For this reason, it becomes a foreground in intermediate foreground separation. As can be understood from the graph 401, the average μDC of the Gaussian distribution divided by f (DC) is the reciprocal of the attenuation, and this is called the correction coefficient A. As shown in the shadow area graph 401, the feature element f (DC) is smaller than the inclusion area of the background Gaussian distribution. If there is such a small area, it becomes an intermediate foreground area and becomes a shadow candidate. Next, it is verified that the region is a shadow region using another feature amount element graph 402 and a shadow verification graph 403 in the small region of the shadow candidate. A graph 402 is an example in which a feature element f (AC) different from f (DC) is attenuated by the influence of a shadow and becomes a foreground. If this small area is a shadow, f (AC) should be attenuated at the same rate as f (DC). The shadow verification graph 403 is a principle diagram for verifying whether it is a shadow area. f (AC) is indicated by a downward dotted arrow. The result of multiplying f (AC) by the correction factor A previously obtained is indicated by a downward-facing practice arrow. That is, if this small area is a true shadow area, the modified f (AC) returns to the originally included background Gaussian distribution. Therefore, in such a case, f (DC) and f (AC) are considered to be uniformly attenuated, and are assumed to be shadow regions.

This shadow removal method can use the feature element and the background Gaussian distribution already calculated by the intermediate foreground separation. In other words, if only the background distribution whose feature quantity element f (DC) is smaller than the average of the background Gaussian distribution is checked, the shadow is verified in the order of

graphs

402 and 403 for this shadow area candidate. Good.

The above is an outdoor monitoring scenario. However, when the wall comes to the background in an indoor corridor, etc., the background range is close to that of outdoor surveillance, which causes a problem. In this state, if the background image area such as the wall of the corridor is divided into small areas and Walsh transformed, almost no high-frequency spectral components can be observed. Therefore, the principle of shadow area detection The other feature quantity elements used in the graphs 402 and 原理 403 in FIG. 4 do not have significant values. Therefore, in Non-Patent Document 1, in order to perform foreground separation in the corridor, processing with a heavy calculation amount called Retinex image enhancement is introduced as preprocessing to enhance high spectral components. However, the amount of calculation more than foreground separation is required only by this image enhancement. The effect of Non-Patent Document 1 that can greatly reduce the amount of calculation by shadow removal is halved by the introduction of this Retinex image processing.

A similar problem occurs indoors with the fingertip gesture input system, which is one of the motion inputs. In ordinary residential rooms, there are walls and furniture, and the higher-order spectral components are zero as in the corridor. FIG. 5 shows an example of a photograph in which the method of Non-Patent Document 1 is performed by transform region foreground separation without Retinex image enhancement. FIG. 5 is composed of three identical frame photographs, namely, an original image photograph 501, a conversion area intermediate foreground separation photograph 502, and a conversion area shadow removal photograph 503. In the original image 501, since one cut of the fingertip video is binarized, the fingertip cannot be seen but the background can be seen. The transformation region intermediate foreground separation photograph 502 is a photograph obtained by performing intermediate foreground separation using a mixed Gaussian model in the transformation region, and the transformation region shadow removal photograph 503 is an output photograph obtained by removing Retinex image enhancement in Non-Patent Document 1. In each of the conversion area intermediate foreground separation photograph 502 and the conversion area shadow removal photograph 503, the white part is the intermediate foreground separation area or the true foreground separation area from which the shadow is removed, and the black part is the background. Even if the foreground separation has been performed correctly as in the folded-angle conversion area intermediate foreground separation photograph 502, only the outline portion remains as in the conversion area shadow removal photograph 503 when shadow removal is performed. In other words, another measure is required to omit Retinex image enhancement.

Furthermore, the rooms of ordinary houses are not very large. Such places have highly reflective furniture such as TVs and cupboards. In this situation, when a lighting device such as a room light is used, the reflected light from the glass of the cupboard enters the camera field of view. Furthermore, consider the case where a camera is worn. When the user's body dynamically blocks the reflected light, a sudden change in the reflection area occurs and becomes the foreground. However, reflections are difficult to see with human eyesight, and can only be seen after intermediate foreground separation.

The present invention provides an apparatus for performing true foreground separation in an indoor / outdoor environment from which shadows and reflection areas are removed by simple post-processing.

The present invention provides the following inventions in order to solve the above problems.

1. The input frame from the camera is divided into small areas, and the color average component signal obtained by calculating the average of the color components in the small area is used as a feature quantity element, and it is matched to the probabilistic variation of each feature quantity element. Extracting an intermediate foreground small region by comparing it with a feature value while modeling the small region with one or more typical probability distributions, and an intermediate value extracted by the mean and variance of the feature value element and the background probability distribution An image processing apparatus characterized by extracting a true foreground area that does not include a foreground part due to a shadow or reflection, and comprises a step of obtaining a true foreground area by identifying and eliminating a shadow or reflection area in the foreground area .

In the above method, considering Gaussian distribution as a typical probability distribution, it is a method to obtain true foreground separation by removing shadow and reflection from intermediate foreground separation using mixed Gaussian model. Instead, the authentic foreground separation result is extracted by using the intermediate foreground separation result. Even when the screen is divided into small areas and all the high-order spatial spectra cannot be obtained, the problem can be solved by changing the feature value of the conversion area so far to the average color component for each small area. The fact that at least two elements of the three primary colors are likely to have significant values is used. In addition, the foreground separation by the mixed Gaussian model becomes prominent when using features such as pixel units or 2x2 pixels in high-definition video, but this method uses color in units of small areas of 4x4 pixels or more. Run using the average value of the signal. For this reason, although it depends on the size of the region, the influence of noise or the like is reduced and the stability is increased. A typical probability distribution may be a Laplace distribution. The reason why the amount of calculation is reduced is that the calculation of the image enhancement processing performed as the pre-processing by the method of Non-Patent Document 1 is completely eliminated, and the calculation of the transform domain spectrum is unnecessary.

2. For each element of the feature quantity, the probability distribution of the variation is approximated by multiple typical probability distributions, and distinguished from the typical probability distribution modeling the foreground and the typical probability distribution modeling the background. The step of extracting the intermediate foreground separation as described in 1 above, wherein the foreground small area is defined as a case where at least one element of the quantity is included in a typical probability distribution modeling the foreground, and the foreground separation feature as an intermediate small area Selecting the largest of the quantity elements as a reference element, determining a correction factor by dividing the average of a typical probability distribution representing each background assigned to the reference element by the value of the reference element, An intermediate foreground area, where a value obtained by multiplying the value of a feature element that is not a reference element by a correction factor is included in one of the probability distributions representing the background of the feature element, and is regarded as an intermediate foreground small area due to shadow or reflection 2. An image processing apparatus for extracting a genuine intermediate region excluding the shadow / reflection region according to 1 above, comprising the step of creating a true foreground region by excluding it from the region.

In the apparatus based on this method, an efficient outdoor shadow removal method dealt with in Non-Patent Document 1 is extended so that the feature amount element of the small region is the color average of the small region, and shadow removal and reflection removal can be executed. is there. This principle is based on the principle explanatory diagram 6 for detecting shadows and reflections shown below when a typical probability distribution is a Gaussian distribution.

Principle of detecting shadow and reflection FIG. 6 shows a shadow area graph 601 of a reference feature element, another feature element graph 602 of a shadow area, a shadow verification graph 603, a reflection area graph 604 of a reference feature element, It consists of six sub-graphs, that is, another feature element graph 605 and a reflection verification graph 606. The horizontal axis, the vertical axis, the raised curve, and the arrow have the same meaning as in the principle diagram 4 of shadow area detection. Explanation of the principle of detecting shadows and reflections The shadow area graph 601 of the reference feature quantity element in FIG. 6, the other feature quantity element chart 602 of the shadow area, and the shadow verification time chart 603 are the color average of the feature quantity elements, and the principle of shadow area detection The shadow area graph 401, the other feature amount element graph 402, and the shadow verification graph 403 of the f (DC) element in FIG. 4 are changed to color average feature amount elements. However, the principle f of the shadow region detection FIG. 4 plays an important role in the feature value f (DC), whereas the principle diagram 6 for detecting shadows and reflections shows f (( N) plays the role of f (DC). Instead of the maximum value of the feature quantity element, a feature quantity element having a certain large value may be used as the reference element. In FIG. 6, the average of the typical background Gaussian distribution is changed to μDC to obtain the average μN of the background Gaussian distribution of the reference feature quantity element. That is, the correction coefficient A, which is the reciprocal of the amount of attenuation in the principle of the shadow area in FIG. 4, was a value obtained by dividing μDC by f (DC), but when using a color average feature element, μN is expressed by f (N). Instead of the divided value, the rest of the operation is the same as the principle 4 of the shadow region, and thus the description regarding the shadow removal is omitted.

On the other hand, the reflection removal is depicted in the reflection area graph 604 of the reference feature quantity element, the other feature quantity element graph 605 of the reflection area, and the reflection verification graph 606. The correction coefficient A in the amplification of the reference feature quantity element is a correction coefficient obtained by dividing μN by f (N) as in the case of the shadow, and the calculation itself is the same as in the case of the shadow removal. Here, it is considered that the influence of the reflected light from outside the camera field of view uniformly amplifies each element of the reflection area feature amount. In the reflection region graph 604 of the reference feature quantity element, the intermediate foreground region due to reflection is originally included in the background Gaussian distribution in the graph 604, but f (N) is included in the background Gaussian distribution due to the influence of reflected light. This is because it becomes brighter. The same applies to the other feature amount element graph 605 in the reflection region. The reflection verification graph 606 shows a case where the feature amount element of the other feature amount element graph 605 in the reflection region is multiplied by the correction coefficient A to return to the inclusion region of the background distribution. In this case, it is determined as the reflection region.

Although A was used as the correction factor, another calculation method that follows mathematically equivalent processing, for example, in the case of shadows, finds the amount of attenuation by dividing the reference feature element by the average of the back Gaussian distribution. In the case of, the amount of amplification is obtained, in the case of shadow, the average of the background Gaussian distribution is multiplied by the amount of attenuation and compared with the feature element, and in the case of reflection, the average of the background Gaussian distribution is multiplied by the amount of amplification and compared with the feature quantity. These methods are also part of the present invention.

3. Divide the frame image into small areas that do not overlap first, then double the size and width of the small areas arranged in a mesh, and then cover the screen with an enlarged small area that contains four previously divided small areas. Performing multiple division by repeating the division for creating the enlarged small region a plurality of times, performing intermediate foreground separation on each of the multiple divided small regions, and shadow / reflection from the intermediate foreground separation of the small region dividing unit A step of removing the region to make a foreground region without shadow / reflection, and a minimum foreground region without shadow / reflection, and all the enlarged regions including the minimum foreground region without shadow / reflection are all foreground regions without shadow / reflection Alternatively, in the image processing apparatus according to the first aspect, the final true foreground separation is performed by setting an intermediate foreground small area as a true foreground small area.

Since the apparatus based on this method uses the feature amount as an average base of colors, it is necessary to increase the stability in consideration of the use up to outdoor use. As a phenomenon that is likely to occur in high-precision image processing such as high-definition images, if the foreground is separated only in a small foreground area, an intermediate foreground is likely to occur due to erroneous determination due to noise. Suppressing such a result is the determination of the intermediate foreground of the enlarged region divided by the multi-resolution. This modifies the multi-resolution processing that uses the intermediate foreground result of the enlarged region to suppress even if the small region is the authentic foreground result when the enlarged region is not the intermediate foreground.

Shadows and reflections are verified under the condition that uniform attenuation and amplification are performed in a subdivided region. For this reason, it is easier to keep the assumption that each of the feature elements changes uniformly when shadows and reflections are removed only when the divided region is small. For this reason, shadows and reflection removal are performed only in small divided areas. When a small divided area becomes a true background by verification of shadows and reflections, it is sufficient to perform intermediate foreground determination with a larger division.

4). When only the intermediate foreground part due to reflection is excluded from the intermediate foreground region in the above 3, the reference feature amount element obtained by the method of obtaining the reference feature amount element in the above 3 is larger than the average of the background probability distributions. Only a means for obtaining a correction coefficient, means for multiplying a feature quantity element other than the reference feature quantity by a correction coefficient, and eliminating the intermediate foreground small area in which the multiplied feature quantity element is included in the background probability distribution of the feature quantity element. An image processing device that performs genuine foreground separation.

This method corresponds to the case where the typical probability distribution is a Gaussian distribution and the reflection area is to be extracted more stably than the shadow area, and the shadow removal is not performed in a certain small area section but the reflection removal is executed. If reflection is dominant, shadow removal may not be used. When only such a reflection region is excluded, processing is performed using only the background Gaussian distribution in which the value of the reference feature quantity element is higher than the average of the background Gaussian distribution. The principle of the processing using the color feature amount is explained. Since only the reflection removal process shown by the reflection region graph 604 of the reference element, the other feature amount element graph 605 of the reflection region, and the reflection verification graph 606 in FIG. Can be halved.

5. When only the intermediate foreground portion due to the shadow is excluded from the intermediate foreground area in 3 above, from the typical background probability distribution in which the reference feature value element obtained by the method of obtaining the reference feature value element in 3 is smaller than the average of the background probability distributions. Only a means for obtaining a correction coefficient, means for multiplying a feature quantity element other than the reference feature quantity by a correction coefficient, and eliminating the intermediate foreground small area in which the multiplied feature quantity element is included in the background probability distribution of the feature quantity element. An image processing device that performs genuine foreground separation.

This method corresponds to the case where the typical probability distribution is a Gaussian distribution and the shadow area is to be extracted more stably than the reflection area. When the shadow removal is not performed in a certain small area processing, but only the shadow removal is to be executed. Or, if shadows are dominant, reflection removal may not be used. When only such a shadow region is excluded, processing is performed using only the background Gaussian distribution in which the value of the reference feature quantity element is lower than the average of the background Gaussian distribution. The processing method is the principle explanation using color feature amount. Since only the shadow removal process shown in the shadow region graph 601 of the reference element, the other feature amount component graph 602 of the shadow region, and the shadow verification graph 603 in FIG. Can be halved.

6). 1 to 5 above, between the typical background probability distribution arranged in descending order of the weighting coefficient of the typical probability distribution to which the reference element belongs and the typical background probability distribution arranged in descending order of the weighting coefficient of the non-reference feature quantity. Thus, the image processing apparatus obtains a true foreground area by verifying an intermediate foreground area caused by a shadow or background using only typical background probability distributions of the same rank.

In this method, when the typical probability distribution is a Gaussian distribution, even if it is determined that it is an intermediate foreground by deviating from the background Gaussian distribution region where the reference feature element is present due to shadows or reflections, the background Gaussian distribution is deviated from. There is no information. For this reason, verification of shadows and reflections is generally omnipresent between all the background Gaussian distributions of the reference feature element and the non-reference feature element. However, as described in detail in the conventional example, in the classification of the background Gaussian and the foreground Gaussian in the mixed Gaussian model, each feature amount element is determined in the order of the weighting coefficient of the Gaussian distribution. The weighting coefficient becomes larger as the Gaussian distribution is used more often. For this reason, when the background gauss of the reference feature quantity element and the other feature quantity elements are arranged in the order of weighting coefficients, the corresponding Gaussian distributions are often background Gaussian distributions without shadows. Using this fact, shadows and reflections are detected only when the order of the weighting factors of the background Gaussian of the reference feature elements used for verification and the order of the weighting coefficients of the background Gaussian distribution of the non-reference feature elements are matched. Simplify the process using Fig. 6.

According to the above problem solving means, genuine foreground separation that does not include a shadow and a reflection area can be realized as post-processing of intermediate foreground separation using an intermediate result of intermediate foreground separation. This eliminates the need for image enhancement and reflection processing on the image, which are heavy processing for authentic foreground separation indoors, and can greatly reduce the amount of calculation. For this reason, it can utilize for the wearable terminal etc. which use the gesture recognition which makes power consumption reduction an important subject as an input means.

It is a figure which shows an example of the moving image processing system which concerns on one Embodiment. It is a figure which shows an example of the feature-value multiple division feature-value production | generation part. It is a figure which shows the operation | movement flowchart of a shadow / reflection area removal part. It is a principle diagram of shadow area detection. It is a figure which shows the example of a conversion area foreground isolation | separation photograph without Retinex image enhancement. It is a principle figure which detects a shadow and reflection. It is a photograph collection showing the flow of authentic background separation by the shadow / reflection removal method.

Hereinafter, the present invention will be described through embodiments of the invention. However, in the following embodiments, not all claims are necessarily essential to the means for solving the invention. The following embodiments do not limit the invention according to the claims. Further, although a Gaussian distribution will be described as a typical distribution, the present invention is not limited to the Gaussian distribution. In addition, parameters written in parentheses such as feature elements f (R), f (G), f (B), f (N) are described as subscripts on the drawing for the purpose of simplifying the drawing. It is described as fR, fG, fB, fN.

FIG. 1 shows an example of a moving image system according to an embodiment. The moving image system of the present invention includes a camera 10, a multiple division feature amount generation unit 20, a mixed Gaussian intermediate foreground processing unit 30, a shadow / reflection area removal unit 40, and a genuine foreground image generation unit 50.

A signal from the camera 10 is input to the multiple division feature generation unit 20 for each of the R, G, and B color components, and the frame image for each color component is divided into 4 × 4 pixel small regions that do not overlap and R for each region. , G, B The color component average is obtained as f (R), f (G), f (B), and these are output as feature quantity elements. Subsequently, the multiple division feature amount generation unit 20 integrates the odd and even sub-regions of the rows and columns formed by the processed sub-region of 4 × 4 pixels into an enlarged sub-region of 8 × 8 pixels. The operation is performed to output the average color as a feature quantity element. Similarly, the divided area is enlarged and, for example, color feature amount elements of each small area divided from 4 × 4 pixels to 64 × 64 pixel areas are generated and output. The three feature quantity elements obtained for each region are sequentially sent to the mixed Gaussian foreground processing unit 30. Details of the specific configuration of the multi-resolution division feature quantity generation unit 20 can be easily configured using the Walsh Parameter Processor that discusses the LSI processor architecture of the multi-resolution feature quantity based on the mixed Gaussian model of Non-Patent Document 2. It will be described later.

The feature quantity from the multiple division feature quantity generation unit 20 is subjected to intermediate foreground separation by a mixed Gaussian model for each feature quantity element in the intermediate foreground processing unit 30. When a feature value of a new small region is input from the multi-division feature amount generation unit 20, the intermediate foreground processing unit 30 uses the Gaussian distribution coefficient of the mixed Gaussian model prepared in the previous frame for this small region processing. Perform foreground separation and adaptively update each Gaussian distribution coefficient for the next frame. For this processing, Gauss Thread Processor (GTP) of Non-Patent Document 2 can be used as it is. However, the current feature value of this small area for shadow reflection removal, all background Gaussian coefficients before updating, and the intermediate foreground flag on if the result of intermediate foreground separation is the intermediate foreground, If there is, turn it off and send it to the shadow / reflection removal unit 40.

The shadow / reflection area removing unit 40 removes the intermediate foreground area due to shadows and reflections using the feature quantity for each small area, the intermediate foreground flag, and the Gaussian distribution coefficient of the background Gaussian distribution input from the intermediate foreground processing unit 30. Prior to removal, the contents of the intermediate foreground flag are first transferred to the authentic foreground flag. If the small area is caused by shadow or reflection, the authentic foreground flag is lowered by the operation described below. This eliminates it from the intermediate foreground area.

In the processing of the shadow / reflection area removing unit 40, first, the largest feature quantity element among the feature quantities of the small area composed of color average elements is used as a reference element. This is to prevent the accuracy of division for calculating the correction coefficient A from deteriorating. The principle according to FIG. 6 for detecting shadows and reflections using this reference element is performed. However, if a small region becomes the foreground in the middle foreground separation, it is clear that the reference element has deviated from the background Gaussian distribution region, but it is clear from which background Gaussian distribution in the mixed Gaussian model for the reference element it has moved. Absent. For this reason, a check of shadows and reflections is performed with respect to a plurality of background Gaussian distributions. First, the correction coefficient A is determined by the average of the first background Gaussian distribution of the reference element and the current reference element. Next, feature amount elements other than the reference element are selected one by one, and it is examined whether or not the result of multiplying it by the correction coefficient A is included in the background Gaussian distribution corresponding to the element. If it is included in the background Gauss, the true foreground flag is lowered and the process proceeds to the true foreground image generation unit 50 because the small area is included in the shadow or reflection part. That is, since the true foreground flag is reset, the small area has been part of the intermediate foreground until then, but is returned to the background. If the correction factor A is not included in the background Gaussian distribution, the correction factor A is selected again by selecting the next Gaussian distribution for the reference feature amount and checked again. Although it is a heavy calculation because it is a brute force method, the number of Gaussian distributions corresponding to each element is typically about 3, so it is not so heavy processing. In addition, the three feature elements should be attenuated or amplified uniformly. However, according to Non-Patent Document 1, since the average of the background Gaussian distribution is used to obtain the value of the correction coefficient A, it is difficult to obtain an accurate correction value for the correction coefficient A due to actual shadows and reflections. For this reason, it is assumed that one of the two feature quantity elements other than the reference element only needs to satisfy the shadow / reflection condition. This method is also adopted in the present invention. Details of the shadow / reflection area removing unit 40 will be described later in detail using an operation flowchart.

The genuine foreground image generation unit 50 receives one genuine foreground flag for each of the small regions divided and divided from the shadow / reflection region removal unit 40. The received authentic foreground flag indicates whether or not the corresponding small area is an authentic foreground. Further, it is known which position in the frame image this small region corresponds to. First, prepare an image memory for one frame to be a true foreground image, and when the true foreground flag from a small area of 4x4 pixels is received from the shadow / reflection area removal unit 40, the position of the corresponding 4x4 pixel on this image memory Each time the authentic foreground flag is copied. That is, if the true foreground flag is on, 4x4 1s are moved to the location of the image memory that creates the corresponding multi-resolution true foreground, and if the true foreground flag is off, 4x4 0s are moved. When all the processing of a small area of 4 × 4 pixels is completed, a true foreground image is created in the image memory with a binary value that is 0 or 1 for each 4 × 4 area.

When the 4 × 4 pixel small region processing is completed, the process proceeds to the 8 × 8 pixel small region processing, and then proceeds to the maximum pixel region sequentially. In the following, processing when a genuine foreground flag from the 8 × 8 pixel area is received will be described. In this case, 8 × 8 1s or 0s are prepared as data according to the true foreground flag. Since the true foreground image frame synthesized from all the results of 4x4 pixels has already been created on the image memory, the prepared 8x8 data is reflected in the corresponding position of this frame image in the following manner. When the true foreground flag is set, 8x8 1s are prepared, and when it is not set, 8x8 0s are prepared, and each 4x4 pixel data at the corresponding small region of 8x8 pixels in the image frame memory and each Take the logical product (AND) and store it in the same location. As a result, when the 8 × 8 data are all 0, the four 4 × 4 pixel regions at the corresponding positions are zero. On the other hand, when 8x8 data is 1, the 1/0 state of four 4x4 areas remains as it is.

Subsequently, 0 or 1 corresponding to the small area corresponding to the true foreground flag is sequentially prepared at the position of the true foreground image frame corresponding to the true foreground flag received from the shadow / reflection area removing unit 40, and the true foreground is obtained by logical product. The operation of changing and returning the value of the image frame area is repeated. When the processing in the image frame by the logical product up to all the maximum pixel areas is completed, a complete true foreground image is completed. In other words, the location of the 4x4 region where 1 appears in this frame survives only when all the pixels from the 8x8 pixel region including the location to the maximum pixel region are all 1s. Therefore, a highly stable authentic foreground image frame is completed.

FIG. 2 shows an embodiment of the multiple division feature value generation unit 20. The multiple division feature value generation unit 20 includes an input terminal set 200 and a WPP sequence 210 from the camera 10. The input terminal set 200 includes an R component signal input terminal 201, a G component signal input terminal 202, and a B component signal input terminal 203, and the WPP column 210 includes three WPPs 211, 21WPP212, and WPP213. Here, as described above, WPP211, WPP212, and WPP213 are multi-resolution foreground separation LSI-based processors WPP (Walsh Parameter Processor) by the mixed Gaussian model of Non-Patent Document 2. In WPP, when a luminance signal is input, the frame image is divided into multiple regions and the luminance signal of each region is Walsh converted, and the lowest spectral coefficient f (DC), which is the average luminance of each region, and the spectrum in the vertical and horizontal directions F (ACV) and f (ACH) obtained by weighting the components are sequentially output. However, only f (DC), which is the average output of the region, is used for this purpose, and f (ACV) and f (ACH) are not used.

R component signal, G component signal, B component signal from camera 10 are input to R component signal input terminal 201, G component signal input terminal 202, B component signal input terminal 203, and WPP211, WPP212 and Supplied to WPP213. In each WPP, the lowest spectral coefficient by Walsh transform is output to the f (DC) output terminal of each WPP for each of the multiple divided small regions from the 4 × 4 pixel region to the maximum pixel region. That is, average component signals of R color component, G color component, and B color component are output. Specifically, f (DC) of WPP 211 is f (R), f (DC) of WPP 212 is f (G), and f (DC) of WPP 213 is f (B). Therefore, the outputs from WPP 211, WPP 212, and WPP 213 sequentially output feature quantity elements for each of the small areas divided by the multi-resolution, and sequentially convey these as feature quantities to the intermediate foreground processing unit 30.

FIG. 3 is an operation flowchart of the shadow / reflection area removing unit 40. The operation flowchart of FIG. 3 includes an input data arrangement block 301, a true foreground flag inspection block 302, a shadow / reflection removal execution inspection block 303, a reference feature element determination block 304, a shadow / reflection verification element setting block 305, and a shadow / reflection verification standard. Element exclusion block 306, reference Gaussian distribution start block 307, reference correction coefficient block 308, non-reference element Gaussian distribution verification start block 309, shadow / reflection candidate verification block 310, shadow / reflection determination block 311, non-reference element Gaussian distribution verification An end inspection block 312, a reference Gaussian distribution inspection end inspection block 313, a feature element change block 314, and a foreground separation element processing end inspection block 315 are included. It is assumed that a mixed Gaussian model consisting of M Gaussian distributions is used for each feature element. Among them, the background Gaussian distribution has PM reference feature elements and non-reference feature elements. Assumes QM. This flowchart operates in accordance with the principle shown in FIG. 6 for detecting shadows and reflections described above as described below.

First, in the flowchart of FIG. 3, data collection from the mixed Gaussian foreground processing unit 30 is performed in the input data reduction block 301. The collected data is the background Gaussian distribution coefficient before correction for each small area, three feature elements, and an intermediate foreground flag. Further, the contents of the intermediate foreground flag are moved to the genuine foreground flag. This is to prevent the processing with the shadow / reflection area removing unit 40 from interfering in the subsequent processing.

Next, the genuine foreground flag check block 302 checks whether the true foreground flag is set. If it is not set, it is a background and is not subject to shadow / reflection removal. Therefore, in this case, the process directly goes to the end of this flowchart. If the true foreground flag is set, the process proceeds to the shadow / reflection removal execution inspection block 303 in order to detect shadow / reflection.

In the shadow / reflection removal execution inspection block 303, shadow / reflection removal is not performed when the small area division is other than a small area of 4 × 4 pixels to 8 × 8 pixels. The symbol written in the block means that the size of the small area is a, and that a is included in SES (Selected Evaluation Sizeblock: 4 × 4 pixels or 8 × 8 pixels). For this reason, in a divided area larger than 8 × 8 pixels, the operation flow of the shadow / reflection area removing unit described below is bypassed and the process ends. Therefore, the process proceeds to the reference feature quantity element determination block 304 only when the small area size belongs to SES.

In a reference feature amount element determination block 304, a foreground separation feature amount element that is a reference in determining shadow / reflection removal is determined. For this reason, the foreground separation feature quantity element having the maximum value is selected. For convenience of the following explanation, f (R), f (G), f (B) are numbered in this order to be f (1), f (2), f (3). Will be shown. The value of the maximum feature amount element is detected by the maximum value detection function Max and is set to f (N). That is, the element number to which the maximum element belongs is described as N.

In the subsequent shadow / reflection verification element setting block 305, feature elements to be combined with the reference element for inspecting the shadow / reflection area are sequentially determined, and the following processing is performed by loop processing. Three loops are sequentially examined for the feature quantity elements. The process proceeds with the feature quantity element set here as the kth element.

Next, the process proceeds to the shadow / reflection verification reference element exclusion block 306. The shadow / reflection process is performed using one of the reference elements and the other foreground separation feature elements. Therefore, it is necessary to select a feature element different from the reference element determined in the shadow / reflection verification element setting block 305. Therefore, it is checked whether or not the feature quantity element set in the shadow / reflection verification element setting block 305 is a reference element. If the same feature quantity element as the reference element is set, the process proceeds to the feature quantity element change block 314. Prepare the (k + 1) th feature element. Next, in the foreground separation element processing end check block 315, when the Gaussian distribution whose (k + 1) -th is the fourth or less is designated, the process returns to the shadow / reflection verification element setting block 305. If k = 4 in the feature quantity element change block 314, there are three feature quantity elements, so the process proceeds to the end of the operation flowchart.

When the reference element and other feature quantity elements are found in the shadow / reflection verification reference element exclusion block 306, the process proceeds to the reference Gaussian distribution investigation start block 307. Here, since the Gaussian distribution that originally included the reference element cannot be understood, a loop setting is performed so that the background Gaussian distribution belonging to the reference element is sequentially called to obtain a correction coefficient by a loop process. The reference element is assumed to have PM background Gaussian distributions, and an operation for deciding the background Gaussian distribution that should originally include the reference feature quantity element is examined by loop processing. In the following, it is assumed that an inspection is performed using the current p-th Gaussian distribution.

In the subsequent reference correction coefficient block 308, the p-th correction coefficient A (p) commonly used for shadow / reflection is calculated. This correction coefficient is obtained by dividing the mean μN (p) of the p-th background Gauss by the value f (N) of the reference element, and is described as A (p) = μN (p) / f (N).

In the next non-reference element Gaussian distribution verification start block 309, QM elements are not included in the feature elements that are not the reference elements determined by the shadow / reflection verification element setting block 305 using the correction coefficient obtained in the reference correction coefficient block 308. The background Gaussian distribution number q is set for executing verification for each of the background Gaussian distributions by loop processing.

Subsequently, in the shadow / reflection candidate verification block 310, this element is obtained by multiplying the non-reference feature quantity element selected in the shadow / reflection verification element setting block 305 by the correction coefficient A (p) obtained in the reference correction coefficient block 308. It is determined whether it is included in the qth background Gaussian distribution to which it belongs.

If the shadow / reflection candidate verification block 310 determines that it is a shadow / reflection, the process proceeds to the shadow / reflection determination block 311 and the true foreground flag is set. Since this operation is sufficient to determine that one of the verification processes using two feature elements other than the reference element to be inspected by the shadow / reflection test is a shadow / reflection, the shadow / reflection decision block As soon as the processing of 311 is completed, the process proceeds to the end of this flowchart.

Conversely, if it is not included in the background Gauss currently being processed in the shadow / reflection candidate verification block 310, the process proceeds to the non-standard Gaussian distribution verification end inspection block 312. If there is a remaining background Gaussian distribution, the (q + 1) th The non-reference element Gaussian distribution verification end check block 312 returns to the non-reference element Gaussian distribution verification start block 309, and the shadow / reflection area check loop processing starts again.

However, if there is no remaining background distribution that can include the corrected feature quantity element in the non-reference element Gaussian distribution verification end inspection block 312, the corresponding background Gaussian distribution cannot be found with the correction coefficient determined in the reference correction coefficient block 308. For this reason, the process proceeds to the reference Gaussian distribution survey end inspection block 313.

If the reference gaussian distribution end inspection block 313 can determine that there is a next alternative to the background gaussian distribution of the reference element, that is, if the background gaussian distribution of the pth reference element is less than or equal to PM, the reference gaussian distribution Returning to the investigation start block 307, the (p + 1) th background Gaussian distribution is set, and the shadow / reflection area check loop processing is started. On the other hand, if all the background Gaussian distributions of the reference element have been checked in the reference Gaussian distribution end inspection block 313, the shadow / reflection area could not be checked in the Gaussian distribution of this reference element. The process proceeds to a foreground separation feature quantity element change block 314 for changing the process, advances k to select another feature quantity element, and proceeds to the foreground separation element processing end check block 315.

In the foreground separation element processing end check block 315, since there are only three feature elements, it is checked whether or not the result of stepping k is 4. In the case of 4 or less, the process returns to the shadow / verification element setting block 305 to perform an operation for finding a shadow / reflection area in the same manner as before for the new foreground separation feature quantity element.

However, if k is 4 in the foreground separation element processing end check block 315, the foreground separation element processing end check block 315 is passed and the operation according to the operation flowchart is ended. That is, in this case, the genuine foreground flag remains standing, meaning that it was not a shadow or reflection. Thus, the operation flowchart of the shadow / reflection area removing unit 40 is completed.

As described above, according to the above-described embodiment, a true foreground area from which shadows and reflection areas are removed indoors can be extracted, and retinex image enhancement, which has been conventionally required, can be omitted. As a result, the amount of operations such as fingertip gesture input is drastically reduced, and an input system for a wearable terminal with low power consumption can be realized.

The flowchart of FIG. 3 detects the reflection area and the shadow area at the same time, and deletes such an area from the intermediate foreground area. However, if only the reflection area is to be detected, the reference Gaussian distribution survey start block 307 is used. When the average of the reference Gaussian distribution selected in step (b) is larger than the value of the reference element, a condition for proceeding immediately to the reference Gaussian distribution survey end inspection block 313 may be set. By doing so, it is possible to remove only the reflection region, so that the calculation amount can be reduced to about ½.

Similarly, when only the shadow area is to be detected, if the average of the reference Gaussian distribution selected in the reference Gaussian distribution search start block 307 is smaller than the value of the reference element, the condition to proceed immediately to the reference Gaussian distribution check end inspection block 313 is set. What is necessary is just to provide. In this way, only the shadow area is removed, so that the amount of calculation can be reduced to about 1/2.

In addition, in the flowchart of the shadow / reflection area removal unit in FIG. 3, when verifying a shadow or reflection candidate in the intermediate foreground area, a certain feature element deviates from the inclusion area of any background Gaussian distribution among a plurality of background Gaussian distributions. Since it is impossible to determine whether the foreground has been obtained, the background Gaussian distribution was examined using a brute force method. However, as already mentioned, in the process of determining the background and foreground Gaussian distribution for the Gaussian distribution of each feature element, all the Gaussian distributions are rearranged in descending order of weighting factors, so that different feature elements are used. If the ranks from the Gaussian distribution with the largest weighting coefficient are the same, the corresponding background Gaussian distribution may be considered. However, it is only when the weighting factors of all the background Gaussian distributions are sufficiently large. If this condition is satisfied, the brute force method can be avoided. The specific correction of the flowchart of FIG. 3 in this case eliminates the loop processing composed of the non-standard element Gaussian distribution verification block 309 and the non-standard element Gaussian distribution verification end block 312, This can be realized by using the p-th non-standard element Gaussian distribution determined by the standard feature quantity element determination block 304 instead of the q-th non-standard element Gaussian distribution given by The method implemented in this way is also part of the present invention.

In the above, division to be an iterative operation is required to obtain the correction coefficient. However, as described in Non-Patent Document 1, the upper and lower limits of the area related to the inclusion area are f (N) serving as the denominator of the correction coefficient. Multiplication is used to eliminate division and reduce the amount of computation. This is also a mere formula modification and is within the scope of the present invention.

For reference, an example of processing according to the flowchart of FIG. 3 is shown in FIG. This photo book is composed of a photograph 701 obtained by binarizing one cut of the input color video, an intermediate foreground separation result photograph 702 using a multi-resolution color average block feature, a shadow area photograph 703, and a genuine foreground separation photograph 704. In addition, shadow processing and reflection processing are performed only in 4x4 pixel blocks. Photo 701, which is a binarized video picture, shows shadows but not reflections. However, in the foreground separation result photograph 702, shadows and reflections are also separated as an intermediate foreground area under the fingertip and arm. On the other hand, the result of extracting only the shadow area using the color average feature amount is the photograph 703, and the shadow area that is visible and the linear shadow area are found at the lower edge of the arm. However, even if only this is removed from the intermediate foreground, the triangular reflection area does not disappear. On the other hand, when the intermediate foreground separation result is processed using the operation flowchart of the shadow / reflection area removal unit shown in FIG. 3, a photograph 704 is obtained as the genuine intermediate foreground separation result. That is, the triangular reflection region can be removed. By the way, this triangular reflection was from a smartphone placed on a desk. In the present invention, a true foreground separation result that does not include shadows and reflections is obtained with a calculation amount far below 10% of the calculation amount of Retinex image enhancement required for shadow removal of Non-Patent Document 1.

As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

10: Camera
20: Multiple division feature generator
30: Mixed Gaussian foreground processing section
40: Shadow / reflection area removal part
50: Authentic foreground image generator
200: Input terminal set
201: R component signal input terminal
202: G component signal input terminal
203: B component signal input terminal
210: WPP column
211: WPP (Walsh Parameter Processor)
212: WPP (Walsh Parameter Processor)
213: WPP (Walsh Parameter Processor)
301: Input data reduction block
302: Authentic foreground flag check block
303: Shadow / reflection removal execution inspection block
304: Reference feature element determination block
305: Shadow / reflection verification element setting block
306: Shadow verification reference element exclusion block
307: Standard Gaussian distribution start block
308: Standard correction coefficient block
309: Non-standard element Gaussian distribution verification start block
310: Shadow / reflection candidate verification block
311: Shadow / reflection decision block
312: Non-standard element Gaussian distribution verification check block
313: Standard Gaussian distribution survey end inspection block
314: Foreground separation feature element change block
315: Foreground separation element processing end check block
401: Shadow region graph of f (DC) element in foreground
402: Other feature element graph
403: Shadow validation graph
501: Original picture photo
502; Transform area middle foreground separation photograph
503: Transform area shadow removal photo
601: Base area shadow area graph
602: Other elements graph of shadow area
603: Shadow verification graph
604: Reference element reflection area graph
605: Other elements graph of reflection area
606: Reflection verification graph
701: Photo of one cut of color video binarized
702: Photo for intermediate foreground separation by multi-resolution color average block feature
703: Shadow area photo by multi-resolution conversion area feature
704: Authentic foreground separation result photo

Claims

The input frame from the camera is divided into small areas, and the color average component signal obtained by calculating the average of the color components in the small area is used as a feature quantity element, and it is matched to the probabilistic variation of each feature quantity element. Extracting an intermediate foreground small region by comparing it with a feature value while modeling the small region with one or more typical probability distributions, and an intermediate value extracted by the mean and variance of the feature value element and the background probability distribution An image processing apparatus comprising: a step of identifying and removing a region where a foreground region has become a shadow or reflection and obtaining a true foreground region, and extracting a true foreground region which does not include a foreground portion due to shadow or reflection.
For each element of the feature quantity, the probability distribution of the variation is approximated by multiple typical probability distributions, and distinguished from the typical probability distribution modeling the foreground and the typical probability distribution modeling the background. The step of extracting the intermediate foreground separation as described in 1 above, wherein the foreground small area is defined as a case where at least one element of the quantity is included in a typical probability distribution modeling the foreground, and the foreground separation feature as an intermediate small area Selecting the largest of the quantity elements as a reference element, determining a correction factor by dividing the average of a typical probability distribution representing each background assigned to the reference element by the value of the reference element, An intermediate foreground area, where a value obtained by multiplying the value of a feature element that is not a reference element by a correction factor is included in one of the probability distributions representing the background of the feature element, and is regarded as an intermediate foreground small area due to shadow or reflection 2. An image processing apparatus for extracting a genuine intermediate region excluding the shadow / reflection region according to 1 above, comprising the step of creating a true foreground region by excluding it from the region.
Divide the frame image into small areas that do not overlap first, then double the size and width of the small areas arranged in a mesh, and then cover the screen with an enlarged small area that contains four previously divided small areas. And performing multiple division by repeating the division for creating the enlarged small region a plurality of times, performing the foreground separation on each of the multiple divided small regions, and shadow / reflection from the intermediate foreground separation of the small region dividing unit A step of removing the region to make a foreground region without shadow / reflection, and a minimum foreground region without shadow / reflection, and all the enlarged regions including the minimum foreground region without shadow / reflection are all foreground regions without shadow / reflection Alternatively, the image processing apparatus according to the first aspect, wherein final foreground separation is performed by setting an intermediate foreground small area as a true foreground small area.
When only the intermediate foreground part due to reflection is excluded from the intermediate foreground region in the above 3, the reference feature amount element obtained by the method of obtaining the reference feature amount element in the above 3 is larger than the average of the background probability distributions. Means for obtaining a correction coefficient only, means for multiplying a feature quantity element other than the reference feature quantity by a correction coefficient, and eliminating the intermediate foreground portion in which the multiplied feature quantity element is included in the background probability distribution of the feature quantity element. An image processing apparatus that performs genuine foreground separation.
When only the intermediate foreground portion due to the shadow is excluded from the intermediate foreground area in 3 above, from the typical background probability distribution in which the reference feature value element obtained by the method of obtaining the reference feature value element in 3 is smaller than the average of the background probability distributions. Only a means for obtaining a correction coefficient, means for multiplying a feature quantity element other than the reference feature quantity by a correction coefficient, and eliminating the intermediate foreground small area in which the multiplied feature quantity element is included in the background probability distribution of the feature quantity element. An image processing device that performs genuine foreground separation.
6). 1 to 5 above, between the typical background probability distribution arranged in descending order of the weighting coefficient of the typical probability distribution to which the reference element belongs and the typical background probability distribution arranged in descending order of the weighting coefficient of the non-reference feature quantity. Thus, the image processing apparatus obtains the true foreground region by verifying the intermediate foreground region caused by the shadow or background using only the typical background probability distribution of the same rank.