WO2008018460A1 - Image processing method, image processing apparatus, image processing program, and image pickup apparatus - Google Patents

Abstract

An image processing method comprises acquiring an image consisting of a plurality of pixels; detecting, based on the acquired image, the edge of a recess the pixel values of which are locally smaller than those of the periphery of the recess; and generating an edge image based on the detected edge of the recess.

Description

 Specification

 Image processing method, image processing apparatus, image processing program, imaging apparatus

 The present invention relates to an image processing method and an image processing apparatus for performing edge detection in an acquired image.

The present invention relates to an image processing program and an imaging apparatus.

 Background art

 [0002] In digital image processing, there is a high demand for processing for detecting a medium facial image of a captured image. For example, in a digital camera, a process that converts a detected face area into a preferred color or gradation, a process that extracts the appearance of a specific person in a video image, or a suspicious person image that is extracted in a surveillance camera There is processing. In addition, based on the photographed face image, individuals are identified, and gender, age, and facial expressions are estimated.

 [0003] In the face image determination process used in the above applications! /, Conventionally, high-frequency components of the input image are extracted to create an edge image, and learning such as a neural network is performed for the edge image. A method for performing a discrimination process has been proposed. By generating an edge image, it is possible to remove image information that is not necessary for face image determination processing, such as the influence of illumination conditions at the time of shooting, and the determination processing can be performed efficiently. As a method for extracting edge components, for example, a method for extracting information on the direction and frequency of the edge structure of the input image by using a Gabor filter having various directions and frequencies has been proposed.

 [0004] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2004-199386

 Disclosure of the invention

 Problems to be solved by the invention

[0005] However, it cannot be said that conventional edge extraction methods such as a Gabor filter can sufficiently extract edge structure information. For example, the positions of eyes, nose, mouth, etc. in the face image are locally darker than the surroundings. Therefore, when identifying facial images, it is important to know if there are locally dark structures at positions corresponding to the eyes, nose and mouth. Also, in the face image showing teeth and laughing, the teeth are locally brighter than the surroundings. Therefore, identify the smile It is important to know if there is a locally bright structure in the mouth position.

[0006] However, in the conventional edge extraction method, the edge structure is locally identified as 喑! /, A force that is a structure, a locally bright structure, or another structure. There was a problem that it was not possible.

 Means for solving the problem

[0007] According to the first aspect of the present invention, an image processing method acquires an image including a plurality of pixels, and based on the acquired image, an edge having a concave structure in which a pixel value is locally recessed from the periphery. An edge image is generated based on the detected edge of the concave structure.

 According to the second aspect of the present invention, in the image processing method according to the first aspect, it is preferable that the edge image is generated by calculating a nonlinear filter that detects the edge of the concave structure with respect to the acquired image.

 According to the third aspect of the present invention, in the image processing method according to the second aspect, the nonlinear finisher is based on a difference between a pixel value in the target region and a minimum value of the pixel values in the peripheral region of the target region! / It is preferable to output the calculation result!

 According to the fourth aspect of the present invention, in the image processing method according to the third aspect, the nonlinear finisher is based on a difference between a minimum pixel value in the target region and a minimum pixel value in the peripheral region. To output the calculation result!

 According to the fifth aspect of the present invention, in the image processing method according to the fourth aspect, when the minimum value of the pixel value in the target area is smaller than the minimum value of the pixel value in the peripheral area of the target area, according to the difference When the value is the value of the edge pixel and the minimum value of the pixel value in the target area is larger than the minimum value of the pixel value in the peripheral area of the target area, it is preferable to perform the clipping process with the value of the edge pixel set to zero.

 According to the sixth aspect of the present invention, an image processing method acquires an image composed of a plurality of pixels, and detects a convex structure edge in which pixel values protrude locally from the periphery based on the acquired image. Then, an edge image is generated based on the detected edge of the convex structure.

According to the seventh aspect of the present invention, in the image processing method according to the sixth aspect, it is preferable that the edge image is generated by calculating a nonlinear filter that detects the edge of the convex structure with respect to the acquired image. According to the eighth aspect of the present invention, in the image processing method according to the seventh aspect, the nonlinear filter is based on the difference between the pixel value in the target region and the maximum value of the pixel value in the peripheral region of the target region! / It is preferable to output the calculation result!

 According to the ninth aspect of the present invention, in the image processing method according to the eighth aspect, the nonlinear filter is based on a difference between the maximum pixel value in the target region and the maximum pixel value in the peripheral region. To output the calculation result!

 According to the tenth aspect of the present invention, in the image processing method according to the eighth aspect, when the maximum value of the pixel values in the target area is larger than the maximum value of the pixel values in the peripheral area of the target area, the difference is determined according to the difference. When the value is the value of the edge pixel and the maximum value of the pixel value in the target region is smaller than the maximum value of the pixel value in the peripheral region of the target region, the edge pixel value is preferably clipped to zero.

 According to the eleventh aspect of the present invention, in the image processing method according to any one of the first to tenth aspects, a luminance image based on a luminance component is generated based on the acquired image, and the generated luminance image is used. Thus, it is preferable to generate an edge image.

 According to a twelfth aspect of the present invention, in the image processing method according to any one of the third to fifth and eighth to tenth aspects, the target area is only one target pixel or the target pixel and its adjacent pixels. It is a 2-pixel area, and the surrounding area is preferably a 2-pixel area located on both sides of the target area.

 According to the thirteenth aspect of the present invention, in the image processing method according to any one of the second to fifth and seventh to tenth aspects, the nonlinear filter is preferably operated in at least two directions. Yes.

 According to the fourteenth aspect of the present invention, an image processing method acquires an image composed of a plurality of pixels, and on the basis of the acquired image, the edge of a concave structure that protrudes locally from the periphery and protrudes. The edge of the convex structure is detected, an edge image of the concave structure is generated based on the detected edge of the concave structure, and an edge image of the convex structure is generated based on the detected edge of the convex structure.

According to the fifteenth aspect of the present invention, an image processing method acquires an image composed of a plurality of pixels, and based on the acquired image, the edge of a concave structure in which pixel values are locally recessed from the periphery. According to the sixteenth aspect of the present invention, an image processing method acquires an image composed of a plurality of pixels, and based on the acquired image, the edge of the concave structure where the pixel value is locally recessed from the periphery and locally At least one of the edges of the convex structure whose pixel value protrudes from the periphery is detected, and an edge image is generated based on at least one of the detected concave structure edge and convex structure edge.

 According to the seventeenth aspect of the present invention, an image processing method acquires an image composed of a plurality of pixels, detects an edge component of the acquired image, performs gamma conversion on the detected edge component, and performs gamma conversion. An edge image based on the edge component is generated.

 According to the eighteenth aspect of the present invention, in the image processing method according to any one of the first to seventeenth aspects, it is preferable to detect a face image using the generated edge image.

 According to a nineteenth aspect of the present invention, the image processing program is an image processing program for causing a computer to execute the image processing method according to any one of the first to eighteenth aspects. According to the twentieth aspect of the present invention. The image processing apparatus is an image processing apparatus equipped with the image processing program of the nineteenth aspect.

 According to a twenty-first aspect of the present invention, an imaging apparatus is an imaging apparatus that mounts the image processing program of the nineteenth aspect.

 The invention's effect

 [0008] Since the present invention is configured as described above, an edge can be accurately detected. For example, since the edge of the concave structure is detected in claim 1, it is possible to accurately detect a local crease or structure such as an eye of the face image as an edge.

 Brief Description of Drawings

 FIG. 1 is a diagram showing an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram showing a flowchart of an image processing program executed by the personal computer 1. FIG. 3 is a diagram showing an edge extraction target pixel and peripheral pixels with coordinates xy.

 FIG. 4 is a diagram showing the result of creating a luminance concave image (x, y) for various luminance structures.

 [Figure 5] Four types of edge images E (x, y) to E (x, y) are generated for a specific facial luminance image

 14

 FIG.

 [Fig.6] Creates a specific edge image! /, Generates facial appearance V (x, y), and calculates facial appearance V

 It is a figure which shows the example which performed the process which issues 1SUM1.

 [Fig.7] A specific value of the lookup table L (E) is shown for each edge size.

 1 (x, y)

 is there.

 FIG. 8 is a diagram showing a flowchart of the processing after obtaining the face-likeness Vsum to Vsum of the partial image in the face determination processing in step S6 of FIG.

 Four

 FIG. 9 is a diagram showing a flowchart of a process for obtaining facial appearance L (E).

 1 (x'y)

 FIG. 10 is a diagram showing a configuration of a digital camera 100 that is an imaging apparatus.

 BEST MODE FOR CARRYING OUT THE INVENTION

 [0010] First Embodiment

 FIG. 1 is a diagram showing an image processing apparatus according to an embodiment of the present invention. The image processing apparatus is realized by the personal computer 1. The personal computer 1 is connected to a digital camera 2, a recording medium 3 such as a CD-ROM, another computer 4, etc., and receives various images (image data). The personal computer 1 performs the image processing described below on the provided image. The computer 4 is connected via the Internet and other telecommunications lines 5.

 [0011] The program executed by the personal computer 1 for image processing is similar to the configuration shown in FIG. 1, such as a recording medium such as a CD-ROM, other computer power via the Internet or other electric communication line, and the like. Provided and installed in the personal computer 1. The personal computer 1 is composed of a CPU (not shown) and its peripheral circuits (not shown), and executes a program in which the CPU is installed.

[0012] If the program is provided via the Internet or other telecommunications line, the program may be connected to a telecommunications line, ie, a signal on a carrier wave carrying a transmission medium. Converted and sent. Thus, the program is supplied as a computer readable computer program product in various forms such as a recording medium and a carrier wave.

 [0013] The personal computer 1 according to the present embodiment performs image processing for detecting a medium facial image of a captured image. Specifically, an edge component is extracted based on the input image to generate an edge image, and it is determined whether there is a face image based on the generated edge image. The processing in the present embodiment is characterized by the edge component extraction method and the face determination method based on the edge image.

 In the following, the expression “image processing is performed on an image” is used, but in practice, this means that image processing is performed on input image data. Also, in this embodiment, an edge is a portion (area, pixel) where the luminance value or pixel value is smaller than the surrounding area, is larger than the surrounding area! /, And the value is protruding! / This refers to locations (regions, pixels) that are (protruding), and locations (regions, pixels) that are not leveled. In particular, the indented area (area, pixel) is called a concave edge, and the protruding area (area, pixel) is called a convex edge.

 [0015] Hereinafter, image processing for detecting a medium face image captured by the personal computer 1 according to the present embodiment will be described in detail. FIG. 2 is a diagram showing a flowchart of an image processing program executed by the personal computer 1.

 [0016] In step S1, an image (image data) to be detected for a face photographed (captured) with a digital camera or the like is input (acquired). Each pixel of the input image includes R, G, and B color components, and each color component ranges from 0 to 255. In step S2, a luminance image Y is generated by the following formula based on R, G, and B of the input image. That is, the luminance image Y plane is generated.

 Y = (R + 2G + B) / 4

[0017] In step S3, the generated luminance image is hierarchically reduced and output. For example, given by the reduction ratio _κ of 0.9 ⁿ for integer n of 0 to 3 1, and outputs the luminance image which has been reduced by the reduction magnification κ of the 32 patterns. As a reduction method, for example, Cubic scaling or linear scaling may be used. The reason why multiple reduced images are generated in this way is that it is unclear whether the input image has a face image of any size, so that it can handle face images of any size. . [0018] In step S4, four types of edge images E (x, y) to E (x, y) are generated by the following procedure for each reduced luminance image Y (x, y) force. In the following, the x direction is the horizontal direction of the image

14

 Or the horizontal direction and the y direction are vertical or vertical.

 [0019] First, an image Y (x

 LV, y) and a horizontally smoothed image Y (x, y) are generated. To extract the edge component in the vertical direction, smooth the horizontal direction.

LH

 This is because, in order to extract the edge component in the horizontal direction using the processed image data, it is preferable to use the image data in which the vertical direction is smoothed.

 Y (x, y) = (Y (x, y-l) + 2XY (x, y) + Y (x, y + 1)} / 4

 LV

 Y (x, y) = (Y (x-l, y) + 2XY (x, y) + Y (x + l, y)} / 4

 LH

 [0020] Next, using the image Y (x, y) smoothed in the horizontal direction, the vertical direction

 LH

 Di-image E (x, y) is generated. Each pixel of the edge image is called an edge pixel.

 E '(x, y) = Min (Y (x, y-l), Y (x, y + 2))

 1 LH LH

 -Min (Y (x, y), Y (x, y + l))

 LH LH

 E (x, y) = γ (E '(x, y))

 Next, a vertical edge image E (x, y) is generated from the following equation.

 2

 E '(x, y) = Y

 2 I (x, y-l) -Y (x, y) |

 LH LH

 + I Y (x, y + l) -Y (x, y) |

 LH LH

 E (x, y) = γ (E '(x, y))

 twenty two

 [0022] Next, using the image Y (x, y) smoothed in the vertical direction, the horizontal direction

 LV

_C to generate the image E (x, y)

 Three

 E (x, y) = Min (Y (x—l, y), Y (x + 2, y))

 3 LV LV

 Min (Y (x, y), Y (x + 1, y))

 LV LV

 E (x, y) = γ (E '(x, y))

 3 3

 Next, a lateral edge image E (x, y) is generated from the following equation.

 Four

 E '(x, y) = I Y (x-l, y) -Y (x, y) |

 4 LV LV

 + I Y (x + l, y) -Y (x, y) |

 LV LV

 E (x, y) = γ (E '(x, y))

 4 4

Here, Min () is a function that returns the minimum value of 0. In addition, Ί (E) is This is a clipping function that performs the following operations and outputs an integer between 0 and 31. This Ml N () process is a non-linear filter process. It can also be called non-linear filter processing, including γ conversion and clipping processing.

 If Ε <0 γ (Ε) = 0

Ε> 63 ( _Ί ) = 31

 When 0≤Ε≤63 γ (E) = (int) (4 Χ ^ Ε)

[0025] Generation of the edge image will be described in more detail with reference to FIG. FIG. 3 is a diagram in which the edge extraction target pixel and the surrounding pixels are represented by coordinates xy. The above E ′ (x, y) is 4 pixels in the vertical direction Y (x, y— 1), Y (x, y), Y (x, y + l),

LH LH LH LH

Out of Y (x, y + 2), the outer two pixels Y (x, y— 1), Y (x, y) with reference to the target pixel ( _x , y)

LH LH LH

 The difference between the minimum value of +2) and the minimum value of the inner two pixels Y (x, y) and Y (x, y + l) is obtained.

 LH LH

 [0026] The positive value of E ′ (x, y) indicates that the value near the target pixel (x, y) is smaller than the value in the vertical peripheral pixel, that is, the pixel value is in the vertical direction. Indicates that it is dented from the surroundings. Therefore, the value of E (x, y) generated in this way is treated as a pixel value, and the generated image is called a vertical luminance concave image.

 [0027] The above E '(x, y) is adjacent to the target pixel (x, y) in the vertical direction on the luminance image Y (x, y) plane.

 2 LH

 A value obtained by adding a difference in luminance value with the adjacent pixel is shown. That is, a large value is generated when the luminance value changes greatly between the adjacent pixels in the vertical direction. Therefore, the value of E (x, y) generated in this way is treated as a pixel value, and the generated image is treated as a vertically adjacent pixel.

 2

 This is called a difference image. The vertically adjacent pixel difference image detects the edge of the concave structure, the edge of the convex structure, and the edge of the step without distinction.

 [0028] The above E '(x, y) and E (x, y), E' (x, y) and E (x, y) are horizontal edge images

 3 3 4 4

 Is for generating. E '() selection £ (x, y), E' () selection £ (x,

 For 1 1 2 2 y), the vertical and horizontal directions are reversed, and the rest is calculated in the same way. Therefore, E (x, y) generated in this way is the horizontal luminance concave image, and E (x, y) is the horizontal adjacent image.

 3 4

 This is called an elementary difference image.

FIG. 4 is a diagram showing the result of creating the luminance concave image E (x, y) for various luminance structures. Fig. 4 (a) shows the case where the luminance is concave, and Fig. 4 (b) shows the case where the luminance is protruding. Fig. 4 (c) shows the case where the brightness is stepped. As can be seen from Fig. 4, the luminance concave image has a positive value only when the luminance is concave. Therefore, if the negative value of the luminance recess image E ′ is clipped to 0, an edge image E (x, y) that reacts only to the luminance recess is generated.

 [0030] According to this luminance concave image, it reacts particularly well to locally dark places such as the eyes and nose and mouth.

 Fig. 5 shows the four types of edge images E (x, y) to E (x, y

 14

FIG. In fact, the luminance depression image has a sharp peak at the position of the eyes and nose! In particular, in the vertical luminance concave image E in FIG. 5, it reacts to the eyes, nostrils, mouth, etc. Among them, it reacts strongly to the eyes, nostrils, etc. and becomes white. In other words, the value of E at that position is large. Therefore, the face can be detected with high accuracy by analyzing such a luminance concave image. However, it is desirable to use not only the luminance concave image but also the edge image created by the conventional method.

 The reason why the edge image is gamma-converted is to convert the edge amount into an appropriate feature amount E. In image analysis, a subtle difference in the edge amount in a place where there is almost no edge has a larger meaning than a slight difference in the amount of edge in a place where there is a large edge! /. By applying gamma conversion to the edge amount, the above effect is achieved. Almost no edge! / The difference in edge amount at the location is converted into a large difference in feature amount E. The difference in edge amount is converted into a small difference in feature amount E.

 Next, returning to FIG. 2, in step S5, a face determination target area of 19 × 19 pixels is set for every other pixel of the reduced image, and a partial image of the edge image in that area is output. This is performed for all reduced images. The 19 x 19 pixel face detection target area is suitable for detecting the eyes, nose, mouth, etc. with about 2 pixels when the area is a face.

 In step S 6, it is determined for each partial image of the edge image output in step 5 whether this area is a face image. In the present embodiment, the determination of the face image is performed by the method described below.

[0034] First, for each pixel position (x, y) (0≤x≤18, 0≤y≤18) of the partial image of the edge image E (x, y), the face of that position is based on the following equation: Produces the likelihood V (x, y). Facialness V ( x, y) is a numerical expression of the facial appearance at each pixel position, and indicates the degree and degree of facial appearance. V (x, y) may be said to be a likelihood representing a degree of likelihood as a face.

 V (x, y) = L (E (x, y))

 1 1 (x, y) 1

 Here, L (E) is described later for each pixel position (x, y) (0≤x≤18, 0≤v≤18).

 1 (x, y)

 This is a look-up table created in advance by statistical processing.

, Y) represents the face likeness when the edge E (x, y) is E.

Then, the generated facial appearance V (x, y) is integrated for all pixels (x, y) (0≤x≤18, 0≤y≤18) to calculate the facial appearance V.

 SUM1

 FIG. 6 is a diagram illustrating an example in which the above processing is performed on a specific edge image. In the face-like image in Fig. 6, parts that look like faces are displayed in white, and parts that do not look like faces are displayed in black. The face-like image generated from the face edge image shown in Fig. 6 (a) has a large overall value. That is, the overall image is whitish. However, the facial image generated from the non-facial edge image shown in Fig. 6 (b) has small values in some places. That is, the image becomes dark in some places.

 [0037] In the non-face example shown in Fig. 6 (b), the area corresponding to both sides of the eyes, the nose, and the mouth does not look like a face. In the face-like image, the pixel value of that area is a small value. Next black! Therefore, the value V obtained by integrating all the pixels of the face-like image of the non-face image is a small value.

 SUM1

 [0038] FIG. 7 shows specific values of the lookup table L (E) for each edge size.

 1 (x, y)

 FIG. In FIG. 7, the larger the face-like value, the more white it is displayed. In Fig. 7, the left side is the facial appearance when the edge is small, and the right side is the facial appearance when the edge is large.

. If all the values in the lookup table L (E) are illustrated,

 1 (x, y)

Is generated from 0 to 31, so 32 diagrams from L (0) to L (31) can be created.

 The In FIG. 7, only eight of them are shown for convenience of illustration.

[0039] Note that the look-up table L (E) in Fig. 7 has a specific value for each edge size.

 1 (x, y)

 It is the figure represented visually. Actually, a table of pixel values with the pixel position (X, y) as an argument is stored in the memory for each edge value. In other words, the table power S of the pixel value with 32 pixel positions (X, y) as an argument is stored in the S memory.

[0040] In Fig. 7, the diagram on the left represents the facial appearance when the edge is small. Looking at the diagram on the left, The facial appearance of the eyes, nose and mouth is small. This means that if the edges of the eyes, nose, and mouth are small, the area is not likely to be a face. For example, in the non-face example in Fig. 6 ω, the edge of the part corresponding to the nose is small, so the part does not look like a face.

 [0041] Further, the diagram on the right side of FIG. 7 represents the facial appearance when the edge is large. Looking at the figure on the right, the face-likeness of parts other than the eyes, nose, and mouth is small. This means that if the edge of a part other than the eyes, nose, or mouth is large, that part does not look like a face. For example, in the non-face example shown in Fig. 6 (a), the edge of the part corresponding to the space between the eyes and both sides of the mouth is large, so the part is not likely to be a face.

 [0042] That is, assuming that a face image is a specific type of image and eyes, nose, mouth, and the like are characteristic elements of the specific type of image, they correspond to characteristic elements of the specific type of image. At the pixel position, the degree of the particular kind of image when the edge component of the pixel is large is smaller than the degree of the particular kind of image when the edge component is small. In addition, at pixel positions corresponding to elements other than the characteristic elements of a specific type of image, the degree of a particular type of image when the edge component of that pixel is large is expressed as the degree of a particular type of image when the edge component is small V. As a small value compared to the degree of!

 [0043] When the process of referring to the lookup table is arranged, first, in the partial image of the edge image E (x, y), the value of the edge of x = 0, = 0 is obtained. Next, look-up table L (E) corresponding to the value of edge E is selected from 32 look-up tables.

 1 (x, y) 1

 Decide. When the lookup table L (E) is determined, this lookup table L

 1 (x, y) 1 1 vx.y)

The value of the pixel position (0, 0) of (E) is obtained. This is the faceness value at the pixel position (0, 0) of the edge image E (x, y). This process is sequentially performed until the pixel power of x = 0, y = 0, and the pixels of x = 18, y = 18, and the face-like image V (x, y) is obtained. Then, Vsum is obtained by summing all V (x, y).

 [0044] Through the above processing, the facial appearance Vsum of the partial image is generated based on the edge image E (x, y). Then, based on the edge images E (x, y) to E (x, y), the facial image Vsu

 twenty four

 The process of generating m to Vsum is performed in the same way.

 twenty four

[0045] FIG. 8 shows the face-likeness Vsum of the partial image in the face determination process of step S6 of FIG. It is a figure which shows the flowchart of the process after calculating | requiring Vsum. Face determination process in step S6

Four

 In fact, as explained above, facial appearance Vsum to Vsum are generated step by step,

 14

 If the integrated evaluation value is larger than the threshold value, the face is determined. However, the process of comparing the evaluation value with the threshold value is performed at each stage as shown in Fig. 8, so that images that are clearly not faces are excluded at an early stage and at a stage so that efficient processing can be performed. RU

 First, in step S 11, an evaluation value for determining whether or not the partial image is a face image is set as the face likelihood Vsum of the edge image E (x, y). In step S12, it is determined whether or not the evaluation value is larger than a predetermined threshold value th. If this evaluation value is larger than the threshold value th, the process proceeds to step S13. If this evaluation value is not larger than the threshold value thl, the partial image is a face image. If it is not an image, the face determination process for the target partial image is terminated.

 [0047] In step S13, the evaluation value is changed to the evaluation value in step S11, and the face appearance of the edge image E (x, y) is displayed.

 2

 Let Vsum be the value added. In step S14, this evaluation value is greater than a predetermined threshold th2.

 2

 If the evaluation value is greater than the threshold value th2, the process proceeds to step S15.If the evaluation value is not greater than the threshold value th2, the face determination process for the target partial image is performed assuming that the partial image is not a face image. finish.

[0048] In step S15, the evaluation value is changed to the evaluation value in step S13, and the face appearance of the edge image E (x, y) is set.

 Three

 Let Vsum be the value added. In step S16, this evaluation value is greater than a predetermined threshold th3.

 Three

 If the evaluation value is greater than the threshold th3, the process proceeds to step S17.If the evaluation value is not greater than the threshold th3, the partial image is determined not to be a face image, and the face determination process for the target partial image is performed. finish.

[0049] In step S17, the evaluation value is changed to the evaluation value in step S15, and the facial appearance of the edge image E (x, y) is displayed.

 Four

 Let Vsum be the value added. In step S18, this evaluation value is greater than a predetermined threshold th4.

 Four

 Judge whether or not. If the evaluation value is larger than the threshold th4 in step S18, it is finally determined that the partial image is a face image. If this evaluation value is not greater than the threshold th4, the partial image is not a face image, and the face determination process for the target partial image is terminated.

[0050] The partial image face determination process described above is performed for each partial image shifted by 1 bit in each reduced image, and all partial images that can be determined as face images are extracted. Proceed to step S7.

 In step S 7, when it is determined in step 6 that a partial image is a face, the face size S and coordinates (X, Y) for the input image of the partial image are output. S 1, X, and Y are given by the following expression using the size S ′ = 19 of the face in the reduced image, the coordinates (Χ ′, Υ ′) of the face area and the reduction magnification κ.

 S = S '/ κ

 X = X '/ κ

 Υ = Υ '/ κ

 As described above, when there is a face image in the input image, the position and size of the face image are detected and output.

[0053] <Statistical processing>

 Next, the statistical processing described above will be described. That is, the edge Ε of pixel position (x, y)

A method for determining the facial appearance L (E) of the pixel when (x, y) is E will be described. Figure 9

 1 (x, y)

 FIG. 10 is a diagram showing a flowchart of processing for obtaining the facial appearance L (E). This process is

 1 (x, y)

 It is executed on the personal computer 1.

 In step S 21, images of several hundred or more faces are acquired. That is, several hundred or more faces are photographed (captured) with a digital camera or the like, and the images (image data) are acquired. The acquired image is an image composed of the same color components as the image input in step S1 of FIG. In step S 22, the image of the face imaged is scaled so that the size of the face area becomes 19 × 19 pixels, and the partial images obtained by cutting out the face area are set as face image sample groups.

 [0055] In step S23, several hundred patterns of non-face image sample groups of 19 X 19 pixels are acquired. This is appropriately extracted from images other than the face photographed with a digital camera and made into a non-face image sample group. It is also possible to extract from an image showing a face while avoiding the face area. In this case, the user may appropriately designate the non-face image area from the image captured on the monitor.

[0056] In step S24, an edge component is extracted from the face image sample group to generate a face edge image sample group. This process is the same as the process for generating the edge image E (x, y) in the face detection process. In step S25, edge components are extracted from the non-face image sample group. And a non-face edge image sample group is generated. This process is also performed in the same manner as the process for generating the edge image E (x, y) in the face detection process.

[0057] In step S26, the frequency P (x, y, E) at which the edge of (x, y) becomes E is obtained for the face edge image sample group! In other words, the image power with a pixel (X, y) value of E ^

 Count. In step S27, for the non-face edge image sample group, (x, y

The frequency P (x, y, E) at which the edge of) becomes E is obtained.

 Non-face

 In step S28, the facial appearance L (E) of the pixel when the edge E (x, y) at the pixel position (x, y) is E is calculated by the following equation.

 1 (x, y)

 L (E) = log {(P (χ, γ, Ε) + 8) / (Ρ (χ, γ, Ε) + ε)}

 1 (x, y) face 1 nonface 2

 Where ε and ε are predetermined constants, introduced to suppress logarithmic divergence and overlearning.

 1 2

 is doing. The value of ε should be set to about 1/1000 of the average value of P (x, y, E).

 1 face

 The value of ε should be set to several tens of times the value of ε.

 twenty one

 [0059] In the above equation (Ε), log {(P (χ, γ, Ε) + ε)} is a monotonically increasing function.

 1 (x, y) face 1

 And log {l / (P (χ, γ, Ε) + ε)} is a monotonically decreasing function. That is, facial appearance L Non-face 2

 (Ε) shows an increase in the distribution of face image samples whose edge E (x, y) is E at the pixel position (x, y).

1 vx.y) 1

 The value increases monotonously in the direction of addition, and the value decreases monotonously in the direction of increase in the distribution of non-face image samples whose edge E (x, y) is E at the pixel position (x, y). It is a function that does. The distribution of face image samples whose edge E (x, y) is E at the pixel position (x, y), and the non-face whose edge E (x, y) is E at the pixel position (x, y) The distribution of image samples is usually normal.

 [0060] A look-up table L (E that converts the edge images E (x, y) to E (x, y) into facial appearance

 2 4 2 (x, y)

) To L (E) to generate the edge component extraction process in steps S24 and S25 above.

4 (x, y)

 In the same way as the processing for generating edge images E (x, y) to E (x, y) in face detection processing

 twenty four

 do it.

 When the processing of the first embodiment described above is performed, the following effects are obtained.

(1) The positions of eyes, nose, mouth, etc. in the face image are locally darker than the surroundings. In the conventional edge extraction method, it was impossible to distinguish between a force having a locally dark structure, a force having a locally bright structure, or a force having another structure. But above Thus, by detecting the edge of the concave structure and generating a concave image that is an edge image, it is possible to appropriately extract the eyes, nose, mouth, and the like that are locally dark structures of the face image. As a result, the face image can be accurately determined.

[0062] (2) According to the luminance concave image, it reacts particularly well to locally dark places such as the eyes and nose and mouth.

 In fact, the luminance concave image has a sharp peak at the position of the eyes, nose and mouth. Therefore, the face can be detected with high accuracy by analyzing such a luminance concave image. In this embodiment, it is possible to use an edge image created by a conventional method in addition to using only a luminance concave image, so that it is possible to determine the face with even higher accuracy! /. RU

[0063] (3) The reason why the edge edge is gamma-converted is to convert the edge amount into an appropriate feature amount E. In image analysis, a subtle difference in the edge amount in a place where there is almost no edge has a larger meaning than a slight difference in edge amount in a place where there is a large edge. By applying gamma conversion to the edge amount, the difference in the edge amount at the point where there is almost no edge is converted into a large difference in the feature amount E, and the difference in the edge amount at the point where there is a large edge is the feature amount E. Is translated into a small difference. As a result, the difference in edge amount matches the difference in image structure. As a result, the accuracy of face determination is increased.

 (4) As is clear from FIG. 4 of the above embodiment, it can be seen that the luminance concave image has a positive value only when the luminance is concave. Therefore, in this embodiment, the negative value of the luminance recess image E ′ is clipped to 0. As a result, an edge image E (xy) that reacts only to the luminance depression is generated, and the processing using the edge image E becomes frustrating.

 [0065] (5) A face image can be detected by a simple and high-speed process in which pixel values of an edge image are converted into facial appearance using a lookup table and integrated. Also, by determining the edge image, there is an effect of suppressing the influence of the lighting conditions when shooting the image.

[0066] Second Embodiment

In the first embodiment, an example has been described in which a luminance concave image is generated and a locally dark spot such as the eyes and nose of the face is appropriately determined. However, the brightness of the mouth laughing while showing teeth and the nose shining with light is locally brighter than the surroundings. Second In the embodiment, an example will be described in which a face image is detected more accurately than in the first embodiment by appropriately detecting a locally bright portion of the face.

 [0067] The second embodiment is realized by the personal computer 1 as in the first embodiment. Therefore, for the configuration of the image processing apparatus of the second embodiment, refer to FIG. 1 of the first embodiment. Further, the image processing program executed by the personal computer 1 is the same as the flowchart of FIG. 2 of the first embodiment, and the processing flow will be described below with reference to FIG.

 [0068] Steps S1 to S3 are the same as those in the first embodiment, and thus description thereof is omitted.

 [0069] In step S4, each of the reduced luminance images Y (x, y) force also generates six types of edge images E (x, y) to E (x, y). Image Y (x, y) smoothed in the vertical direction and smoothed in the horizontal direction

1 6 LV

 Generation of image Y (x, y) and edge images E (x, y) to E (x, y)

 LH 1 4

 Therefore, the description thereof will be omitted, and the edge images E (x, y) and E (X

 5 6

, Y) will be described.

 [0070] First, using the image Y (x, y) smoothed in the horizontal direction, the vertical direction

 LH

 Di-image E (x, y) is generated.

 Five

 E '(x, y) = Max (Y (x, y— 1), Y (x, y + 2))

 5 LH LH

 -Max (Y (x, y), Y (x, y + l))

 LH LH

 E (x, y) = γ (E '(x, y))

 5 5

 [0071] Next, using the image Y (x, y) smoothed in the vertical direction, the horizontal direction

 LV

_C to generate the image E (x, y)

 6

 E '(x, y) 2 Max (Y (x—1, y), Y (x + 2, y))

 6 LV LV

 -Max (Y (x, y), Y (x + l, y))

 LV LV

 E (x, y) = γ (E '(x, y))

 6 6

Here, Max () is a function that returns the maximum value among 0. 7 (E) is a function similar to that of the first embodiment. Clipping processing is also performed in the same way as in the first embodiment. The E (x, y) generated above is called the vertical luminance convex image, and E (x, y) is called the horizontal luminance convex image. [0073] For generation of the edge images E (x, y) and E (x, y), refer to FIG. 3 of the first embodiment.

 5 6

 This will be explained. E '(x, y) above is 4 pixels Y in the vertical direction on the luminance image Y (x, y) plane.

 5 LH

 Of (x, y— 1), Y (x, y), Y (x, y + 1), Y (x, y + 2), the target pixel (x, y)

LH LH LH LH

 Similarly, the maximum value of the outer two pixels Y (x, y— 1) and Y (x, y + 2) and the inner two pixels Υ (x, y), Υ

 LH LH LH L

 Find the maximum difference of (x, y + 1)!

 H

 [0074] The value of E ′ (x, y) indicates a positive value because the value near the target pixel (x, y)

 Five

 It is larger than the prime value, that is, the pixel value is more prominent than the periphery in the vertical direction. Therefore, E (x, y) generated in this way is referred to as a vertical luminance convex image.

 Five

 [0075] The above E '() selection (x, y) is for generating a lateral edge image.

 6 6

 . Think about E '() selection (x, y) by turning it upside down and the same

5 5

 It is to calculate. Therefore, E (x, y) generated in this way is used as the horizontal luminance convex image.

 6

 Say a statue.

 [0076] In step S5, a 19 x 19 pixel face determination target area is set for every other pixel of the reduced image, and partial images of edge images E (x, y) to E (x, y) in that area are set. Output. This

 1 6

 Do this for all reduced images.

In step S 6, it is determined for each partial image of the edge image output in step 5 whether or not this area is a face image, as in the first embodiment. Edge image E (X,

 5) The process of generating the facial appearance Vsum to Vsum of the partial image based on y) to E (x, y)

6 5 6

 The facial appearance Vsum to Vsum of the partial image is generated based on the images E (x, y) to E (x, y)

 This is based on the same idea as 1 4 1 4 processing. That is, the concept of the concave portion may be replaced with the concept of the convex portion.

 The partial image face determination process described above is performed for each partial image shifted by 1 bit in each reduced image, and all partial images that can be determined as face images are extracted, and the process proceeds to step S 7. move on.

[0079] In step S7, when it is determined in step 6 that a partial image is a face, the size S and coordinates (X, Y) of the face with respect to the input image of the partial image are set in the first implementation. Output in the same way as S, X, and Y are given by the following equation using the size of the face S '= 19 in the reduced image and the coordinates of the area that is the face (XY and reduction magnification κ). S = S '/ κ

 X = X '/ K

 Y = Y '/ K

 As described above, when a face image is included in the input image, the position and size of the face image are detected and output.

 When the processing of the second embodiment described above is performed, the following effects are obtained.

 (1) Luminance is brighter locally in the mouth laughing while showing teeth, and in the nose when it shines when exposed to light. According to the present embodiment, such a locally bright spot is effectively detected to create an edge image. Therefore, if the edge image created in this way is used in the same manner as the determination of the face image by the concave image in the first embodiment, in addition to the locally dark portion of the face image, the face image It is possible to perform face determination considering local bright spots. As a result, the face can be detected with higher accuracy than in the first embodiment.

 [0082] Third Embodiment

 In the second embodiment, a luminance convex image is generated in addition to the luminance concave image, and in addition to a locally dark spot such as the eyes, nose and mouth of the face, a mouth laughing with a tooth or a light hits it. In the case of a nose that is shining, the example in which the location where the brightness is locally brighter than the surroundings is also detected appropriately. In the third embodiment, an example will be described in which information on the luminance concave portion image and the luminance convex portion image is combined into a luminance uneven portion image and processed.

 [0083] The third embodiment is realized by the personal computer 1 as in the first embodiment. Therefore, for the configuration of the image processing apparatus of the third embodiment, refer to FIG. 1 of the first embodiment. The image processing program executed by the personal computer 1 is the same as the flow chart of FIG. 2 used in the first embodiment and referenced in the second embodiment. Similarly, the following explanation will be made with reference to FIG.

 [0084] Steps S1 to S3 are the same as those in the second embodiment, and thus description thereof is omitted.

[0085] In step S4, edge images E (x, y) to E (x, y) are generated as in the second embodiment. To do. Then, based on the following equation, longitudinal luminance unevenness portion image E ₇ (x, y) and transverse luminance concave protrusions image E (x, y) to produce a.

 8

 [Number 1]

[Equation 1]

_{(Α (χ,)> 5} (χ, the case of a)

E ₅ (x, y) a) If ≤E ₅ (x,))

[Equation 2]

[Equation 2]

[0086] In step S5, a face determination target area of 19 X 19 pixels is set every other pixel of the reduced image, and edge images E (x, y), E (x, y), E (x , Y), E (x, y)

 1 2 7 8

 Output an image.

 [0087] In step S6, the edge images E (x, y), E (x, y), E (x, y), Ε (x

 1 2 7 8

, Y), it is determined in the same manner as in the first embodiment whether this area is a face image. The facial appearance Vsu of the partial image based on the edge images E (x, y) to E (x, y)

 7 8

 The process of generating m to Vsum is also a partial image based on the edge images E (x, y) to E (x, y)

7 8 1 4

 This is based on the same idea as the process of generating Vsum to Vsum. That is, the recess

 14

 This concept may be replaced with a concept combining concave and convex portions.

[0088] Step S7 is the same as in the first embodiment. As described above, when a face image is included in the input image, the position and size of the face image are detected and output.

When the processing of the third embodiment described above is performed, the following effects are obtained.

(1) By combining the information on the luminance concave portion image and the luminance convex portion image into the luminance uneven portion image, the face can be detected with an accuracy close to that of the second embodiment while omitting the amount of information used for the discrimination process. [0090] A variation

In the above embodiment, in the determination of the face image, is generated and processed.

 The example to do was explained. However, the edge image output in step 5 may be subjected to a known learning discrimination process such as a neural network to determine whether this region is a face image.

 A facial expression determination process may be performed by applying a known technique to the edge image in the detected face image area. In particular, in the second and third embodiments, it is detected that the brightness of the mouth of the face that is laughing while showing teeth is locally high, so such a smile is determined with high accuracy. be able to.

 [0092] The edge detection filter that generates the vertical luminance concave image may be as follows.

 E '(x, y) = Min (Y (x, y- l), Y (x, y + l)) — Y (x, y)

 1 LH LH LH

 That is, the target pixel and three pixels adjacent in the vertical direction may be used. The same applies to the horizontal luminance concave image. The same applies to the edge detection filter that generates the brightness convex image.

[0093] The edge detection filter that generates the vertical luminance concave image may be as follows.

 E '(x, y) = Min (Y (x, y- l), Y (x, y + 2))

 1 LH LH

 — (Y (x, y) + Y (x, y + l)) / 2

 LH LH

 The same applies to the horizontal luminance concave image. The same applies to the edge detection filter that generates the brightness convex image.

 [0094] A plurality of different sizes may be used as the size of the filter for creating the luminance concave image or luminance convex image, and a convex structure or a concave structure in a plurality of frequency bands may be detected. Note that convex structures or concave structures in a plurality of frequency bands may be detected by calculating filters of the same size for a plurality of luminance images with different reduction magnifications.

In the above embodiment, an example has been described in which the difference between the minimum value of the pixels near the target pixel and the minimum value of the surrounding pixels is output as E ′ (x, y). That is, the minimum value and the circumference of the pixels near the target pixel The example which outputs the difference of the minimum value of a side pixel in the form of the difference value was demonstrated. However, the ratio of these values may be output as a value representing the difference! /.

 [0096] In the above embodiment, the example in which the personal computer 1 performs the image processing for detecting the face image from the photographed image has been described. However, the above-described processing may be performed on the captured image in an imaging apparatus such as a digital still camera.

 FIG. 10 is a diagram showing a configuration of a digital camera 100 that is such an imaging apparatus. The digital camera 100 includes a photographing lens 102, an image sensor 103 including a CCD, a control device 104 including a CPU and peripheral circuits, a memory 105, and the like.

 The image sensor 103 captures (captures) the subject 101 via the photographing lens 102 and outputs the captured image data to the control device 104. The control device 104 performs image processing for detecting the face image described above on the image (image data) captured by the image sensor 103. Then, the control device 104 performs white balance adjustment and various other image processing on the image captured based on the detection result of the face image! /, And stores the image data after the image processing in the appropriate memory 105. Store. Further, the control device 104 can also use the detection result of the face image for autofocus processing or the like. The image processing program executed by the control device 104 is stored in a ROM (not shown).

 [0099] The processing described above can also be applied to a video camera. Furthermore, it can also be applied to surveillance cameras that monitor suspicious individuals and devices that identify individuals based on captured face images and estimate gender, age, and facial expressions. That is, the present invention can be applied to all devices such as an image processing device and an imaging device that extract and process a specific type of image such as a face image.

 [0100] While various embodiments and modifications have been described above, the present invention is not limited to these contents. Other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

 [0101] The disclosure of the following priority application is hereby incorporated by reference.

 Japanese patent application No. 215944 (filed Aug. 8, 2006)

Claims

The scope of the claims  [1] An image processing method,  Acquire an image consisting of multiple pixels,  Based on the acquired image, the edge of the concave structure where the pixel value is recessed from the periphery locally is detected,  An image processing method for generating an edge image based on the detected edge of the concave structure.  [2] In the image processing method according to claim 1,  An image processing method for generating the edge image by calculating a non-linear filter for detecting the edge of the concave structure with respect to the acquired image. [3] In the image processing method according to claim 2,  The non-linear filter is an image processing method for outputting a calculation result based on a difference between a pixel value in a target region and a minimum pixel value in a peripheral region of the target region. [4] In the image processing method according to claim 3,  The non-linear filter is an image processing method for outputting a calculation result based on a difference between a minimum pixel value in the target region and a minimum pixel value in the peripheral region. [5] In the image processing method according to claim 4,  When the minimum pixel value in the target area is smaller than the minimum pixel value in the peripheral area of the target area, the value corresponding to the difference is set as the edge pixel value, and the minimum pixel value in the target area is An image processing method for clipping the edge pixel value to zero when the pixel value is larger than the minimum pixel value in the peripheral area of the target area. [6] An image processing method,  Acquire an image consisting of multiple pixels,  Based on the acquired image, the edge of the convex structure where the pixel value protrudes locally from the periphery is detected,  An image processing method for generating an edge image based on the detected edge of the convex structure. [7] The image processing method according to claim 6, An image processing method for generating the edge image by calculating a non-linear filter for detecting the edge of the convex structure on the acquired image. [8] In the image processing method according to claim 7,  The non-linear filter is an image processing method for outputting a calculation result based on a difference between a pixel value in a target region and a maximum value of pixel values in a peripheral region of the target region. [9] The image processing method according to claim 8,  The non-linear filter is an image processing method for outputting a calculation result based on a difference between a maximum pixel value in the target region and a maximum pixel value in the peripheral region. [10] The image processing method according to claim 8,  When the maximum pixel value in the target region is larger than the maximum pixel value in the peripheral region of the target region, the value corresponding to the difference is set as the value of the edge pixel, and the maximum pixel value in the target region is An image processing method in which the edge pixel value is clipped to zero when the pixel value is smaller than the maximum value in the surrounding area of the target area. [11] In the image processing method according to any one of claims 1 to 10,  Based on the acquired image, generate a luminance image by a luminance component,  An image processing method for generating the edge image using the generated luminance image.  [12] The image processing method according to any one of claims 3 to 5 and 8 to 10,  The image processing method, wherein the target area is a single pixel of only the target pixel or a two-pixel area of the target pixel and its neighboring pixels, and the peripheral area is a two-pixel area located on both outer sides of the target area. [13] In the image processing method according to any one of claims 2 to 5 and 7 to 10,  An image processing method for calculating the nonlinear filter in at least two directions. [14] An image processing method comprising:  Acquire an image consisting of multiple pixels,  Based on the acquired image, a concave structure edge where the pixel value is recessed locally from the periphery and a protruding convex structure edge are detected,  An image processing method for generating an edge image of a concave structure based on the detected edge of the concave structure, and generating an edge image of the convex structure based on the detected edge of the convex structure. [15] An image processing method comprising: Acquire an image consisting of multiple pixels,  Based on the acquired image, a concave structure edge where the pixel value is recessed locally from the periphery and a protruding convex structure edge are detected,  An image processing method for generating an edge image including both a concave structure edge and a convex structure edge based on the detected concave structure edge and convex structure edge. [16] An image processing method comprising:  Acquire an image consisting of multiple pixels,  Based on the acquired image, at least one of the edge of the concave structure where the pixel value is locally recessed from the periphery and the edge of the convex structure where the pixel value protrudes locally from the periphery is detected. ,  An image processing method for generating an edge image based on at least one of the detected edge of the concave structure and the edge of the convex structure. [17] An image processing method comprising:  Acquire an image consisting of multiple pixels,  Detecting an edge component of the acquired image;  Gamma conversion is performed on the detected edge component,  An image processing method for generating an edge image based on the gamma-converted edge component. [18] In the image processing method according to any one of claims 1 to 17;  An image processing method for detecting a face image using the generated edge image. [19] An image processing program for causing a computer to execute the image processing method according to any one of claims 1 to 18.  20. An image processing apparatus equipped with the image processing program according to claim 19. [21] An imaging device equipped with the image processing program according to [19].