JP2003216929A - Subject extraction device - Google Patents

Abstract

(57) [Summary] [Problem] To enable quick setting of an initial contour by a simple operation without providing a special input device. An image input from an image input unit is displayed on a display unit and sent to a subject extracting unit. The initial contour setting unit 5 sets a figure having a predetermined size at a predetermined position on the screen as an initial contour, and the display unit 4 superimposes the figure on the input image and displays the image. The initial contour is the moving part 6
Can move up, down, left and right, and can be scaled up and down by the scaling unit 7. When the initial contour reaches a desired position and a desired size, it is determined as an initial contour used for subject extraction by the determination unit 8 and sent to the subject extraction unit 2. The subject extraction unit 2 extracts a subject using the input image and the initial contour. The extracted subject is recorded on a recording medium by the recording / reproducing unit 3 and is displayed on the display unit 4 when it is reproduced.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a subject extracting apparatus for setting an initial contour in order to extract a subject existing in an image.

[0002]

2. Description of the Related Art Conventionally, as a method for detecting the contour of a subject from a photographed image obtained by photographing the subject with, for example, a video camera, M. There is a technique called Snakes proposed by Kass et al. (For example, “Snake”).
s: Active Contour Model ”,
Int. Journal of Computer
Vision, pp. 321-331 (1998)). This method uses a dynamic contour model, and solves the problem of pattern matching that finds the contour pattern of the shape closest to the contour of the object in the image as a problem of energy minimization. In this method, first, it is necessary to give a closed curve that roughly surrounds the subject as an initial contour.

As a method of giving this initial contour, there is a method disclosed in, for example, Japanese Patent Application Laid-Open No. 2001-111882. In this method, when it is desired to detect a subject 82 included in a shooting scene as shown in FIG. 9A, the ellipse 81 is first set at a predetermined position in the center of the screen. Next, the user looks at the monitor screen and sets the camera 8
9B, the subject 82 comes to the position of the ellipse 81, and the ellipse 8 in that state is shown in FIG. 9B.
By determining 1 as the initial contour, the initial contour can be easily and accurately determined.

[0004]

However, the above method has a problem in that the position of the subject must be in the center of the screen at the start of shooting, which imposes a great constraint on the composition. In order to improve this, Japanese Unexamined Patent Application Publication No.
Japanese Patent Laid-Open No. 11-11882 discloses a method of setting an initial contour at an arbitrary position on a screen by using line-of-sight detection, but has a drawback that a device for line-of-sight input and complicated processing are required. .

Further, in the above method, the elliptical region 81 is constant regardless of the size of the subject, and therefore there is a problem that it becomes inappropriate as an initial contour depending on the subject. In order to improve this, there is also a Japanese Patent Laid-Open No. 2001-11188.
Japanese Patent Publication No. 2 discloses a method in which the entire screen is divided into a plurality of blocks, the distance is measured for each block, and the contour of a region in which blocks having the same distance are connected is used as an initial contour. The disadvantage is that it requires a device for detection and complicated processing. Moreover, it is actually difficult to perform accurate distance detection in block units.

[0006] It should be noted that a PC can input data with a mouse and a PDA can input data with a pen. However, in the case of an object extraction apparatus having a video camera with an object extraction function, these are generally used. Since no input device is equipped, it is costly to equip them for initial contour setting. Even if these input devices are equipped, the method of tracing the outline of the subject while looking at the small monitor screen is not easy to operate, so that it is not possible to input as desired and there is a problem that it takes time to input. is there.

It is an object of the present invention to provide a subject extracting device which can set an initial contour quickly by a simple operation without providing a special input device.

[0008]

In order to solve the above problems, the present invention is a subject extracting apparatus for extracting a desired subject from an image, the image input unit inputting the image and the display unit displaying the image. And an initial contour setting unit that presents a graphic of a predetermined size as a default contour at a predetermined position on the display unit,
A moving unit that moves the presented initial contour up and down, left and right, and a scaling unit that changes the size of the initial contour,
It is characterized by further comprising a determining unit that determines when the initial contour is set to a desired position and size and a subject extracting unit that extracts a subject from a range surrounded by the determined initial contour.

The enlarging / reducing unit may change the size of the initial contour so that the rate of change in the horizontal direction becomes equal to that in the vertical direction, or the size of the initial contour may be changed independently in the horizontal and vertical directions. The size may be changed. Also,
The initial contour is a polygon, the initial contour setting unit,
The initial contour may be presented so that the lower side of the display unit matches the lower side of the initial contour.

Further, the present invention is a subject extracting device for extracting a desired subject from an image, comprising an image input unit for inputting an image, a display unit for displaying the image, and a predetermined position at a predetermined position on the display unit. An initial contour setting unit that presents a graphic of a size as an initial contour, an enlarging / reducing unit that changes the size of the initial contour, and a determining unit that determines when the initial contour is set to a desired position and size. A subject extraction unit that extracts a subject from the range surrounded by the determined initial contours;
A sound input unit capable of inputting sound of two or more channels, and a sound source direction analysis unit that detects the direction of the sound source from the input sound are provided. It is characterized by setting.

It is also possible to provide a moving section for moving the initial contour vertically and horizontally so that the position of the initial contour set by using the analysis result of the sound source direction analyzing section can be adjusted.

Further, the present invention is a subject extracting device for extracting a desired subject from an image, comprising an image input unit for inputting an image, a display unit for displaying the image, and a predetermined position at a predetermined position on the display unit. An initial contour setting unit that presents a graphic of a size as an initial contour and a subject extraction unit that extracts a subject from the range surrounded by the determined initial contour, and the initial contour setting unit preliminarily determines the position and size. The present invention is characterized by including a template selection unit that holds a plurality of templates of initial contours for which a predetermined size is determined and selects an appropriate template from the templates.

Further, the present invention may include a recording / reproducing unit for recording / reproducing an image, and may extract a desired subject from the image recorded on the recording medium.

[0014]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment> FIG. 1 shows a first embodiment of the present invention.
3 is a block diagram of a subject extraction device in the embodiment of FIG. The image input from the image input unit 1 is displayed on the display unit 4 and sent to the subject extracting unit 2. The initial contour setting unit 5 sets a graphic having a predetermined size at a predetermined position on the screen as an initial contour, and the display unit 4 superimposes it on the input image and displays it. The initial contour is the moving part 6
Can be moved up and down, left and right, and can be enlarged and reduced by the enlargement and reduction unit 7. When the initial contour has a desired position and a desired size, the determining unit 8 determines the initial contour to be used for subject extraction, and sends it to the subject extracting unit 2. The subject extracting section 2 extracts a subject using the input image and the initial contour. The extracted subject is recorded on the recording medium by the recording / reproducing unit 3, and when it is reproduced, it is displayed on the display unit 4.

FIG. 2 shows an example of a user interface in this embodiment, which is an image of a portable information terminal with a camera having a subject extracting function. Normally, the image input from the lens unit 14 is displayed on the monitor 16, but in the mode of extracting a subject, the initial contour is displayed so as to be superimposed on the input image. The lens unit 14 is 270
2 shows a case in which the photographer who is looking at the monitor 16 is the subject himself. By rotating the lens unit 14 and directing the lens to the opposite side, a subject other than the photographer can be photographed. The shutter key 15 is a key for determining an initial contour and starting / stopping image capturing, and corresponds to the determining unit 8 in FIG. The cross key 11 is a key for moving the initial contour presented on the monitor 16 vertically and horizontally, and corresponds to the moving unit 6 in FIG. Expand key 12 and Reduce key 13
Is a key for enlarging and reducing the initial contour presented on the monitor 16, and corresponds to the enlarging / reducing unit 7 in FIG.

The screen displayed on the monitor 16 is, for example, as shown in FIG. Here, the lens unit 14
When the subject 22 and the subject 23 exist in the input image from, the subject 22 is assumed to be the subject to be extracted.
In FIG. 3A, the initial contour setting unit 5 of FIG. 1 displays the ellipse 21 as the initial contour in the center of the screen. As it is, the ellipse 21 is far away from the subject 23,
The ellipse 21 is moved to the right using the cross key 11 in FIG. Further, since the size of the ellipse 21 is smaller than that of the subject 22, the ellipse 21 is enlarged using the enlargement key 12 in FIG. 2 until the subject can be surrounded. As a result,
The state shown in (b) is reached, and the ellipse 21 is now appropriate as the initial contour for extracting the subject 22, so the shutter key 15 in FIG. 2 is pressed. In this way, the initial contour is determined and shooting and subject extraction are started. By doing so, an appropriate initial contour can be set regardless of the position and size of the subject. The subject extraction unit 2 extracts a subject 22 within a range surrounded by an ellipse 21 which is an initial contour.

When the extraction result is recorded on the recording medium, only the extracted subject may be recorded, or an image in which the background portion not extracted is replaced with another image prepared in advance may be recorded. Good. In addition, by recording the input image and separately recording the shape data indicating the extraction result, it is possible to perform the extraction and the replacement of the background portion at any time after the recording. When such a recording method is used, the subject 82 to be extracted in FIG.
Since it is necessary to bring the object to the center of the screen, it is not possible to shoot the subject 83 that is off the screen, whereas in the present embodiment, the initial contour can be moved as shown in FIG. 23 can also be photographed according to the first composition.

The input image in this embodiment may be a still image or a moving image. When targeting a still image, press the shutter key 15
The subject is extracted at the same time when one image is input. When a moving image is targeted, recording of the moving image and subject extraction are started when the shutter key 15 is pressed, and the operation is stopped when the shutter key 15 is pressed again. Even when a moving image is targeted, the initial contour is required only for the first frame, so the initial contour is displayed on the display unit 16 only for the first frame and not for the second frame and thereafter. To do so.

After determining the initial contour, the shutter key 1
The subject extraction process may be performed by pressing 5 halfway, and the result may be displayed on the monitor 16. At this point of time, the extraction result is not recorded in the recording / reproducing unit 3, but recording is started when the shutter key 15 is completely pressed.

When the enlargement / reduction of the initial contour is performed by using only the two keys of the enlargement key 12 and the reduction key 13, the rate of change in the size of the initial contour is set to be the same in the horizontal direction and the vertical direction. . That is, for example, it is doubled in the horizontal direction and doubled in the vertical direction at the same time. By doing so, the initial contour can be set quickly by a simple operation.

By the way, since there are vertically and horizontally long subjects to be extracted, it is desirable that the size of the initial contour can be changed separately in the horizontal and vertical directions. Therefore, a horizontal direction designation key 17 and a vertical direction designation key 18 are prepared. For example, when the enlargement key 12 is pressed while holding down the horizontal direction designation key 17, the vertical size of the initial contour is maintained and the horizontal direction is maintained. If the user presses the reduction key 13 while pressing the vertical direction designation key 18 while enlarging the size, the size of the initial contour is reduced in the vertical direction while maintaining the horizontal size. At this time, if only the enlargement key 12 or only the reduction key 13 is pressed without pressing the horizontal direction designation key 17 or the vertical direction designation key 18, the rate of change in size between the horizontal direction and the vertical direction becomes the same. I shall. By doing so, it is possible to give appropriate initial contours to objects of various shapes.

When the shape of the initial contour is changed by using the horizontal direction designation key 17 or the vertical direction designation key 18, the area may be maintained. That is, in the case of expanding in the horizontal direction, the area may be reduced in the vertical direction so that the area becomes constant, and in the case of expanding in the vertical direction, the area may be reduced in the horizontal direction while keeping the area constant.

Although the shape of the initial contour is an ellipse in the above description, the shape is not limited to this and may be any shape such as a rectangle or a polygon. The appropriate initial contour condition depends on the subject extraction algorithm. That is, in the method such as Snakes described above, the initial contour is shown in FIG.
It is necessary to be outside the subject as shown in.

On the other hand, if the assumption is that the background is a substantially uniform color and the color of the subject is clearly different from the color of the background, the histogram of pixel values for the entire screen is conceptually shown in FIG. It becomes as shown in (a). Figure 4
In (a), the horizontal axis represents the pixel value and the vertical axis represents the frequency. However, when the histogram makes two peaks in this way, the background and the subject can be separated by setting an appropriate threshold Th. it can. When separating objects using this method, the initial contour need not be completely outside as long as it covers most of the objects to be extracted.

In other words, as shown in FIG. 3C, even if the subject 22 is protruding from the ellipse 21, it does not matter as an initial contour. In this case, histograms of pixels inside the ellipse and pixels outside the ellipse are as shown in FIG. Since some pixels of the background are included in the ellipse and some pixels of the subject are included outside the ellipse, the histogram is not clearly separated as shown in FIG. By setting the threshold Th, it is possible to separate the background from the subject.

When a person is a subject, a composition for photographing the upper half of the body is often used. In such a case, the subject 24 is cut off at the lower side of the frame 25 as shown in FIG. It will be shaped like The initial contour given to the subject 24 is also the frame 25.
Since it is desirable to extend to the full bottom edge, if the presentation is made so that the bottom edge of the rectangular initial contour 26 and the bottom edge of the frame 25 match as shown in FIG. 5B, the effort of movement and scaling is reduced. be able to.

In the above description, one subject in the screen is extracted, but it is also possible to extract a plurality of subjects. That is, in FIG. 3B, both the subject 22 and the subject 23 are extracted. In such a case, the multiple extraction mode is provided, and after setting the initial contour for extracting the subject 22, first,
An initial contour for extracting the subject 23 may be set, and subject extraction may be started when both settings are completed. This can be realized, for example, by pressing the shutter key 15 for a short time to determine the initial contour for each object, and pressing the shutter key 15 for a long time to start object extraction.

<Second Embodiment> Next, a second embodiment of the present invention will be described. FIG. 6 is a block diagram of a subject extracting apparatus according to the second embodiment of the present invention. In this embodiment, the subject is a person. The subject extracting apparatus of FIG. 6 is obtained by removing the moving unit 6 from the subject extracting apparatus shown in FIG. 1 and adding a voice input unit 9 and a sound source direction analyzing unit 10. Voice input unit 9 is 2
It may be a stereo microphone of two channels, which is composed of one or more microphones. The sound source direction analysis unit 10 detects the direction in which the sound is emitted based on the input from the sound input unit 9.

As the detection algorithm, for example, the methods disclosed in Japanese Patent Application Laid-Open No. 3-285190 and Japanese Patent Application Laid-Open No. 7-287062 can be used. For sound waves, the direction of the sound source can be obtained from the time difference, phase difference, amplitude difference, etc. between the channels. It should be noted here that the sound source direction analysis unit 10 only needs to know the direction of the sound source, and does not need to obtain the distance to the sound source.

Similar to the subject extracting apparatus shown in FIG. 1, the initial contour setting unit 5 sets a figure having a predetermined size at a predetermined position on the screen as an initial contour. Ask them to say a word or two. When the voice input unit 9 captures this voice and the sound source direction analysis unit 10 obtains the voice arrival direction, the initial contour is automatically moved in a direction that coincides with the voice arrival direction. Then, the enlargement / reduction unit 7 adjusts the size, and the determination unit 8 determines the appropriate initial contour.
By doing so, the initial contour setting can be performed more easily.

In the subject extracting apparatus of FIG. 6 as well,
By providing a moving unit that moves the initial contour vertically and horizontally, after automatically setting the initial contour position by sound source direction analysis,
It is possible to adjust the initial contour.

<Third Embodiment> Next, a third embodiment of the present invention will be described. FIG. 7 is a block diagram of a subject extracting apparatus according to the third embodiment of the present invention. The present embodiment is characterized in that the initial contour setting unit 31 includes a template selection unit 32 and sets the initial contour using a template. As the template for setting the initial contour, an orthodox template in which an ellipse or a rectangle is set in the center of the screen, a template set in a position other than the center of the screen as shown in FIG. 8A, and shown in FIG. 8B. It is conceivable that a plurality of areas are set as described above. By using the template shown in FIG. 8B, it is possible to extract both the subject 22 and the subject 23 shown in FIG. 3B at the same time. When the initial contour is set using the template, it is possible to display a list of thumbnails of the template as shown in FIG. 8C and select the shape using the cross key 11 in FIG.

Also, in the subject extracting apparatus of FIG. 7,
By providing a moving unit that moves the initial contour vertically and horizontally and an enlarging / reducing unit that performs enlarging / reducing, the initial contour can be adjusted after the template is selected. By doing so, it is possible to simplify the initial contour setting and perform various subject extractions.

[0035]

According to the subject extracting apparatus of the present invention, the initial contour can be set at an arbitrary position on the screen by the moving section without using an extra input device such as a mouse or a pen. Since the size can be freely changed by the enlarging / reducing unit, an appropriate initial contour can be set regardless of the position and size of the subject.

According to the subject extracting apparatus of the present invention,
By changing the size of the initial contour so that the rate of change in the horizontal direction becomes equal to that in the vertical direction, the initial contour can be set quickly by a simple operation.

According to the subject extracting apparatus of the present invention,
By changing the size of the initial contours independently in the horizontal and vertical directions, it is possible to set appropriate initial contours for subjects of various shapes.

Further, according to the subject extracting apparatus of the present invention,
By setting the lower side of the frame and the lower side of the initial contour to match the subject of a person, it is possible to reduce the trouble of moving and scaling at the time of setting.

Further, according to the subject extracting apparatus of the present invention,
Since the position of the initial contour that surrounds the subject is automatically set based on the analysis result of the sound source direction analysis unit, the labor of moving the initial contour can be reduced.

Further, according to the subject extracting apparatus of the present invention,
Since the moving unit that moves the initial contour vertically and horizontally is provided, the initial contour can be moved to a more appropriate position for fine adjustment, and an appropriate initial contour can be set.

According to the subject extracting apparatus of the present invention,
Since the template selection unit that selects an appropriate template from among the plurality of templates of the initial contour is provided, for example, subjects at distant positions can be simultaneously extracted, and subject extraction can be efficiently performed.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a subject extraction device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing an example of an external appearance of the subject extraction device according to the first embodiment of the present invention.

FIG. 3 is a diagram illustrating a method of setting an initial contour according to the present invention.

FIG. 4 is a diagram illustrating a region separation method using a threshold for a pixel value.

FIG. 5 is a diagram illustrating an initial contour setting method when a subject is a person.

FIG. 6 is a block diagram showing a configuration of a subject extraction device according to a second embodiment of the present invention.

FIG. 7 is a block diagram showing a configuration of a subject extraction device according to a third embodiment of the present invention.

FIG. 8 is a block diagram showing an example of a template for initial contour setting.

FIG. 9 is a diagram illustrating a method of setting an initial contour in a conventional subject extraction device.

[Explanation of symbols]

1 Image input section 2 Subject extraction section 3 Recording / playback section 4 Display 5 Initial contour setting section 6 moving parts 7 Enlargement and reduction section 8 decision department 9 Voice input section 10 Sound source direction analysis unit 11 cross key 12 Expand key 13 Reduction key 14 Lens part 15 Shutter key 16 Display 17 Horizontal direction key 18 Vertical direction key 21 ellipse 22, 23, 24 subject 25 frames 26 Initial contour 31 Initial contour setting section 32 Template selection section

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B057 BA02 BA24 BA26 DA08 DB02                       DB06 DB09 DC16 DC22                 5C076 AA02 AA14 AA17 AA19 AA21                       AA22 CA02                 5L096 AA02 AA06 CA02 CA24 FA06                       FA37

Claims (8)

[Claims] 1. A subject extracting apparatus for extracting a desired subject from an image, comprising: an image input unit for inputting the image; a display unit for displaying the image; and a graphic of a predetermined size at a predetermined position of the display unit. An initial contour setting unit that presents the initial contour as a starting contour, a moving unit that moves the presented initial contour vertically and horizontally, a scaling unit that changes the size of the initial contour, and a desired position of the initial contour. A subject extraction device comprising: a determination unit that is determined when the size is set, and a subject extraction unit that extracts a subject from the range surrounded by the determined initial contours. 2. The subject extracting apparatus according to claim 1, wherein the enlarging / reducing unit changes the size of the initial contour so that the rate of change in the horizontal direction is equal to that in the vertical direction. 3. The subject extracting apparatus according to claim 1, wherein the enlargement / reduction unit can change the size of the initial contour independently in the horizontal direction and the vertical direction. 4. The initial contour is a polygon, and the initial contour setting unit presents the initial contour so that a lower side of the display unit and a lower side of the initial contour match each other. 1. The subject extraction device according to 1. 5. A subject extracting apparatus for extracting a desired subject from an image, comprising: an image input unit for inputting an image; a display unit for displaying the image; and a graphic of a predetermined size at a predetermined position of the display unit. An initial contour setting unit that presents the initial contour as an initial contour, a scaling unit that changes the size of the initial contour, and a determination unit that determines when the initial contour is set to a desired position and size. A subject extraction unit for extracting a subject from the range surrounded by the initial contour; a voice input unit capable of inputting voice of two or more channels; and a sound source direction analysis unit for detecting a direction of a sound source from the input voice, A subject extracting apparatus, wherein a position of the initial contour surrounding the subject is set according to an analysis result of a sound source direction analyzing unit. 6. A moving unit that moves the initial contour vertically and horizontally, and is capable of adjusting the position of the initial contour set using the analysis result of the sound source direction analysis unit. The subject extraction device according to claim 5. 7. A subject extracting apparatus for extracting a desired subject from an image, comprising: an image input unit for inputting the image; a display unit for displaying the image; and a graphic of a predetermined size at a predetermined position of the display unit. Is provided as an initial contour, and an object extracting section that extracts an object from a range surrounded by the determined initial contour, the initial contour setting section having a predetermined position and size. A subject extracting apparatus, comprising: a template selecting unit that holds a plurality of templates of initial contours and selects an appropriate template from the templates. 8. A recording / reproducing unit for recording / reproducing an image,
The subject extracting apparatus according to claim 1 or 7, wherein a desired subject is extracted from an image recorded on a recording medium.