KR20090093904A - Apparatus and method for scene variation robust multimedia image analysis, and system for multimedia editing based on objects - Google Patents

Abstract

The present invention relates to an apparatus and method for analyzing a multimedia image that is robust to scene changes, and an object-based editing system using the same. The present invention relates to a predetermined image unit (scene, video shot) in analyzing and editing a multimedia image such as a still image or a moving image. After segmentation, a representative frame is selected for each divided image unit, and the analysis is performed by analyzing / editing (eg, classifying an object, searching / tracking an object, adding object information, etc.) to a multimedia image using the representative frame as a target. / Minimize the time, effort, and cost of editing.

To this end, the present invention, in the object-based multimedia editing system, to divide the multimedia image into a predetermined image unit to select a representative frame for each divided image, and to analyze the multimedia image with the selected representative frame first priority Multimedia image analysis means; And management means for inputting corresponding object information to the searched / tracked object through analysis of the multimedia image analyzing means, or performing scene / object editing through a user interface.

Description

Apparatus and method for multimedia image analysis robust to scene change, and object-based multimedia editing system using the same

The present invention relates to an apparatus and method for analyzing a multimedia image that is robust to scene changes, and an object-based multimedia editing system using the same. More particularly, the present invention relates to a certain image unit (scene) in analyzing and editing a multimedia image such as a still image or a moving image. , Video shot) and select a representative frame for each divided image unit, and then analyze / edit multimedia images (for example, object classification, object search / tracking, and object information). The present invention relates to an apparatus and method for multimedia image analysis that is robust to scene changes, and an object-based multimedia editing system using the same, capable of minimizing time / effort / cost required for analysis / editing.

In recent years, as the demand for multimedia information increases, various technologies related to editing, searching, and objectification of multimedia contents (for example, moving images) have been widely developed.

For example, a video generation method based on a network described in Korean Patent Laid-Open Publication No. 2000-0014421, " Method for Creating and Updating Video Data Linked Playback Information on Rewritable Recording Media " and Korean Patent Laid-Open Publication No. 2000-0017815 System and method of generating the same, ' Method of producing digital video album through the Internet ' described in Korean Patent Publication No. 2000-49833, ' Movie editing system and object information service using the same as described in Korean Patent Publication No. 2000-58241 Method ',' Video information providing and retrieval method 'described in Korean Patent Laid-Open No. 2000-58970, etc. disclose various technologies related to such a conventional video in detail.

However, since such conventional video using techniques are only a fraction of using arbitrarily given video information, only using the conventional video processing techniques as described above provides an interactive function to the video information. There are many limitations to this.

To overcome these limitations, MPEG (Moving Picture Expert Group) uses MPEG-4 Binary Format for Scenes (MPEG), MPEG-4 Light Application Scene Representation (LASeR), etc. How to attach event information is presented.

However, the MPEG standard also does not provide a way to separate objects in video or other multimedia data and how to add events to the separated objects. In the end, it was expensive to provide interactive functions.

On the other hand, the ' Multimedia Editing Tool and Multimedia Editing Method Using the Same' described in Korean Patent Laid-Open Publication No. 2002-0063754 have presented a more convenient method. An error occurs, which causes a lot of cost / effort / time for editing videos.

Editing to give an interactive function to video and multimedia data is generally composed of three processes. The first process is to select and separate the location of the object, the second process is to enter object information, and the third process is to inspect. Hereinafter, each process will be described.

First, a first process (a process of selecting and separating an object's position) is a process of selecting a position and an area with respect to a semantic object required by input video and multimedia data. These locations and areas must contain information that changes continuously during the time the object appears.

Next, the second process (entering object information) extracts meaningful objects according to the editor's intention, and extracts other data (e.g., different video data, different audio data, different URL links) into the extracted object. Meta data such as information) is a process of inputting information that can be connected in real time.

Next, the third process (verification process) is the process of verifying the results of the previous two processes.

Of the three editing processes described above, the process that consumes the most effort / time / cost is the first process, 'selecting and separating the object's location'. For example, there is an urgent need for automation through image processing algorithms based on recognition such as face recognition) and automatic object tracking. In addition, it is necessary to increase the efficiency of the object information input process and the inspection process.

In particular, in the conventional multimedia image analysis technology, all frames constituting the multimedia image are treated in the same position, so that a lot of time / effort / cost and computational burden is required for searching / detecting a specific object. There is also a problem that takes a lot of time / effort / cost in editing the image.

Accordingly, the present invention provides a multimedia image analysis apparatus and method that is robust to scene changes, which can minimize the time / effort / cost required for analysis / editing by simplifying and automating the process of analyzing and editing multimedia images such as moving images, and the like; The object is to provide an object-based multimedia editing system using the same.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention which are not mentioned above can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

In order to achieve the above object, the present invention divides a multimedia content image into predetermined image units (scenes or shots), selects a representative frame for each divided image unit, and then selects the selected representative frames. Process such as object analysis / editing (eg, object classification, object search / tracking, object information addition, etc.).

In addition, the present invention is characterized in that the object detection / tracking method is applied differently according to the type of the object (for example, a face, an object, etc.).

In addition, the present invention uses a feature-based object detection method in the object detection (extraction) process in the representative frame, and a block-based object tracking method in the object tracking process in the individual frame belonging to the representative frame. It is done.

More particularly, the present invention provides a multimedia video analyzing apparatus that is robust to scene changes, comprising: scene segmentation means for segmenting a multimedia video into predetermined video units and selecting a representative frame for each divided video; And object searching means for classifying the selected representative frames according to whether objects are identical based on a central object type.

In addition, the present invention provides a multimedia video analyzing apparatus that is robust to scene changes, comprising: scene segmentation means for segmenting a multimedia video into predetermined video units and selecting a representative frame for each divided video; And object searching means for searching for a search object based on the selected representative frames.

The present invention also provides a multimedia image analysis method that is robust against scene changes, comprising: a segmentation step of dividing a multimedia image into predetermined image units; A representative selecting step of selecting a representative frame for each divided image; And classifying the selected representative frames according to whether objects are identical based on a central object type.

In addition, the present invention, a multimedia image analysis method that is robust to scene changes, the division step of dividing the multimedia image into predetermined image units; A representative selecting step of selecting a representative frame for each divided image; And an object searching step of searching for a search target object based on the selected representative frames.

In the object-based multimedia editing system, the multimedia image is divided into predetermined image units to select a representative frame for each divided image, and the multimedia image for analyzing the multimedia image based on the selected representative frames. Analysis means; And management means for inputting corresponding object information to an object searched / tracked through analysis of the multimedia image analyzing means, or performing scene / object editing through a user interface.

As described above, the present invention has the effect of minimizing the cost / time / effort required for image analysis / editing by simplifying / automating the image analysis / editing process through scene division in scene / shot units.

That is, the present invention selects a representative frame for each predetermined image unit (scene or video shot, etc.) in consideration of characteristics such as a scene or a video shot in analyzing / editing a multimedia content image. By analyzing / editing, it is possible to minimize the cost / time / effort for analyzing / editing images.

In addition, the present invention can minimize the errors in the analysis / editing process by analyzing / editing the content image file grouped by scene / shot unit, and also the effect that can be quickly (easily) correct (error) generated errors There is.

In addition, the present invention provides various user interface functions such as automating the input of object information, arranging objects by time / object characteristics, and distinguishing whether objects are identical by using different identification marks (colors, etc.). By doing so, there is an effect that allows an editor to easily and quickly analyze / edit the input image file.

 In addition, the present invention, by minimizing the content production cost for the interactive service through the analysis / editing of the simple / fast video content, services in IPTV and interactive TV, two-way UCC, Internet TV, various VoD systems Service) and revenue model, and the framework to aim for it in various multimedia contents as well as text in the next generation semantic web.

1 is a schematic conceptual diagram of an object-based multimedia editing service according to the present invention;

2 is an explanatory diagram of a basic reference unit referred to by the editing unit according to the present invention;

3 is a block diagram of an embodiment of an object-based multimedia editing system according to the present invention;

4 is a detailed configuration diagram of an embodiment of the multimedia image analyzing apparatus of FIG. 3 according to the present invention;

5 is a flowchart illustrating an embodiment of an object-based multimedia editing method according to the present invention;

6 is a flowchart illustrating an object detection method using DB information according to the present invention;

7 is a flowchart illustrating an object extraction and object region designation method in a representative frame according to the present invention;

8 is a flowchart illustrating an embodiment of an object information storage process according to the present invention;

9 is a screen configuration diagram for scene division according to the present invention;

10 is a screen configuration diagram for the operation and some verification in accordance with the present invention;

11 is a screen configuration diagram for feature-based object detection / tracking according to the present invention;

12 is a block diagram of a block-based object tracking / verification according to the present invention,

13 is a block diagram of an object information input interface screen according to the present invention;

14 is a diagram illustrating a shot view screen according to the present invention;

15 is a block diagram of an object management screen according to the present invention;

16 is a block diagram of an object tracking information screen according to the present invention;

17 is a view illustrating a screen configuration according to a scene divider, a shot view, and a face grouping according to the present invention;

18 is another exemplary diagram of a screen configuration for work and partial verification according to the present invention.

Explanation of symbols on the main parts of the drawings

100: object-based multimedia editing system 30: control management unit

31: data management unit 32: multimedia image analysis device

33: user interface management unit 35: input / output management unit

36: object / object information DB 303: frame extraction unit

311: scene and video shot management unit 312: object information management unit

313: feature extraction function management unit for each object 321: scene divider

322: face detector 323: face recognizer

324: non-face object detector 325: non-face object recognizer

326: object tracker 331: object information input unit

332: editor 40: object finder

The present invention relates to a multimedia authoring tool, that is, a multimedia image analysis / editing (eg, object classification, object search / tracking, object information addition, etc.), and closely links a series of analysis / editing modules. To arbitrarily link a given original multimedia object (e.g., some cells of the original video) with other data (e.g., other still images, video data, other audio data, other URL data, etc.) at the editor's option. Or add other events.

That is, the present invention develops a multimedia analysis / editor that makes it possible to analyze / edit content having an event capable of performing any action similar to hypertext when selecting an object such as a specific person, object, or material from a video or still image. This is related to technology.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram illustrating an object-based multimedia editing service according to the present invention.

Video media (multimedia video, multimedia content video) is composed of a plurality of various scenes. Objects appear in various scenes without rules and appear in different shapes and sizes depending on the angle and distance of the camera or the environment around the objects. For example, in the case of a person (object), its shape and size are different depending on the clothes or accessories being worn.

In particular, the objects placed in the three-dimensional space are mapped to the two-dimensional information through the camera, and due to the obscuration between the objects, the shape and size of the objects actually seen vary widely. In addition, it is very difficult to estimate the number of objects of interest in a video based on the plan of the service.

The present invention extracts and tracks an object of interest through minimal editor interaction in such an environment, and measures and stores the exact amount of change of the object of interest even when the object of interest appears in a non-contiguous scene by an automated object tracking technique. do. That is, the present invention utilizes scene segmentation, object detection, object recognition, face recognition, face region detection, object tracking, object region extraction, and the like, thereby enabling the editor to generate desired content with minimal interaction.

Hereinafter, the present invention will be described with reference to FIG. 1.

The object-based multimedia editing system 100 according to the present invention divides an input file (multimedia content image) into groups of predetermined image units (for example, scenes or video shots) in consideration of scene characteristics, and groups each input group. After selecting a representative frame (representative screen), an object search (object detection and object recognition) is performed for the selected representative frame. Detailed configurations and descriptions of the object-based multimedia editing system 100 will be described with reference to FIGS. 3 and 4.

In this case, the image unit (scene division unit) may be set as a scene, but for a more accurate object search, a video shot constituting the scene (generally, one scene is composed of a plurality of video shots, Video shots) may be set in video units. If the video unit is a video shot, scene division is performed for ease of editing by a user (editor), and the multimedia video is divided in units of shots for each divided scene.

Scenes and shots (Video Shot) are described in more detail as follows. In general, a video produced by a plan is shot using a large number of cameras, which are edited into a single video. In this case, it is assumed that three cameras are used for the events occurring at the same time and in the same place, and there are two characters, A and B. Camera 1 shoots 'Portrait A', Camera 2 shoots 'Portrait B', and Camera 3 shoots 'Portrait A and B' at the same time.These videos are from the same place and at the same time. It is divided into sets. A set of similar images in which person A appears by camera 1 is defined as a "shot" (video shot), and a set of events of the same place and time of these shots is defined as "scene".

That is, typically, one scene is composed of a plurality of shots, and one shot is composed of a plurality of image frames. However, depending on the characteristics of the content video scene, the scene may be a shot or an individual frame may be a shot.

Next, we will look at the information (object information) for a specific object. In general, what kind of information needs what object in the actual original image (input file) should be already planned before performing the multimedia authoring / editing process. The planned service information (for example, when the object is a product such as "glasses", product information, price, place of purchase, purchase site, etc.) is systematically managed through the service information manager 111, and the user (editor) requests Is input as metadata of a specific object. That is, the service information previously planned as described above is connected to the corresponding object by the object information input unit (“331” of FIG. 3).

In addition, inputting detailed information using the object information input unit 331 increases the cost of multimedia editing.

If the object-based multimedia editing system 100 stores / manages detailed information about an object through its own database (eg, object / object information DB), it is determined that the object is the same object in the object search process (object In the case where it is determined that the information is stored, detailed information existing in the existing object / object information DB ("36" in FIG. 3) is input as it is. That is, object information is automatically input for a specific object.

On the other hand, if it is recognized as a new object in the object search process, if the corresponding object information is found and input through the search of the web site 112, the editing cost can be reduced.

The service information managed by the service information manager 111 may be transmitted to the e-commerce server 170 through the information server 160 and used to provide a service. That is, the information server 160 may provide an internet shopping service in association with the e-commerce server 170.

Meanwhile, metadata of an object generated as a result of object tracking and editing of the object-based multimedia editing system 100 is converted into BIFS, LASeR, etc. according to a service standard.

The multiplexing device (multiplexer) 130 multiplexes the encoded metadata and the encoded data through the audio / video encoder 120.

Data output from the multiplexing device (multiplexer) 130 is eventually serviced to the output device 150 (eg, set-top box, user terminal, etc.) through the streaming server 140. Here, the demultiplexer (demultiplexer) of the set top box reproduces metadata in a state in which user interaction can be applied, and requests and receives specific information from the information server 160 through the backbone channel when the user interaction is applied. do.

2 is an explanatory diagram of a basic reference unit referred to by the editing unit according to the present invention, and shows which basic reference unit 20 the object-based multimedia editing system 100 corresponding to the editing unit 22 refers to.

The basic reference unit 20 includes a video parser 201, a decoder 202, a low-level image processor 203, a fast mid-level image processor 204, and a high level. -level) There is an image processor 205, an image file processor 206, a video player 207, an encoder 208, and the functions of each processor are generally known, and detailed description thereof will be omitted. .

The object-based multimedia editing system 100 according to the present invention corresponds to the editing unit 22 and uses the basic reference unit 20 in performing object tracking and editing.

On the other hand, the output unit 24 includes service transport stream (Service TS) data, object / object information DB, and metadata. Here, the object / object information DB may be considered to be included in the object-based multimedia editing system 100.

3 is a block diagram of an embodiment of an object-based multimedia editing system according to the present invention.

As shown in FIG. 3, the object-based multimedia editing system 100 according to the present invention includes a control manager 30, a data manager 31, a multimedia image analyzing apparatus 32, a user interface manager 33, and an input. And the output management unit 35 and the object / object information DB 36. Here, the control manager 30, the data manager 31, the user interface manager 33, and the input / output manager 35 may be collectively referred to as a "management part". In particular, the management unit may input the corresponding object information on the searched / tracked object through analysis of the multimedia image analyzing apparatus 32, or perform scene / object editing through a user interface.

Hereinafter, each component will be described in detail.

First, the control manager 30 will be described. The control manager 30 includes an object processing controller 301, a codec manager 302, a frame extractor 303, and an object / object information DB manager 304.

The object processing control unit 301 performs overall control functions related to object processing such as object detection, recognition, tracking, and the like. That is, the object processing control unit 301 controls the frame extraction unit 303 and the object / object information DB management unit 304, and the scene and video shot management unit 311 so that object detection / recognition, object area extraction, etc. can be performed smoothly. ), The feature extraction function management unit 313 for each object, the multimedia image analysis device 32, and the like, and interwork with the user interface management unit 33 to achieve a user interaction ratio.

The codec management unit 302 manages a combination of a detailed function unit of the basic reference unit 20, for example, a video parser 201, a decoder 202, an encoder 208, and the like for processing an input file (video). do.

The frame extractor 303 extracts a frame-by-frame video (hereinafter, simply referred to as a frame) from a multimedia content video corresponding to an input file, and includes a scene and video shot management unit 311 and an individual object information display unit 341 for each frame. To pass). In addition, the frame extractor 303 transmits the frame list to the scene and video shot management unit 311 or / or the individual object information display unit 341 for each frame for tracking or verifying on a frame basis.

The object / object information DB manager 304 manages the object / object information DB 36 and interworks with the object processing controller 301 or the object information structurer 313. In particular, the object / object information DB manager 304 outputs / manages DB stored in the object information structurer 313.

Next, the data management unit 31 will be described. The data manager 31 includes a scene and video shot manager 311, an object information structure manager 312, an object information structurer 313, and an object-specific feature extraction function manager 314.

The scene and video shot manager 311 classifies and stores / manages the scene and the video shot divided by the scene divider 321 to increase the speed of object detection while minimizing the number of objects that are not detected at the same time. In addition, the scene and video shot manager 311 stores / manages a representative frame for each scene or video shot selected by the scene divider 321.

The object information management unit 312 collectively stores and manages object information. The object information management unit 312 stores and manages object information input through the object information input unit 331. Each object information display unit 341 for each frame is edited. The information is output through the information display unit 353 and the metadata output unit 354. Here, the object information includes quantitative information, relational information, an area within a frame, a frame number, a feature vector, event information, and depth information of an object.

The feature extraction function management unit 313 for each object manages a feature extraction function differently for each object type, and applies different feature data and algorithms (algorithms for object detection / recognition) to recognize the object for each object type. For this purpose, accurate object recognition is possible by storing the feature data and the type of feature extraction algorithm for each object. For example, if the feature data and the algorithm are applied to the case of the object recognition and the face recognition in the same manner, it may occur that the recognition itself is impossible. Therefore, it is desirable to apply different algorithms and different feature data according to the object type.

Next, the multimedia image analyzing apparatus 32 will be described. The multimedia image analyzing apparatus 32 includes a scene divider 321, a face detector 322, a face recognizer 323, a non-face object detector 324, a non-face object recognizer 325, an object tracker 326, and an area extractor. 327. This will be described in detail with reference to FIG. 4.

Next, the user interface manager 33 will be described. The user interface management unit 33 includes an object information input unit 331 and an editing unit 332, and the editing unit 332 includes an editing operation control unit 3331 and an editing operation management unit 3322.

The object information input unit 331 receives basic information (eg, object name, etc.) and link information about an object from a user, and enables to check structure information, feature information, area information, etc. between objects on a screen. It provides a user interface that allows you to modify it.

The editing task controller 3331 provides an interface for the user to access tools and solutions required during the editing task. The editing operation management unit 3322 prepares for editing mistakes by recording the progress of the editing operation. That is, the editing task manager 3322 manages the editing history so that the recorded editing process can be canceled and repeated.

Next, the input / output management unit 35 will be described. The input / output management unit 35 includes a metadata output unit 354, a video rendering unit 351, an audio rendering unit 352, and an editing information display system 184.

The video rendering unit 351 renders the video data output from the codec manager 302, and the rendering screen is shared with the editing operation controller 3322.

The audio rendering unit 352 renders audio data output from the codec manager 302 and outputs the audio data to the speaker.

The editing information display unit 353 displays special information of the editing process, thereby minimizing mistakes in the editing process and inducing the necessary editing process.

The metadata output unit 354 generates time, region, and event information of an object stored in the object information structuring system as metadata that conforms to any standard. Such metadata standards include BIFS and LASeR.

Next, the verification management unit 34 will be described. The verification manager 34 includes a frame-specific individual object information display unit 341 and a sequential object information display unit 342.

The individual object information display unit 341 for each frame displays the detection, extraction, and tracking results of each object so that the user can recognize the frame at a glance, and if there is an error, displays the error easily. .

The sequential object information display unit 342 displays the edited object information so that the user can recognize at a glance how the edited object information appears during the reproduction of the input data.

4 is a detailed block diagram of an embodiment of the multimedia image analyzing apparatus of FIG. 3 according to the present invention.

The multimedia image analyzing apparatus 32 robust to scene change according to the present invention divides a multimedia image into predetermined image units, selects a representative frame for each divided image, and analyzes the multimedia image by first selecting the representative frames. will be.

As illustrated in the drawing, the multimedia image analyzing apparatus 32 includes a scene divider 321, a face detector 322, a face recognizer 323, a non-face object detector 324, and a non-face object recognizer 325. , And an object tracker 326. Here, "322" to "325" are referred to as the object finder 40, "401" corresponds to the object detector, and "402" is called the object recognizer. Although not shown in the drawing, various object detectors and object recognizers may be included for each object type (human face, animal face, clothing, etc.) according to the characteristics of the scene.

In the multimedia image analyzing apparatus 32 illustrated in FIG. 3, additional control functions related to object extraction (detection) / recognition / tracking, region extraction, etc. (object feature extraction function management, output of a multimedia analysis result, a user interface function, etc.) Although it is configured to be implemented through a separate management unit according to each type, according to the embodiment may be implemented to operate the control function related to the functional means in each of the corresponding functional means as shown in FIG. Considering the correspondence of FIG. 4, the same identification number as that of FIG. 3 will be used). For example, the object tracker may be configured to perform a function of outputting the tracking result by the object tracker 326 to the user.

The scene divider 321 divides the multimedia image into image units (scenes and shots), and selects a representative frame for each divided image. More specifically, the scene divider 321 is semantically related and temporally related to a series of frames recorded by one camera, i.e., video shots separated by shot boundaries as a physical basic unit of video. Identifies scenes that are collections of neighboring shots and selects a representative frame (representative screen) that best reflects the content of one video scene and / or shot. Here, a frame including the most information among all frames belonging to the divided shot or scene for each divided shot or scene may be selected as the representative frame of the divided shot / scene, and the representative frame selection method is used. If the search is performed in the representative frame, the error can be reduced. The information in such a frame may include low-level information such as focus information and the number of darkened objects (faces and non-faces).

The object finder 40 is an 'object detector' 401 that detects (extracts) a specific object from an image frame (a representative frame, an individual frame), and an object (a search target object, an object to be compared with the detected object). Object) " to determine whether the object is the same. Here, the object detector 401 includes a face detector 322 and a non-face object detector 324, and in some embodiments, may include an object detector corresponding to each object type. The object recognizer 402 is configured in the same manner as the object detector 401.

First, the overall description of the object finder 40 is as follows.

The object finder 40 classifies the representative frames selected by the scene divider 321 according to whether the objects are the same based on the central object type. That is, when the central object type is "person face", the representative frame is classified based on the person face. A representative frame group including an actor A's face, a representative frame group including an actor B's face, and the like, and there are a plurality of feature data for the same actor (for example, a shooting angle, facial expression, etc.). Feature data may vary), a plurality of representative frame groups may exist for the same actor.

Here, the object finder 40 classifies an object (eg, a face of a person) corresponding to a central object type that has been determined to be the same object through an identification mark in a corresponding representative frame, but classifies each other according to a classification group. Use different identification markers, that is, use different identification markers for different classification groups, or use different identification markers such as squares and pentagons. Here, the identification mark may be an image ("1402" in FIG. 14) of a polygonal shape (but not limited thereto) that is color-coded so that the overlapping objects can be seen, which is an object to be identified (a face object). ) Is overlapped (overlay).

In addition, the object searcher 40 may search for an object (eg, a specific actor) by targeting a corresponding representative frame (one or a plurality of representative frames belonging to a specific representative frame group) for each classification group (classified representative frame group). Search for a face, a specific object (glasses of a certain type, etc.), and structure the search result so that the user can view it. That is, the representative frames including the search object are grouped and managed. According to an exemplary embodiment, the search target object may be searched for the representative frames selected by the scene divider 321 without performing the classification process according to the central object type. Here, the objects (recognized faces, specific objects, etc.) that are recognized as the same object as the search object are classified by using the identification mark within the representative frame, but the same identification marks are used for the same objects. This is the same method as described above. The object to be searched for may be an object stored / managed through the database 36 or an object input for a search from a user (editor).

In addition, the object searcher 40 may apply an object search related algorithm differently for each type of search object. That is, an object search related algorithm may be applied differently according to whether the search object belongs to one of a face object and a non-face object. In addition, the object searcher 40 uses a feature-based object detection method for object search in the representative frame.

Hereinafter, the description will be made for each component of the object finder 40 as follows.

The face detector 322 extracts a face region of a person from an arbitrary representative frame, and the face recognizer 323 compares feature vectors of a face to determine whether they are the same person. Since an object has different feature vectors (feature data) for each object type, it is preferable to apply an algorithm for object detection / recognition differently for each object type. In particular, faces have different characteristics from other types of objects (e.g., non-face objects such as "car", "vase", "hand", "clothing", etc., at least face and non-face objects It is desirable to apply different algorithms.

The non-face object detector 324 uses the feature vector of the object / object information DB 36 stored in advance or the feature vector of the object extracted by the user's interaction, to detect the same object in the representative frame of the video shot or scene. Find. Meanwhile, the non-face object recognizer 325 determines whether two objects to be compared are the same object by comparing the extracted feature vectors.

Next, the object tracker 326 will be described.

The object tracker 326 tracks the searched object with respect to individual frames belonging to the representative frame from which the searched object is searched. In particular, the object tracker 326 uses a block-based object tracking method.

In addition, the object tracker 326 detects an object detected in the representative frame of the scene / video shot and locates the same object in every frame included in the video shot or the scene including the representative frame. That is, the object tracker 326 obtains the temporal position (frame number) of the search object and the spatial position within the frame with respect to the individual frames belonging to the representative frame.

In addition, the object tracker 326 recognizes the tracked object as the same object as the searched object and distinguishes the tracked object using an identification mark (as described in the object searcher) within the frame, but the same identification mark for the same objects. Use

Meanwhile, the region extractor 327 (not shown in FIG. 4) accurately extracts the object of interest by minimal user action when the object is extracted by the user interactive. That is, the extracted object of interest does not include other objects of interest.

Object information may be input to the detected / tracked object, and there are automatic input methods using a user input through the object information input unit 331 and an object / object information DB 36. That is, when the object / object information DB 36 has detailed information about the object, when it is determined that the object is the same object in the operation steps of the non-face object detector 324 and the face recognizer 323, the existing object / object information DB ( The detailed information existing in 36) is input as it is (automatic input), and in the case of a new object, the input method by the web search 112 can be used, thereby significantly reducing the editing cost / effort / time. Can be.

FIG. 5 is a flowchart illustrating an object-based multimedia editing method according to the present invention, and shows a method performed in the object-based multimedia editing system 100 having the configuration as shown in FIG. 3.

The multimedia editing method illustrated in FIG. 5 includes (1) a process of grouping objects through scene division for input video files (including object detection process using DB information) 50, (2) user (editor) selection It can be divided into a process 52 for tracking a specific object designated by (3), (3) error verification and editing process 54 by the user (editor). Hereinafter, each process will be described.

When a source file (multimedia content image file, that is, a video file) to be edited is input from a user (editor) (500), a process of detecting a scene and / or a shot from the input source video file is performed. A representative frame (representative screen) is selected for each scene and shot (see 502, Fig. 9). The information about the divided (divided) scene / shot and the representative frame (representative screen) is stored / managed by the front and video shot manager 311.

Since different types of objects have different characteristics, it is preferable to make the approach different for each object type. In FIG. 5 according to the present invention, a case in which a type of a semantic object (center object) is set as a “person face” will be described. The semantic object (center object) is determined according to the scene characteristics of the multimedia content image, and is not necessarily limited to the "person face". This can be the central object. In addition, the central object may be arbitrarily selected by a user (editor).

When the division into scenes / shots and the representative frames 912, 962, and 1401 for each scene / shot are determined (selected), face detection and grouping through the face detector 322 and the face recognizer 323 proceed. (504). That is, after detecting a face object in each representative frame using only the representative frames, the representative frames including objects recognized as the same face are classified into the same group by comparing the detected face objects. For example, in the object tree of FIG. 17, under the "face" 1700, there are the "shared_face" group 1701, the "Yun Eun Hye_face" group 1702, and this is the result of grouping on the face. to be. In addition, the "shared_face" group 1701 includes representative frames including the "shared" face object recognized as the same face object, regardless of whether the included representative frames belong to different scenes or different shots. . Even the same person may be classified into different face objects according to camera angles and facial expressions.

In addition, the face objects having been detected and grouped as described above are registered in the object information manager 312 and updated with the object list of the object tree window 1500 and the object list of the scene / object view window 950.

When the grouping is completed using the face as the center object as described above, it is checked whether there is the object / object information DB 36 in which the additional object to be searched (the object to be searched) is stored (506), and the object / object information. If there is a DB 36, the object detection process using the DB information is performed (508).

The process of "508" is described as follows. After the grouping is performed using the face object as the center object, the object / object information DB 36 has specific object data (for example, a face image of a specific actor, a car object of a specific shape, etc.) to be searched. If it is confirmed, the target object of each group is detected and the object information is input to the detected object. That is, if the object information on the object to be searched is stored in the object / object information DB 36, the object information is copied and automatically input (connected) as the object information on the detected object (that is, the object / object The object information is automatically input through matching with the object stored in the information DB 36). A more detailed description of the process "508" is given below in FIG. 6.

In the case of a general object, the center object (meaning object) is registered by the editor (user) input (in the case of "504", it can be regarded as the case where the face object is set as the center object by default, and in some embodiments, the editor is the center object). Can be set arbitrarily), the location information is registered through the automatic recognition and tracking process for the registered object. However, when the object / object information DB 36 is input together with the input of the original file (input video file), the object having the information in the DB 36 is "object detection process using DB information" (508). Is performed, and an automatic extraction (detection) and tracking process is performed.

The representative frame of the scene / shot detected through the process “504” or “508” is displayed and provided to the user (510).

Thereafter, a scene or shot to be detected / tracked by the user (editor) is selected, and an object of interest (a search target object) is selected in the representative frame of the selected scene or shot (512). Herein, the object of interest refers to an object to be detected / tracked (a search object), and the user is selected in an area designation method such as a square box shape on the representative frame. For example, "glasses" on the face of a particular person may be selected as the object of interest.

When the selection process is made at "512", the region of interest is extracted from the selected scene or the representative frame of the selected shot (if the specific frame is selected, the representative frame of the scene), and then the extracted object region of interest is designated. 514, which will be described in detail with reference to FIG. 7.

Thereafter, the user (editor) may input object information (including event and structure information) with respect to the extracted object (516), and the input object information is stored / managed through the object information management unit 312. .

Then, in all other frames in the selected scene / shot, the same object as the object extracted at "514" is detected / tracked (518). If requested by the user, the tracking result in each frame is provided for the user to view (520).

In other words, the object tracking result of "518" may be represented as an example 930 of the user interface through the object information display unit 341 for each frame (520), through which the editor has the object tracking result in the frame. Will be verified.

Next, an error verification and editing process 54 by a user (editor) will be described.

When all of the "50" and "52" process is terminated, the inspection of the edited material is performed (522). If the process is not performed, the process proceeds to the storage process process 530, and if so, the sequential object information display unit 342 Alternatively, a process 524 of outputting edited information on the edited part is performed through the individual object information display unit 341 for each frame, and if there is an error of information or an editor's correction (526), a manual process of editing a frame ( 528). As described above, the "54" process will be described in detail.

 If there is a verification (inspection) request of the user (editor) (522), the edit information for the edited portion is output (524). That is, the results of editing (broad meaning including scene segmentation, object detection / tracking, etc.) obtained through the processes of “50” and “52” are displayed.

Then, the user (editor) checks whether there is an error in the edit result or the edit information (526) and performs an edit to correct the error (528). For example, if the actor A's face group includes the actor B's face frame, the user may “drag and drop” the wrongly classified frame into the actor B's face group.

Object information (for example, scene segmentation / grouping information, object classification, object detection / tracking information, etc.) obtained through the above “50”, “52”, and “54” processes may be stored in various forms through the storage process. Stored (530). Detailed description thereof will be described with reference to FIG. 8.

FIG. 6 is a flowchart illustrating an object detection method using DB information according to an embodiment of the present invention, and shows “object detection process using DB information” 508 of FIG. 5.

The object / object information DB 36 checks whether there is object information (object data) to be searched for (600), and if there is object data to be searched, it is taken from the object / object information DB 36 (602). If there is no search object data, it is terminated.

Representative frames of the groups generated through the grouping process of "504" are taken from the scene and video shot management unit 311 (606), to determine the object type (face or non-face, such as objects) (608) Object detection (extraction) and recognition algorithms are applied differently according to the type of the object (610 to 616). That is, according to the type of the object to be searched, an appropriate function managed by the object-specific feature extraction function manager 313 is selected and applied.

As a result of the object type check 608, if the search target object is a face, the face region is detected in the representative frame (610) and the face recognition process 612 is performed. If not (non-face), after detecting the object (non-face object) in the representative frame (614), the object recognition process 616 is performed. Here, the face recognition 612 and the object recognition 616 are processes for checking whether the object (face / non-face object) detected in the representative frame and the search target object are the same.

As a result of performing the face recognition 612 or the object recognition 616, if the object does not match, the method returns to the representative frame investigation step 604. If the matching is successful, the object information copying process 620 is performed. That is, the object information stored in the object / object information DB 36 is input (automatically input) to the object recognized as the same object as the search object, and more specifically, the object information management unit 312 provides the object. The contents of the object information DB 36 are copied ("automatically input") (620).

Next, the object tracker 326 tracks the position of the object to be searched for in the entire frame belonging to the scene or shot indicated by the representative frame (representative frame taken from “606”) (622).

There are two types of object tracking methods: block-based tracking and feature-based tracking. The block-based tracking method is very fast in tracking an object, but tracking may be attempted even in a frame in which the object disappears due to a scene change, resulting in the wrong object being tracked. In the case of feature-based tracking, accurate tracking is possible due to the recognition process by the feature vector, but when this method is applied to all frames, the editor has a long waiting time due to the speed problem due to the huge amount of computation. .

Therefore, in the present invention, the problem is solved by dividing into scenes, performing feature-based detection in representative frames of scenes and / or shots, and applying a block-based tracking method to frames belonging to the scenes / shots. Here, in order to reduce the error of tracking the object appearing in the middle of the scene, the representative frames are extracted and managed by taking a more detailed shot, and these representative frames are applied to the feature-based detection process but are not shown in the user interface. By not doing so, it solves the problem and at the same time reduces the number of editor interactions. This is possible because the object of interest shown in the shot is very likely to appear in the scene.

The above process is fed back until all the representative frames are performed (i.e., if there is an unprocessed representative frame, a process of " 606 " or less is performed) 604, and also in the DB 36 It is fed back 600 until it is performed for all searched objects.

Next, the representative screen (representative frame) of the scene (except the shot) detected through the above process through the user interface (see "910" in FIG. 9) (by the editor's choice, in units of "shot" in the scene). By displaying the representative screen (see " 510 " in FIG. 5), the editing process of the actual editor is started. When the editor selects a scene or shot that includes the object of interest and the interfaces 512 and 920 to select the object of interest are activated, the object extraction and object area designation process 514 is executed in the representative frame. As described in 5.

7 is a flowchart illustrating an object extraction and object region designation method in a representative frame according to the present invention, and shows a method performed by the object-based multimedia editing system 100.

When the editor (user) selects the object of interest (object to be searched) through the work tool 903 (512), the object area is set (divided) by using an intelligent image segmentation algorithm for tracking the object of interest. (700).

Then, the feature values for the object are extracted (702), the relationship with other objects is established through object information grouping and hierarchization (704), and stored in the object information manager (312).

Next, an object region approximation process 706 is performed to exclude unnecessary peripheral elements other than the object of interest, and the result data is stored in the object information management unit 312. The object of interest recognized in the above process is input through the object information input units 331 and 1010 (516) and tracked in all other scenes / shots (518).

FIG. 8 is a flowchart illustrating an object information storage process according to the present invention, and illustrates the process “530” of FIG. 5.

The object information managed by the object information management unit 312 may be stored in a metadata file according to the purpose of use, stored in the object / object information DB 36 as metadata, or configured and stored as predetermined standard information. (800).

The editor may check the state of the metadata stored (input) in the storage (storage unit) through the above process (802). That is, the user interface such as 'Preview' checks whether the metadata is stored (input) without error.

Meanwhile, at the request of the editor, the stored (input) metadata may be converted and stored into media (media data) of flash, MPEG 4, or other service type (804).

9 to 12 illustrate a configuration of a user interface screen provided by the object-based multimedia editing system 100 according to the present invention.

The interface screen 900 provided by the object-based multimedia editing system 100 basically includes a menu 901, a main tool 902, and a status bar 970 which an application has. 903), a task window 940, an information window 1040, a scene selection, and an object view window 950.

The task pane 940 is comprised of a scene view pane 910, a media player and a work area pane 920, and a frame view pane 1030. The work tool bar 903 changes its combination according to the active state of the work window 940.

The information window 1040 manages and displays the object information window 1010 for inputting and displaying detailed information of a specific object, all object information windows 1020 for identifying structure and group information between objects, and editors' work contents. The week is composed of a job information window 1030.

The scene selection and object view window 950 consists of a list of objects 951, 952, a time axis 954, and a chronological scene indicator 964.

Hereinafter, each of FIGS. 9 to 12 will be described.

9 is a screen configuration diagram of scene division according to the present invention, which is a representative screen (representative frame) 912 and information (scene number, time information, number of frames, etc.) of a scene generated by the scene divider 321 ( 913 and provide a user interface for editing the split scene itself and going to the edit scene. In addition, the scene view window 910 also indicates that the current scene is being separated (step 904).

The work tool 903 is used to edit the divided scene itself. The information 913 of the actual separated scenes is displayed together with the representative screen 912 when it is displayed on the screen, and is synchronized to the chronological scene indicator 964. In this case, in order to show the connectivity between the time axis 954 and the scene, the scene information 956 on the time axis and the actual representative screen are connected as the same color.

The time axis 954 can be adjusted by the time axis zoom control 953 to allow simultaneous access to details and outlines on the screen with limitations. It also provides the ability to directly select scenes and objects that are hidden by scrolling (959, 957). In FIG. 9, the combo box is used, but the function may be used by any tool.

In addition, in order to provide a convenience of the verification step and a flexible user interface, the function 961 may view only the scene including the object selected in the object list 951. When the scene in which the object of interest (object of interest) exists in the current screen is selected from the representative screen (representative frame) list 911, the next step is started. Of course, the scene divider 321 may be driven by a separate thread so as not to disturb the user interface.

And, through the status bar 970 at the bottom of the screen of FIG. 9, the "Edited file Full Path" 971, the "Current Scene" 972, the "Splitting Scene" 971, and the "" Progress state (974) and the like.

10 is a screen configuration diagram for a task and some verifications according to the present invention.

The interface screen shown in FIG. 10 relates to the operation (editing) and playback of the scene selected in the previous step, and shows a state where the media player and the work area 920 are activated, and the work tool window 903 corresponds to this. There are tools. In the state where the time of the scene selected in the previous step is paused as the current playback time on the work and playback screen 1001, the object selection work is in progress. In relation to the object selection, a tool of the work tool window 903 is selected to provide a function of separating the object via the area extractor 327. Here, since the work screen and the player screen are used at the same time, a control box (control function) 1009 related to playback is provided.

The time navigator 1004 displays the position of the current working scene with the point control 1002, and provides a function of displaying the range of the current working scene over the entire time by placing an area indicator 1003 on the seek bar of the video control. In addition, this time information may include time axis 954 in the scene selection and object view window 950, scene information 956 for the time axis, chronological scene indicator 964, and area indicators in the time axis 954. 955) and the like.

When the separated object is created, it is added to the object list 951, and the object information may be newly input through the object information input unit 331 providing the screen information as shown in FIG. 13 connected to the object. It also provides basic video control functions (1005) related to the operation and allows the "play only current scene" button (1006), "play only current object" button (1007), " Display all objects on the screen "button 1008. In operation 1011, the selected object (object) information may be displayed in relation to the “object information”.

FIG. 11 is a screen configuration diagram of feature-based object detection / tracking according to the present invention and shows a user interface screen 1100 used in feature-based object detection / tracking process 325, 323, 608, 618, or 622.

The object determined in the previous step is also detected / tracked in other scenes and representative frames of all shots. In this process, scenes in which the object is detected / tracked are displayed in the scene list 1106, and a representative screen (representative frame) 1101 and corresponding information 1102 are output for each scene 1103, and the verification / The function of simultaneously presenting the object indicator 1104 for confirmation is provided. In this case, information used by the recognizers 323 and 325 for the recognition is displayed on the scene information item, thereby providing a function for allowing an editor to check the information used by the recognizers 323 and 325. And, "1105" is a screen component indicating that the object is currently being searched.

When a particular scene is selected in the scene list 1106, functions such as a media player and a work area 920 are provided in the "preview" screen 940 to provide a function of editing every moment.

12 is a screen configuration diagram for block-based object tracking / verification according to the present invention, which shows a user interface when the "frame view" 930 is activated.

The frame list screen 1200 individually displays all the frames 1201 included in the current scene received from the frame extractor 303. At this time, each frame is provided with a function of displaying the result (tracked yellow area) 1202 of the tracked object together so that the editor can quickly verify it.

If there is a frame to be modified on the frame list screen 1200, the media player and the work area window 920 may be activated by selecting it, and at this time, the time is determined as the start point of the selected frame, and the state is paused. The editor can then perform the editing and modification process.

FIG. 13 is a block diagram illustrating an object information input interface screen according to the present invention, and shows a user interface screen provided by the object information input unit 331.

The user (editor) inputs general information on the object through the object information input unit 331. That is, the object information input unit 331 receives the name 1301 of the object, information on the representative article 1302 to be shown, any one action 1306 of the unspecified number of actions after the object selection, and the like. Does not limit the specific scope of these actions. However, the list is illustrated (1308) to provide a function supporting various possibilities. And, "1303" is related to "acting after object selection", the specific type thereof is selected through the behavior selection of "1308", an example is the URL shortcut 1305. In connection with URL input, there are "1307" and "1309".

14 is a diagram illustrating a shot view screen according to the present invention.

When a representative frame of a scene divided by the scene divider 321 is displayed on the screen through the scene view window 910, if a scene 912 is selected in the scene view window 910, the selected specific scene is selected. The representative frame 1401 for the shots included in is displayed through the shot viewing window 1400.

Since the shot (video shot) is a series of very similar images, when there is a specific object in the representative frame 1401 for each shot, it is useful to edit and display the area of the object by color coding 1402. That is, if the object management list 951 and the object location information 1543 of the "representative screen including the object" 1542 in the object tree window 1500 are synchronized, the information on the object may be hourly / structured during the inspection process. Very easy to inspect.

15 is a configuration diagram of an object management screen according to the present invention, which shows an example of an object tree window 1500 for managing a divided object.

Object management basically represents a relationship between objects in a tree, and the objects represented in the object tree are divided into virtual objects 1511 and 1512 and real objects 1514.

The virtual object is managed including the real object, and the real object is managed in the screen area of “1540” including the feature data 1541, the frame 1542 including the object, and the location information 1543 of the object. do. The virtual object may include feature data in a higher level concept between the actual objects included therein. The virtual object 1512 may have a lower object 1513. In addition, an object (sunglasses A) 1515 for a specific object may be separately managed.

By managing the objects in this way, it is possible to modify the false recognition objects simply, and to construct and check the relationship diagram between the objects.

The feature data 1541 may be expressed as shown in FIG. 15, but may not be represented. In addition, the object tree screen 1510 supports the editor's inspection by separately representing the object 1516 that needs information input and the object 1517 that needs to be tracked by an icon.

When a newly registered object is selected in the object tree screen 1510 and the "Register as object tracking queue" button 1520 is pressed, the position information of the object registered in a single frame is stored in a continuous frame through the object tracker 326. All location information is automatically selected. Since this process is performed by a background processor, information about an object can be input at the same time, thereby reducing the editing time. In order to input information about an object, the object information input window 1010 is activated by inputting the information necessary by selecting the object and clicking the information input button 1530.

On the other hand, when the same object (object to be searched) exists in the object / object information DB 36, the data received from the DB 36 is generated in advance, and the editor corrects it if necessary. In the case of a new object, the editor can directly input it, but it is also possible to input it conveniently using the search result of the web site 112.

16 is a block diagram of the object tracking information screen according to the present invention and shows a window 1600 showing object tracking information.

By assigning object tracking and recognition steps to the Background Processor, it saves the entire editing time by providing editors with a way to do manual processing other than automatic processing.

FIG. 16 illustrates a process of automatically tracking an object 1601 corresponding to "Shared_Sunglasses A_Front" 1514, "Shared_Sunglasses A_Front", and "Shared_Sunglasses A_Front2" of FIG. 15. Indicates.

FIG. 17 is a diagram illustrating a screen configuration according to a scene divider, a shot view, and a face grouping according to the present invention, and FIG. 18 is a diagram illustrating another screen configuration for work and partial verification according to the present invention.

On the other hand, the method of the present invention as described above can be written in a computer program. And the code and code segments constituting the program can be easily inferred by a computer programmer in the art. In addition, the written program is stored in a computer-readable recording medium (information storage medium), and read and executed by a computer to implement the method of the present invention. The recording medium may include any type of computer readable recording medium.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

Claims (42)

In the multimedia image analysis apparatus robust to scene change, Scene dividing means for dividing the multimedia image into predetermined image units and selecting a representative frame for each divided image; And Object retrieval means for classifying the selected representative frames according to whether objects are identical based on a central object type Multimedia image analysis device for including. The method of claim 1, The object searching means, The multimedia image analyzing apparatus further performing a function of searching a search target object for a corresponding representative frame for each classification group. The method of claim 2, Object tracking means for tracking the search object for individual frames belonging to the searched representative frame of the search object Multimedia image analysis device further comprising. The method of claim 1, The object searching means, The object corresponding to the central object type, which has been determined to be the same as an object, is classified through an identification mark in a corresponding representative frame, and uses different identification marks for each classification group. The method of claim 4, wherein The identification mark, Multimedia image analysis apparatus, characterized in that the image is displayed through the image superimposition method, the image of the polygonal form is color-coded so that the overlapping objects can be seen. The method according to any one of claims 1 to 3, The central object type is, The multimedia image analyzing apparatus, characterized in that automatically determined based on the scene characteristics of the multimedia image or selected by the user. The method of claim 6, The central object type is, Multimedia image analysis device, characterized in that the face of the person. In the multimedia image analysis apparatus robust to scene change, Scene dividing means for dividing the multimedia image into predetermined image units and selecting a representative frame for each divided image; And Object searching means for searching for a search target object based on the selected representative frames Multimedia image analysis device comprising a. The method of claim 8, Object tracking means for tracking the search object for individual frames belonging to the representative frame from which the search object is searched Multimedia image analysis device further comprising. The method according to claim 2 or 8, The object searching means, And managing the representative frames including the search target object by grouping them. The method according to any one of claims 2, 3, 8, or 9, The search target object, And at least one of an object stored / managed through a database or an object input by a user. The method according to claim 1 or 8, The video unit, Multimedia image analysis device, characterized in that any one of the scene or video shot. The method of claim 12, If the video unit is a video shot, A scene analysis apparatus for performing scene segmentation for user editing, and segmenting the multimedia image by shot unit for each divided scene. The method according to claim 2 or 8, The object searching means, Multimedia image analysis device for applying a different object search-related algorithm for each type of the search object. The method of claim 14, The object searching means, And an object searching related algorithm is applied differently according to whether the object to be searched belongs to one of a face object and a non-face object. The method according to claim 2 or 8, The object searching means, An object recognized as the same object as the search target object and searched for is classified by using an identification mark within a corresponding representative frame, and the same image identification apparatus uses the same identification mark. The method according to claim 2 or 8, The object searching means, Multimedia analysis apparatus for performing the object search in the representative frame using a feature-based object detection method. The method according to claim 3 or 9, The object tracking means, Multimedia analysis apparatus for performing the object tracking in the individual frame using a block-based object tracking method. The method according to claim 3 or 9, The object tracking means, An object identified and tracked as the same object as the search target object is classified using an identification mark in a corresponding frame, and the same identification mark is used for the same objects. The method according to claim 3 or 9, The object tracking means, And a spatial position (frame number) of the search object and a spatial position within the frame with respect to individual frames belonging to the representative frame. The method according to claim 3 or 9, Means for collectively inputting corresponding object information on the object tracked through the object tracking means; Multimedia analysis device further comprising. The method according to claim 1 or 8, The scene dividing means, And selecting a frame including the most information among all the frames belonging to the divided image unit as the representative frame of the divided image. In the multimedia image analysis method that is robust to scene change, A division step of dividing the multimedia image into predetermined image units; A representative selecting step of selecting a representative frame for each divided image; And A classification step of classifying the selected representative frames according to whether objects are identical based on a central object type. Multimedia image analysis method comprising a. The method of claim 23, An object searching step of searching for an object to be searched based on a representative frame corresponding to the classification group generated in the grouping step; Multimedia image analysis method further comprising. The method of claim 24, An object tracking step of tracking the object to be searched for individual frames belonging to the representative frame in which the object to be searched is searched Multimedia image analysis method further comprising. The method of claim 23, The classification step, The object corresponding to the central object type, which has been determined to be the same as an object, is classified through an identification mark within a corresponding representative frame, and uses different identification marks for each classification group. The method according to any one of claims 23 to 26, The central object type is, The multimedia image analysis method, characterized in that it is automatically determined based on the scene characteristics of the multimedia image or selected by the user. The method of claim 27, The central object type is, Multimedia image analysis method characterized in that the face of the person. In the multimedia image analysis method that is robust to scene change, A division step of dividing the multimedia image into predetermined image units; A representative selecting step of selecting a representative frame for each divided image; And An object searching step of searching for a search target object based on the selected representative frames; Multimedia image analysis method comprising a. The method of claim 29, An object tracking step of tracking the object to be searched for individual frames belonging to the representative frame in which the object to be searched is searched Multimedia image analysis method further comprising. The method of claim 24 or 29, The object search step, And managing representative frames including the search target object by grouping them. The method according to any one of claims 24, 25, 29 or 30, The search target object, A multimedia image analysis method comprising at least one of an object stored / managed through a database or an object input by a user. The method of claim 23 or 29, The video unit, Multimedia image analysis method, characterized in that any one of the scene or video shot. The method of claim 24 or 29, The object search step, The multimedia image analysis method of applying an object search-related algorithm differently for each type of the search object. The method of claim 24 or 29, The object search step, And identifying objects that are recognized and searched as the same object as the search object by using identification marks in the representative frame, and using the same identification marks for the same objects. The method of claim 24 or 29, The object search step, Multimedia analysis method for performing the object search in the representative frame using a feature-based object detection method. The method of claim 25 or 30, The object tracking step, Multimedia analysis method for performing the object tracking in the individual frame using a block-based object tracking method. The method of claim 25 or 30, The object tracking step, And identifying the tracked objects that are recognized as the same object as the search target object by using identification marks in the corresponding frame, and using the same identification marks for the same objects. The method of claim 25 or 30, The object tracking step, And obtaining a temporal position (frame number) of the search object and a spatial position within the frame with respect to individual frames belonging to the representative frame. The method of claim 25 or 30, Collectively inputting corresponding object information on the object tracked through the object tracking step; Multimedia analysis method further comprising. In the object-based multimedia editing system, Multimedia image analyzing means for dividing the multimedia image into predetermined image units to select a representative frame for each divided image, and analyzing the multimedia image based on the selected representative frames; And Management means for inputting corresponding object information to the searched / tracked object through analysis of the multimedia image analyzing means or performing scene / object editing through a user interface Multimedia editing system comprising a. 42. The method of claim 41 wherein The multimedia image analysis means, 10. The multimedia editing system according to any one of claims 1, 2, 3, 8, or 9, wherein the multimedia image analyzing apparatus is used.