KR20010089664A - System and method for three-dimensional modeling - Google Patents

Abstract

Image processing systems and methods describe providing a three-dimensional model using information from image frames of an input pair. Three-dimensional surfaces are obtained by first identifying features in the input frame. Feature correspondence matching is performed using the difference information based on the frame. The identified features may also be correlated using time information.

Description

System and method for three-dimensional modeling}

Video / image communication applications over the Internet or public switched telephone network (PSTN) are growing in popularity and usage. In conventional video / image communication technology, an image (JPEG or GIF format) is captured and then transmitted over a transmission network. However, this approach requires large bandwidth because of the size of the picture (i.e. the amount of data).

When coding moving images with data rates between 64K and 2Mbits / sec, block-based hybrid coders are typically used. The coder subdivides the sequence of each image into blocks that move independently. Each block is then coded by two-dimensional motion prediction and transform coding. Depending on the transmission speed, the received result image does not run smoothly and does not run in real time.

The method has been used to enhance video / image communication and / or reduce the amount of information that needs to be transmitted. One such method has been used for videophone applications. The image is encoded by three sets of parameters that define its motion, shape and surface color. Since the subject of visible communication is typically a human, the main focal point may be directed at the subject's head or face.

In a typical videophone communication system, a single camera is typically used to acquire an image of the person making the video call. Since only one camera is used, it is difficult to obtain a true three-dimensional (3D) facial surface (ie, shape parameter). Typically, to create a three dimensional surface, a number of two dimensional viewpoints (usually six) to the object are required. Using this point in time, a distance deformation is applied. For example, an ellipse can be used as a generation function to obtain the shape of the Z coordinate at the object as a function of the distance to the object's boundary. This contour can only approximate a three-dimensional shape.

Another embodiment is called model based coding. Low bit rate communication can be achieved by encoding and transmitting only each facial parameter of the subject's head. In remote places, face images are synthesized using the transmitted parameters. In general, model-based coding requires at least four tasks: facial segmentation, facial feature extraction, feature tracking, and motion estimation.

One known method for segmentation of faces is to generate a dataset describing the parameterized face. This dataset defines the three-dimensional description of the face object. The parameterized face is provided as an anatomical-based structure by modeling muscle- and skin-actuator and force-based deformations.

As shown in FIG. 1, a set of polygons defines a human face model 100. The vertex of each polygon is defined by the X, Y, and Z coordinates. Each vertex is identified by an index number. A particular polygon is defined by a set of indices surrounding the polygon. In addition, the code may be added to a set of indices, specifying a color for a particular polygon.

Systems and methods are also known for analyzing digital images, recognizing human faces and extracting facial features. Conventional facial feature determination systems use methods such as facial color tone determination, template matching, or edge detection approaches.

One of the most difficult problems in model-based coding is providing fast, easy and robust facial feature mapping. In sequential frames, the same facial features must match exactly. Typically, block-matching processing is used to compare the pixels in the current frame and the next frame to determine the feature correspondence. When the entire frame is retrieved for the feature correspondence, the processing is slow and may produce inaccurate results due to mismatching of regions having the same slope value. If only a subset of frames are retrieved, processing time can be improved. In this case, however, the process may fail to determine any feature correspondence.

There is a technical need for improved systems and methods for three-dimensional modeling of objects contained in digital images for such reduced data rate transmission.

FIELD OF THE INVENTION The present invention generally relates to the field of three-dimensional modeling, and in particular, the present invention relates to systems and methods for three-dimensional modeling of objects contained in digital images using information-based disparity. will be.

1 is a schematic front view of a human face model used for coding based on a three-dimensional model.

2 is a block diagram of a three-dimensional modeling system in accordance with an aspect of the present invention.

3 is a block diagram of an example computer system capable of supporting the system of FIG.

4 is a block diagram illustrating the architecture of the computer system of FIG.

5 is a block diagram illustrating an exemplary apparatus in accordance with a preferred embodiment of the present invention.

It is an object of the present invention to address the limitations of conventional video / image communication systems and coding based on the models described above.

It is another object of the present invention to provide an object-oriented, cross-platform method for deriving real-time compressed video information.

It is another object of the present invention to enable coding of a particular object in an image frame.

It is another object of the present invention to complete synthetic and natural visible objects interactively or in real time.

In one aspect of the present invention, an image processing apparatus includes at least one feature extraction determiner configured to extract feature position information from a pair of input image signals and a corresponding feature in the input image signal according to the feature position information and the disparity information. It includes a matching unit to match.

One embodiment of the invention is directed to a method of determining parameters associated with a three-dimensional model. The method includes extracting feature position information associated with the pair of input images and matching the feature correspondent to the pair of input images according to the extracted feature position information and disparity information. The method also includes determining a parameter for the three-dimensional model in accordance with feature correspondence matching.

These and other embodiments and aspects of the invention are illustrated in the following detailed description.

The features and advantages of the present invention can be understood by reference to the detailed description of the preferred embodiments described below in conjunction with the drawings.

Referring to FIG. 2, a three-dimensional modeling system 10 is shown. In general, the system 10 includes at least one feature extraction determiner 11, at least one set of time information 12, and a feature correspondence matching unit 13. The left frame 14 and the right frame 15 are input to the system 10. The left and right frames consist of image data, which can be digital or analog. If the image data is analog, analog-digital circuitry can be used to convert the data into a digital format.

Feature extraction determiner 11 determines the feature position (position / location) in the digital image, such as the feature position of the nose, eyes and mouth of the face. Two feature extraction determiners 11 are shown in FIG. 2, where one determiner can be used to extract positional information from left and right frames 14 and 15. The temporal information 12 includes data such as previous and / or future frames used to provide constraints on the exact feature correspondence. As will be appreciated, the current frame to be processed requires a first frame input to the system 10. Test frames are used to establish some hysteresis.

In a preferred embodiment, system 10 is executed by computer readable code executed by a data processing apparatus. The code may be stored in memory in the data processing device and read / downloaded from a memory medium such as a CD-ROM or floppy disk. In other embodiments, hardware circuitry may be used in place of or in combination with software instructions to carry out the present invention. For example, the present invention can be implemented on a digital television platform using a Trimedia processor for processing and a television monitor for display. The invention may also be practiced on the computer 30 shown in FIG.

As shown in FIG. 3, the computer 30 interfaces with a network connection 31 for interfacing to a data network, such as a variable bandwidth network or the Internet, and another remote location such as a video or digital camera (not shown). And a fax / modem connection 32 for the purpose of connection. The computer 30 also includes a display 33 for displaying information (including video data) to the user, a keyboard 34 for entering text and user commands, and a cursor on the display 33. A mouse 35 for inputting a user command, a disk drive 36 for reading from and writing to a floppy disk to be installed in the disk drive, and a CD-ROM drive for accessing information stored on the CD-ROM. (37). The computer 30 also has one or more peripheral devices attached to the computer, such as a pair of video conference cameras for inputting images and the like, and a printer 38 for outputting images, text, and the like.

4 illustrates an internal structure of a computer 30 that includes a memory 40 that may include a computer readable medium such as random access memory (RAM), read-only memory (ROM), and a hard disk. Items stored in memory 40 include an operating system 41 and data 42, and an application 43. The data stored in memory 40 may also include time information 12. In a preferred embodiment of the present invention, operating system 41 is a Windows operating system such as UNIX, although the present invention can be used not only in Microsoft Windows 95 but in other operating systems. Among the applications stored in the memory 40 are a video coder 44, a video decoder 45, and a frame grabber 46. Video coder 44 encodes the video data in a conventional manner, and video decoder 45 decodes the video data that was encoded in the conventional manner. Frame grabber 46 allows a single frame to be captured and processed in the video signal stream.

In addition, a central processing unit (CPU) 50, a communication interface 51, a memory interface 52, a CD-ROM drive interface 53, a video interface 54, and a bus 55 are connected to the computer 30. Included. The CPU 50 includes computer readable code outside the memory 40 as described above, that is, a microprocessor or the like for executing an application. Such an application may be stored in memory 40 (as described above) or alternatively on a floppy disk of disk drive 36 or in a CD-ROM in CD-ROM drive 37. The CPU 50 accesses the application (or other data) stored on the floppy disk via the memory interface 52, and accesses the application (other data) stored on the CD-ROM through the CD-ROM drive interface 53. do.

Application execution and other tasks of the computer 30 may be initiated using the keyboard 34 or the mouse 35. The output from the application running on the computer 30 can be displayed on the display 34 to the user or alternatively can be output via the network connection 31. For example, input video data may be received via video interface 54 or network connection 31. The input video data can be decoded by the video decoder 45. The output video data may be coded by the video coder 44 for transmission over the video interface 54 or the network connection 31. Display 33 preferably includes a display processor for forming a video image based on decoded video data provided by CPU 50 via bus 55. Output results from various applications may be provided to the printer 38.

Referring to FIG. 2, the left frame 14 and the right frame 15 preferably comprise a pair of stereo digital images. For example, the digital image may be received from two (still photo or video) cameras 60 and 61 (shown in FIG. 5) and stored in memory 40 during subsequent processing. Other frames or frame pairs obtained with an angular or visual difference can also be used. Cameras 60 and 61 may be part of another system, such as a video conferencing system or animation system.

Cameras 60 and 61 are located close to each other and subject 64 is located a short distance away from cameras 62 and 63. As shown in Fig. 5, the cameras 60 and 61 are separated from each other by a distance b (center-center). Subject 62 is separated by distance f from each of cameras 60 and 61. Preferably, b equals approximately 5 to 6 inches and f equals approximately 3 feet. However, it is to be understood that the present invention is not limited to this distance, which distance is for illustrative purposes only.

Preferably, the camera 60 takes the front side and the camera 61 takes the offset or side of the object 62. This allows a comparison that will consist of the left frame 14 and the right frame 15 to determine the difference map. In the preferred embodiment of the present invention, the left frame 14 (image A) is compared with the right frame 15 (image B). However, reverse comparison can also be performed.

A digital frame or image can be conceptualized by including a number of horizontal scan lines and a number of vertical columns forming array pixels. The number of scan lines and columns determines the resolution of the digital image. To determine the difference map, a scan line is secured, for example scan line 10 of image A matches scan line 10 of image B. The pixel on scan line 10 of image A then matches its corresponding pixel in scan line 10 of image B. So, for example, the fifteenth pixel of scan line 10 of image A matches the tenth pixel of scan line 10 of image B, and the difference is calculated as follows: 15-10 = 5. It will be appreciated that when the left and right cameras 60 and 61 are located close, for example the pixels of the frontmost information of the human face will have a greater difference than the pixels of the background information.

The difference map based on the difference calculation may be stored in the memory 40. Each scan line (or column) of an image has a profile that includes a difference for each pixel in that scan line (or column). In this embodiment, the grayscale level of each pixel includes the magnitude of the difference calculated for that pixel. The darker the grayscale level, the lower the difference.

The difference threshold may be chosen for example 10, any difference above the difference threshold indicates that the pixel is the frontmost information (i.e. subject 64), and any difference less than 10 indicates that the pixel is background information. Indicates that The selection of the difference threshold is based on a portion of the camera distance described above. For example, a lower difference threshold may be used when the object 62 is located farther from the cameras 60 and 61, and a larger difference threshold may be used when the cameras 60 and 61 are farther from each other. Can be used.

The difference map is used to extract the coordinate or facial feature locations from the left and right frames 14 and 15. Preferably, the systems and methods described in US patent application 08 / 385,280, filed August 30, 1999, include feature extraction determiner 11. Preferably, facial feature locations include the head, nose, and mouth positions as well as the outline positions of the head. With reference to FIG. 1, this position correlates with various vertices of face model 100. For example, with respect to the nose, facial feature extraction determinants preferably provide information directly related to vertices 4, 5, 23 and 58, as shown in FIG. 1.

However, feature extraction determiner 11 provides only the X and Y coordinates of facial features. The feature counterpart matching unit 13 provides the Z coordinate. Preferably, the feature extraction determiner 11 uses a triangulation procedure based on the estimation of the position of the three-dimensional point provided in the perspective views on the left and right stereo image frames 14 and 15. For example, providing the X and Y coordinates of feature points F _L and F _R in left and right frames 14 and 15, the three-dimensional surface (ie, Z or depth information) can be determined by the equation Can be.

[expression]

Z = f * b / (| F _L -F _R |),

Here, the distance f (shown in FIG. 5) is the focal length of the cameras 60 and 61,

Distance b (shown in FIG. 5) is the baseline distance between cameras 60 and 61,

| F _L -F _R | represents the difference calculated as described above.

In this embodiment, the equation provides a relationship between the surface Z and the difference under some geographical conditions. In particular, at the front of each camera the image plane is the focal length f and the two cameras are oriented equal to the X axis of the camera frame of reference, which is directed along the line defined by the position of the cameras 60 and 61. It is assumed that the focal lengths of the cameras 60 and 61 are the same. It is also assumed that any geographic distortion of the lenses of cameras 60 and 61 is compensated for. Other geographic arrangements can also be used. However, the relationship between the difference and the surface Z becomes more complicated.

The other vertices of the face model 100 shown in FIG. 1 are interpolated and extrapolated based on the position in the feature extraction determiner 11 (ie, facial feature vertex information) and the determination in the feature correspondence matching unit 13. Can be. Interpolation can be based on linear or nonlinear or scalable models or functions. For example, a vertex between two other known vertices can be determined using a predetermined parabolic function where all three vertices are satisfied. Other face models with additional vertices can also be used to provide enhanced or improved modeling results.

The face model 100 shown in FIG. 1 is a general face with an obscure facial expression. Control of the face model 100 is scalable. The face model 100 template may be stored or loaded at a remote location before any communication is initiated. Using extracted facial features, polygon vertices can be adjusted to match a particular person's face more closely. In particular, based on the processing and information performed by the feature correspondence matching unit 13 and the feature extraction determiner 11, the face model 100 template is suitable for enabling movement, facial expression, and animate audio. (Ie voice) to match the screen. In essence, the generic face model 100 dynamically transforms to a particular face in real time. Real-time or non-real time transmission of model face parameters / data provides low bit rate animation of the synthetic facial model. Preferably, the data rate is 64 Kbit / sec or less, but 64 Kbit to move the image. Data rates between / sec and 4Mbit / sec are also possible.

In another embodiment, the time information 12 is used to verify the feature matching result from the feature correspondence matching unit 13 and / or to perform an alternative feature matching process. In this embodiment, for example, matching is performed only by the feature correspondence matching unit 13 on the selected frame, preferably on the "key" frame (in MPEG format). As soon as a key frame matches a feature, the correspondence matching (ie depth) of the feature in another non-key frame (or other key frame) can be determined by tracking the corresponding feature point in a temporal manner. . If an initial feature correspondence is provided, the three-dimensional motion may be determined from two views to a scale in one translation direction (i.e., time information 12 may be two left or two right sequential or Preferably, the feature correspondence matching unit 13 is used to periodically feature match another key frame to remove any established error from temporal feature matching. The feature correspondence matching unit 13 may be configured to perform feature correspondence matching and time correspondence matching as necessary. Temporal feature matching can be performed faster than feature counter matching which is advantageous for real time processing.

The present invention can be applied in many fields such as video conferencing and animation / simulation of real objects, or in any application requiring object modeling. For example, typical applications include videogames, multimedia products and improved navigation through the Internet.

In addition, the present invention is not limited to a three-dimensional face model. The invention can be used in models of other physical objects and scenes, such as three-dimensional models of cars and rooms. In the present embodiment, the feature extraction determiner 11 collects position information related to the specific object or scene in question, for example, the position of the wheel or the position of the furniture. Then another process is based on this information.

While the invention has been described above by way of specific embodiments, it will be appreciated that the invention is limited or not limited to the embodiments described herein. For example, the invention is not limited to any particular type of filtering or mathematical transformation, or to any particular input image scale or object. On the contrary, the invention covers various structures and modifications of the invention, which come within the spirit and scope of the appended claims.

Claims (11)

In the image processing apparatus 10, At least one feature extraction determiner 11 configured to extract feature position information from the pair of input image signals 14, 15, and An image processing apparatus (10) comprising a matching unit (13) coupled to said feature extraction determiner (11) arranged to match corresponding features in input image signals (14, 15) according to feature position information and difference information ). An image processing apparatus (10) according to claim 1, wherein the matching unit outputs three-dimensional (3D) information related to the input images (14, 15). The image processing apparatus (10) of claim 2, wherein the three-dimensional surface information is based on a predetermined model (100). 5. An image processing apparatus (10) according to claim 4, wherein said predetermined model is a human face model (100). 2. An image processing apparatus (10) according to claim 1, wherein said matching unit performs a matching on at least one frame (14) of input image signals and a temporal feature matching (12) on at least one other frame. 7. An image processing apparatus (10) according to claim 6, wherein said temporal feature matching is performed using sequential input image frames. 7. An image processing apparatus (10) according to claim 6, wherein one frame is a key frame. In the method of determining the parameters associated with the three-dimensional model 100, Extracting feature position information associated with the pair of input images 14, 16; Matching corresponding features to a pair of input images according to the extracted feature position information and the difference information, and Determining parameters for the three-dimensional model in accordance with the result of the feature correspondence matching. 10. The method of claim 9, wherein the three-dimensional model is a human face model (100). A method of coding an object in a digital image for transmission, said method comprising: Extracting feature position information associated with the at least one pair of digital images 14, 15; Matching corresponding features in the pair of digital images according to the extracted feature position information and the difference information, and Coding the information for transmission in accordance with the result of the feature correspondence matching. 12. The method of claim 11, wherein the information for transmission includes parameters related to the three-dimensional model (100).