JP3533754B2 - Image synthesis device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus for synthesizing a moving image in which an object image in a moving image previously recorded is replaced with another input object image.

[0002]

2. Description of the Related Art In recent years, it is possible to input an image from an image scanner, a video camera, or the like, and synthesize the image with a prerecorded background or foreground image, or to synthesize an image obtained by mixing a plurality of images. The device is realized. In particular, a face image is often handled as a target image, and a technique for synthesizing an image in which all or part of the face of a pre-recorded image is replaced with the face image of another person is used for criminal investigation. It is thought that it will spread not only to practical applications such as montage systems and videophones, but also to the arts and amusement fields.

As an example of the prior art for realizing the image synthesizing apparatus as described above, in the case of treating a face image as an object,
By utilizing the fact that the relative positional relationship of each part such as human eyes and nose is generally common, standard structural information is associated with face images, and structural information between multiple facial images is associated. There is a technique for synthesizing images by using correspondence.

For example, according to a research report by Harashima et al. (Article SIG-HICG-8901-3 of the Society for Artificial Intelligence), three-dimensional structural information of a face is approximated by a polyhedron for two face images. It is said that it is possible to interpolate the textures of the two face images and the reference face models corresponding to the respective face images to synthesize a so-called split image by giving a corresponding relationship with the reference face model.

When this technique is used, a reference face model is made to correspond to an image containing a face recorded in advance, and a correspondence relationship with the reference face model is obtained for an inputted face image. It is possible to replace the face portion of the recorded image with the input face image.

[0006]

However, in the above-described method according to the conventional technique, the shape of the input face image is generally different from the shape of the face image recorded in advance, and therefore the shapes of the two face images are different. There is a possibility that a part without a background image may occur at the boundary part between the face and the background due to the difference between, and there is a problem that a natural image cannot be synthesized. For example, when a face in an image containing a prerecorded background is horizontally long and is replaced with an input vertically long face image, in order to obtain a natural composite image, the left and right sides of the recorded face image are Background images at both ends are required, but it is difficult to obtain a natural composite image because there is no information about this part.

The above problem can be solved by a method of recording extra information of a background image in advance. However, when the above-mentioned conventional technique is applied to a moving image,
It is possible to synthesize a moving image by repeating the same method for each frame in a prerecorded moving image. However, when recording a moving image, the capacity is very large, and therefore, it is necessary for each frame. The extra recording of extra image information has a problem that the storage capacity is further increased.

Therefore, according to the present invention, since a moving image to be recorded in advance can be layered and recorded by processing it in advance, the moving image can be efficiently coded by layering. Moreover, it is obtained by paying attention to the fact that it is possible to obtain a background image that cannot be obtained from one frame image.

Therefore, in view of the above problems, the present invention generates a natural moving image with no background loss when replacing an object image in a previously recorded moving image with another input object image. In addition, it is an object of the present invention to provide an image synthesizing device capable of recording a moving image with a smaller storage capacity.

[0010]

In order to achieve the above object, the present invention provides structure information time-series data having structure information indicating a correspondence relationship between an object image in each frame in a moving image and the structure of the object. a database unit for recording and constitute a hierarchical image coding coded moving picture data to the moving image is separated in a spatial context, it reads the encoded moving image data from the previous SL database unit, stratified a hierarchical image decoding unit that generates a hierarchical image a plurality of images respectively decoded was, from the read structure information time-series data from said database unit, structure information selection unit for selecting and outputting structure information that corresponds And an input image of another object having a structure similar to that of the object in the moving image and input structure information which is structure information corresponding to the input image, respectively, and the structure information Processing the structural information using the output structure information in selecting section and the input structure information, further wherein the input image object image synthesizing unit you synthesize synthetic object image using a hierarchy said decoding A hierarchical image synthesizing unit that superimposes the transformed image and the synthesis target image and outputs a synthesized image is provided.

[0011]

According to the present invention, by using the above-described structure, it is possible to record a moving image with a small storage capacity by using the encoded moving image data in which the moving image is separated in the context of the hierarchical image. And, even when the shape of the object image in which the combined object image is recorded is different,
By superimposing and combining the decoded background image and the like as another hierarchical image, it is possible to generate a natural moving image with no loss of the background image or the like.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image synthesizing apparatus according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of an image synthesizing apparatus according to an embodiment of the present invention.

In FIG. 1, the database unit 101 is
Structure information time-series data 112 indicating a correspondence relationship between the object image and the structure of the object in each frame in the moving image in advance,
Encoded moving image data 111, in which moving images are separated according to the context, form hierarchical images and are encoded, is recorded. The hierarchical image decoding unit 102 reads the encoded moving image data 111 from the database unit 101, and combines the hierarchical image 11 into a composite image.
7 is written in the image memory unit 105, and at the same time, the frame information 115 indicating the position of the frame to be decoded is output.

The structure information selecting unit 103 reads the structure information time-series data 112 recorded in the database unit 101, selects the structure information 116 corresponding to the frame information 115, and outputs it. Based on the input image 113 input from the outside and the input structure information 114 corresponding thereto, the object image synthesis unit 104 determines the correspondence between the structure information 116 output from the structure information selection unit 103 and the input structure information 114. , The input image 113 by transforming the input structure information 114
Is used to generate a synthesis target image 118 and write it in the image memory unit 105. The image memory unit 105 displays the hierarchical image 1
17 and the composite object image 118 are temporarily stored. The hierarchical image synthesis unit 106 reads the hierarchical image 117 and the synthesis target image 118 from the image memory unit 105, superimposes them, and outputs them as one synthetic image.

The operation of the image synthesizing apparatus configured as described above will be described below with reference to FIGS. 1, 2 and 3. FIG. 2 is a diagram showing a method of encoding and decoding the encoded moving image data 111 recorded in the database unit 101 of FIG. In the figure, reference numeral 201 is a base moving image, and a layered image 202 is generated by extracting regions and the like described by the same motion parameter from the moving image and overlapping them. By encoding each of the hierarchical images 202, the encoded moving image data 203 recorded in the database unit 101 can be obtained, and this is recorded in advance.

Regarding the method of hierarchically encoding and decoding moving images and the detailed configuration of a device for realizing the method, Japanese Unexamined Patent Publication (Kokai) No. 6-253400 and research report by J. Wang et al. and E. Adelson: "Layered Representati
on for Image Sequence Coding ", Proc. IEEE Int. Con
f. Acoustic Speech Signal Processing '93, pp.V221-
224, 1993 and J. Wang and E. Adelson: "Layered R
epresentation for Motion Analisis ", Proc. Computer
Vision and Pattern Recognition, pp.361-366, 1993)
Etc.

FIG. 3A shows the database unit 10 of FIG.
A part of the structural information time series data 112 recorded in 1;
The relationship between the input structure information 116 and the corresponding image is illustrated, and FIG. 3B is a specific example of the structure information. Each structure information includes at least information about the structure of the object and the position in the image. In the example of FIG. 3, the structure of the object is described by the structure of a polyhedron composed of a set of polygons, and the vertices of each polygon correspond to the coordinate values in the two-dimensional coordinate space of the image. . As long as the objects have similar structures, structural information having the same structure can be associated with different objects. For example, structural information such as eyes and nose in a face image can be associated. The parts having a common structure are associated with vertices and polygons at specific positions in the structure information.

Such an image and the structural information corresponding to it can be manually given in advance, and Kaku et al.'S research report (Yoshiyasu Kaku et al .: "3D face image system capable of facial expression and age change"). ,, IEICE Technical Report, HC-9
1-43, 1992) and the like also disclose a method of automatically generating structural information from standard structural information.

Coded moving image data 111 having the above contents
The structure information time-series data 112 is recorded in the database unit 101 in advance. First, after the input image 113 and the input structure information 114 are input, the operation is started according to a user's instruction or the like, and the following operation is repeated for each cycle of one frame of the moving image.

The hierarchical image decoding unit 102 reads the encoded moving image data 111 from the database unit 101 and sequentially decodes it for each frame. The plurality of hierarchical images 117 obtained by decoding are written in the image memory unit 105. Since the detailed method of decoding is described in the above-mentioned document, detailed description is omitted here. The hierarchical image decoding unit 102 outputs the frame information 115 indicating the position of the frame being decoded simultaneously with the decoding.

Next, the structure information selection unit 103 inputs the frame information 115 output from the hierarchical image decoding unit 102, reads the structure information time-series data 112 recorded in the database unit 101, and stores it in the frame information 115. The corresponding structure information 116 is selected and output. In the object image synthesis unit 104, the input image 113 input from the outside
And the input structure information 114 corresponding thereto, based on the correspondence between the structure information 116 output from the structure information selection unit 103 and the input structure information 114, the input structure information 114 is modified and combined using the input image 113. Object image 118
Is generated and written in the image memory unit 105.

Specifically, for example, the structure information selection unit 10
The average value of the information on the position coordinates of the image in the structure information 116 output from No. 3 is calculated, and the input structure information 114 is processed so as to match the average of the position coordinates in the input structure information 114. By replacing each polygon of the structure information obtained in this way with the corresponding polygon of the input image 113 and the corresponding input structure information 114, the composite object image 118 is obtained.
Is generated and written in the image memory unit 105. The synthesis target image 118 obtained by the above method is the structure information 11
It is almost the same position in the two-dimensional image space of the object indicated by 6. That is, the composite target object image 11 with another target object at a substantially almost same position in the two-dimensional image space as the hierarchical image 117 of the target object decoded by the hierarchical image decoding unit 102.
8 can be synthesized. Further, in addition to processing so that the average values of the image position coordinates in the structure information match, by processing so that the specific vertex coordinates match, the synthesis target image 118 with the same specific position in the image is synthesized. It is also possible to obtain a combined object image having the same size and the same rotation angle by performing processing such that the relative positional relationship between two specific vertices is the same.

The image memory unit 105 stores a plurality of hierarchical images 117 and the composite object image 1 obtained by the above procedure.
18 is temporarily stored in the hierarchical image synthesizing unit 106.
Reads out the hierarchical image 117 and the composite object image 118 from the image memory unit 105, superimposes them, and outputs them as one composite image. Specifically, as a method of superimposing a plurality of images, the context of each hierarchical image in the moving image,
It is given in advance which hierarchical image includes the image of the target object, and the hierarchical image of the target object is replaced with the composite target object image 118. Then, by reading the hierarchical images on the front side preferentially while simultaneously reading the hierarchical images, a combined moving image can be obtained.

The background hierarchical image decoded by the hierarchical image decoding unit 102 is obtained by adding the background images in the moving image as shown in the example of FIG. It is possible to decode as a hierarchical image even if it is hidden in a certain frame as long as it is included in the moving image. Therefore, even if the background image is hidden in the object in the corresponding one frame in the original moving image and there is no image in the composite object image, the background hierarchical image and the composite object are included. By superimposing the images, it is possible to generate an image having no unnatural portion where the background image is missing.

Further, the method of hierarchically coding and decoding the moving image described above can be coded for each region which can be described by the same motion parameter as described in the research report of Wan et al. It can be transmitted and recorded with the amount of coding.

As described above, according to the present embodiment, as the moving image data of the database section, the encoded moving image data obtained by the method of separating the moving images in the front-back relation to form a hierarchical image and to encode them is recorded. By doing so, there is an effect that a moving image can be recorded with a database unit having a small storage capacity. Furthermore, when replacing the object image in the image with another input object image, the structure information of the object in the moving image and the input image input, and the combined object image synthesized using the input structure information By superimposing the background information included in the decoded hierarchical image, it is possible to synthesize a natural image having no missing background image.

In this embodiment, the structure information time-series data and the input structure information have the two-dimensional position coordinates of the corresponding image, but the structure information may have three-dimensional structure information. 3 at each vertex of the polygon in the structure information
By adding a dimensional coordinate and further adding information about the angle between the arbitrary two vertices with respect to the viewpoint direction of the image, that is, the orientation information in the image of the object, the orientation of the object in the input image is different. Since it is possible to obtain a target composite image of orientations and to combine a moving image close to the motion of the target object in the original moving image, it is possible to obtain a more natural moving image. is there.

Further, the object image synthesizing unit 104 of the present embodiment.
In the above, the input image, the input structure information, and the structure information output from the structure information selection unit are input, and the synthesis target image is to be combined, but the input image, the input structure information, and the structure information output from the structure information selection unit. In addition to the above, the combined object image may be combined using the hierarchical image of the object that is combined by the hierarchical image decoding unit and written in the image memory unit, and the object in the moving image is replaced with the input image. At this time, there is an effect that it is possible to synthesize an image including not only the structural relationship of the target object but also information about the color as the image. For example, if the color of the illumination on the object in the moving image changes over time, combining the combined object images based only on structural relationships will result in an unnatural image in the combined moving image. However, by using the information about the color change of the object in the hierarchical image and changing the color of the combined object image, a more natural moving image can be combined.

[0029]

As described above, according to the present invention, as a method of synthesizing an image in which an object in a moving image is replaced with an image of another input object, the moving image is separated in the front-rear relationship and the hierarchical image is obtained. By recording as encoded moving image data, the moving image can be recorded with a small storage capacity, and the object image in the image is replaced with another input object image. At this time, by using the background information included in the decoded hierarchical image,
It is possible to provide an image synthesizing device that is effective in synthesizing a natural image without a missing background image.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an image synthesizing apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram showing a method of generating moving image data in the embodiment.

3 (a) and 3 (b) are diagrams showing specific examples of structural information in the embodiment.

[Explanation of symbols] 101 Database Department 102 layer image decoding unit 103 Structure Information Selection Section 104 object image composition unit 105 image memory section 106 layer image composition unit 111 encoded video data 112 Structural information time series data 113 Input image 114 Input structure information 115 frame information 116 Structural information 117 Hierarchical image 118 Composite object image 201 moving image 202 Hierarchical image 203 coded moving image data

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-1-274193 (JP, A) JP-A-2-127886 (JP, A) JP-A-4-199474 (JP, A) JP-A-5- 249953 (JP, A) JP-A-6-28449 (JP, A) JP-A-6-350984 (JP, A) JP-A-7-46534 (JP, A) JP-A-8-272998 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 11/80 G06T 3/00 G06T 13/00 H04N 5/272 CSDB (Japan Patent Office)

Claims (4)

(57) [Claims] 1. Structure information time-series data having structure information indicating a correspondence relationship between an object image in each frame in a moving image and a structure of the object and the moving image are separated in a spatial context. a database unit for recording and constitute a hierarchical image coding coded moving picture data, before SL reads the encoded moving image data from the database unit, generates a hierarchical image by respectively decoding the plurality of image stratified a hierarchical image decoding unit you from the read structure information time-series data from said database unit, and the structure information selection unit for selecting and outputting structure information that corresponds, similar to the object in the moving image structure another and the input image of the object the input image to the input structure information and a structure information corresponding to the input from the outside, respectively having the structural information structure information outputted by the selection unit and said input structure information Processing the structure information used, further said input image object image synthesizing unit you synthesize synthetic object image with, superimposed the composite object image as stratifying by the decoding, synthesis An image synthesizing apparatus comprising a hierarchical image synthesizing unit for outputting an image. 2. The structure information includes information about a three-dimensional position of an object, and the object image synthesizing unit receives the three-dimensional information of the structure information and the input structure information output from the structure information selecting unit and the input image. The image synthesizing apparatus according to claim 1, wherein the image of the synthesizing target is synthesized by using. Wherein the object image synthesizing unit, the input image, the input structure information, in addition to the structure information output from the structure information selecting unit, synthesizing the synthesized object image using the hierarchical image of the Target product The image synthesizing device according to claim 1 or 2. 4. An object image in each frame in a moving image is paired with the object image.
Structural information time series data showing the correspondence with the structure of the elephant
And the moving image is separated in a spatial context, and a hierarchical image is obtained.
And the encoded moving image data that is encoded and recorded.
Database part and image memory part that temporarily stores images
Read the encoded moving image data from the database section.
Then, the multiple hierarchized images are decoded and
At the same time when a layer image is generated and written in the image memory unit,
Indicates the frame position for decoding in the encoded moving image data.
And a hierarchical image decoding unit for outputting frame information,
The structural information time-series data is read from the database,
Select and output the structural information corresponding to the frame information.
Structure information selection unit and a structure similar to the object in the moving image.
The input image of another object with a structure and the input image
The input structure information, which is the corresponding structure information, is
Information input from the structure information output section
And process the structural information using the input structural information, and
A composite object image is composed using the input image to create the image.
The object image composition unit to be written in the image memory, and the image memo
Section image and the composite object image written in the
The layered image compositing unit that superimposes
An image synthesizing device characterized by being provided.