WO2025047334A1 - Video processing system, video processing method, and program - Google Patents

Abstract

The present disclosure provides a video processing system, a video processing method, and a program with which it is possible to generate a composite video having a higher degree of completion. At the time of imaging, a first video processing unit generates and displays a simple composite video by performing first video processing for simply synthesizing CG with video and creates a metafile including meta-information used in synthesizing the CG with the video. After imaging, a second video processing unit generates a highly accurate composite video by performing second video processing for synthesizing CG with video at high accuracy using a background in which the simple composite video generated by using the metafile is displayed. The present technology can be applied, for example, to a video processing system that synthesizes two-dimensional or three-dimensional CG with video.

Description

Image processing system, image processing method, and program

This disclosure relates to an image processing system, an image processing method, and a program, and in particular to an image processing system, an image processing method, and a program that are capable of generating a more complete composite image.

Conventionally, for example, technology has been developed that tracks the movement of feature points in an image over time in order to synthesize two-dimensional or three-dimensional CG (Computer Graphics) in accordance with the movement of the image.

For example, Patent Document 1 discloses a synthesis method for synthesizing 3D CG in real time by identifying the positions of feature points on images from pre-captured video data through inter-frame tracking.

JP 2008-15576 A

However, conventionally, when shooting, users would have to check the image that had not been composited with CG while shooting, and would have to imagine what the composite image would look like with CG composited in. As a result, the composite image that has been composited with CG in the later image processing stage may have been shot with a camerawork that differs from the user's imagination, reducing the quality of the composite image.

This disclosure was made in light of these circumstances, and makes it possible to generate more complete composite images.

The image processing system of one aspect of the present disclosure includes a first image processing unit that performs a first image processing for simply synthesizing CG with the image during shooting to generate and display a simplified composite image and creates a metafile including meta information used to synthesize the CG with the image, and a second image processing unit that performs a second image processing for synthesizing CG with high precision with the image after shooting in the background where the simplified composite image generated using the metafile is displayed, to generate a highly accurate composite image.

An image processing method or program according to one aspect of the present disclosure includes performing a first image processing step during shooting to simply combine CG with the image to generate and display a simplified composite image, creating a metafile containing meta information used to combine the CG with the image, and performing a second image processing step after shooting to highly accurately combine CG with the image in the background while the simplified composite image generated using the metafile is displayed, to generate a highly accurate composite image.

In one aspect of the present disclosure, when shooting, a first image processing is performed to simply composite CG onto the image, a simplified composite image is generated and displayed, a metafile is created containing meta information used to composite the CG onto the image, and after shooting, a highly accurate composite image is generated by performing a second image processing to highly accurately composite CG onto the image in the background while the simplified composite image generated using the metafile is displayed.

FIG. 1 is a diagram illustrating a usage example of a video processing system to which the present technology is applied. 1 is a block diagram showing an example of the configuration of an embodiment of a video processing system; 11 is a flowchart illustrating image processing for generating a simplified composite image during shooting. 10 is a flowchart illustrating image processing for generating a highly accurate composite image during main line image processing. FIG. 1 is a diagram illustrating an example of a use of 2DCG in a video processing system. FIG. 13 is a diagram showing an example of a UI image for instructing execution of video processing during shooting. 13 is a diagram showing an example of a UI image for instructing execution of video processing during main line video processing; FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied.

Below, specific embodiments of the application of this technology will be described in detail with reference to the drawings.

<Examples of using the video processing system>
An example of use of a video processing system to which the present technology is applied will be described with reference to FIG.

The upper part of Figure 1 shows an example of shooting using the image processing system 11, where the camera 21 shoots an image of the subject 12 while transmitting the image acquired by the shooting to the smartphone 22. Various communication standards such as HDMI (High-Definition Multimedia Interface) (registered trademark), SDI (Serial Digital Interface), USB (Universal Serial Bus) (registered trademark), and wireless LAN (Local Area Network) can be used to transmit the image from the camera 21 to the smartphone 22. The smartphone 22 then performs image processing (tracking, compositing, and rendering) to simply composite a 3DCG model onto the subject 12 shown in the image captured by the camera 21, and displays the resulting simplified composite image.

Therefore, by using the video processing system 11, the user can check in real time on the smartphone 22 a simply composite image in which a 3DCG model is simply composited onto the subject 12 during shooting. This allows the user to shoot with camerawork that allows for framing that takes into account the composition of the 3DCG model onto the subject 12. Furthermore, when reviewing the shot video, the user can check the simply composite image in which a 3DCG model is simply composited onto the subject 12, so that the user can consider at that point whether or not it is necessary to re-shoot the subject 12.

In the image processing system 11, after the camera 21 has finished capturing the image of the subject 12, the image processing can be performed using the personal computer 23. In other words, the image processing during capture only results in a simple synthesis of the 3DCG model, and it is necessary to synthesize the 3DCG model with higher accuracy.

The lower part of Figure 1 shows an example of main line video processing using the video processing system 11, where the personal computer 23 acquires a main line video file from the camera 21 via the recording medium 24 and acquires a 3DCG metafile by communicating with the smartphone 22. The main line video file is a file that contains video that has not been composited with 3DCG, and the 3DCG metafile is a file that contains various meta information used to composite a 3DCG model onto the subject 12 through simple video processing performed by the smartphone 22. The personal computer 23 then plays back the simple composite video using the 3DCG metafile, while in the background performing video processing to composite a 3DCG model onto the subject 12 with high precision, generating a higher quality composite video.

In this way, by using the video processing system 11, the user can have the personal computer 23 generate a more complete composite video immediately after the camera 21 finishes capturing images during main line video processing. This allows the user to more quickly begin editing and releasing the video.

Here, in FIG. 1, the image processing for superimposing a 3DCG model on the subject 12 has been described, but the present technology is not limited to image processing for superimposing a 3DCG model on the subject 12. For example, the present technology can also be applied to image processing for superimposing a 3DCG model on a space other than the subject in the image. That is, image processing can be performed to superimpose a 3DCG model of a refrigerator, bed, etc. on the space in an image of a room containing only a desk and chairs. Furthermore, when performing such image processing, the present technology tracks the camera 21, and the position and orientation of the 3DCG model can be superimposed so that it moves in accordance with the movement of the camera 21 and matches the space in the image.

As shown in FIG. 1, when the image processing system 11 is used with the smartphone 22 fixed to the camera 21, the smartphone 22 may perform simple image processing using the output of the six-axis sensor provided in the camera 21, or may perform simple image processing using the output of the six-axis sensor provided in the smartphone 22. On the other hand, when the image processing system 11 is used without fixing the smartphone 22 to the camera 21, it becomes necessary to supply the output of the six-axis sensor provided in the camera 21 to the smartphone 22, and the smartphone 22 performs simple image processing using the output of the six-axis sensor of the camera 21.

<Example of video processing system configuration>
FIG. 2 is a block diagram showing an example of the configuration of an embodiment of a video processing system to which the present technology is applied.

As shown in FIG. 2, the video processing system 11 includes a camera 21, a smartphone 22, and a personal computer 23, and is configured to import a 3DCG model from a model storage unit 25 to the smartphone 22 and the personal computer 23.

The camera 21 is configured with a posture recognition unit 31, a photographing unit 32, a time code generation unit 33, a data transmission unit 34, a display unit 35, and a recording media drive 36.

The attitude recognition unit 31 recognizes the attitude of the camera 21 and sequentially supplies attitude information indicating the attitude to the data transmission unit 34. For example, the attitude recognition unit 31 can recognize the attitude of the camera 21 based on six-axis data indicating six-axis acceleration or angular velocity detected by a six-axis sensor equipped in the camera 21.

The image capturing unit 32 captures an image of the subject 12 using an image capturing element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and supplies the image captured by the capture to the data transmission unit 34, the display unit 35, and the recording media drive 36.

The time code generation unit 33 generates a time code according to the frame rate of the video captured by the shooting unit 32 and supplies it to the data transmission unit 34 and the recording media drive 36.

The data transmission unit 34 adds a time code supplied from the time code generation unit 33 to each frame of the video supplied from the image capture unit 32. Furthermore, the data transmission unit 34 adds posture information obtained at the timing corresponding to each frame from the posture information of the camera 21 sequentially supplied from the posture recognition unit 31 to each frame of the video supplied from the image capture unit 32. The data transmission unit 34 then sequentially transmits the video to which the time code and posture information of the camera 21 have been added to the smartphone 22.

The display unit 35 displays a live view image, which is a real-time image being captured by the image capture unit 32.

The recording media drive 36 adds a time code supplied from the time code generation unit 33 to each frame of the video captured by the imaging unit 32, and records the main line video file containing that video on the recording media 24 (Figure 1). The main line video file can also contain a proxy file that reduces the resolution of the video captured by the imaging unit 32, as necessary. Note that the main line video file may contain orientation information of the camera 21. For example, if the 3DCG metafile described below contains orientation information of the camera 21 or smartphone 22, the main line video file does not need to contain orientation information of the camera 21.

The smartphone 22 is configured with a posture recognition unit 41, a simple tracking unit 42, a simple synthesis processing unit 43, a simple rendering unit 44, a display unit 45, and a metafile creation unit 46.

The attitude recognition unit 41 recognizes the attitude of the smartphone 22, and sequentially supplies attitude information indicating the attitude to the simplified tracking unit 42 and the metafile creation unit 46. For example, the attitude recognition unit 41 can recognize the attitude of the smartphone 22 based on six-axis data indicating six-axis acceleration or angular velocity detected by a six-axis sensor provided in the smartphone 22.

The simplified tracking unit 42 simply tracks the self-position of the camera 21 based on the posture information provided by the posture recognition unit 41 and feature points extracted from the video (the subject 12, the scenery other than the subject 12, and everything in the video). For example, when the smartphone 22 is fixed to the camera 21 and the video processing system 11 is used (see FIG. 1), the simplified tracking unit 42 can use the posture information of the smartphone 22 provided by the posture recognition unit 41 and the posture information of the camera 21 added to the video transmitted from the camera 21. On the other hand, when the video processing system 11 is used without fixing the smartphone 22 to the camera 21, the simplified tracking unit 42 can only use the posture information of the camera 21 added to the video transmitted from the camera 21. The simplified tracking unit 42 then provides tracking information (e.g., information indicating the simplified position, orientation, posture, etc. of the camera 21) indicating the result of the simplified tracking of the self-position of the camera 21 to the simplified synthesis processing unit 43.

The simple synthesis processing unit 43 performs synthesis processing to simply synthesize the 3DCG model imported from the model storage unit 25 onto the video transmitted from the camera 21 based on the tracking information supplied from the simple tracking unit 42. For example, the simple synthesis processing unit 43 determines the simplified position, orientation, posture, etc. of the 3DCG model to be synthesized onto the video so as to correspond to the scenery in the video and the position, orientation, posture, etc. of the subject 12. The simple synthesis processing unit 43 then generates simple 3DCG model synthesis information indicating the simplified position, orientation, posture, etc. of the 3DCG model and supplies it to the simple rendering unit 44.

The simple rendering unit 44 positions the 3DCG model based on the simple 3DCG model synthesis information supplied from the simple synthesis processing unit 43, sets a virtual camera corresponding to the camera 21 based on the tracking information, and uses the virtual camera to simply render the 3DCG model. The simple rendering unit 44 then generates a simple synthesis image by superimposing the image of the 3DCG model obtained by rendering on each frame of the image transmitted from the camera 21, and supplies this to the display unit 45.

The display unit 45 displays the simplified composite image supplied from the simplified rendering unit 44, i.e., an image obtained by simply combining the image captured by the camera 21 with an image of a 3DCG model rendered by a virtual camera corresponding to the camera 21.

The metafile creation unit 46 creates a 3DCG metafile that includes various meta information used to synthesize a 3DCG model into an image through simple image processing performed on the smartphone 22, and transmits the created 3DCG metafile to the personal computer 23. For example, the 3DCG metafile includes at least simple 3DCG model synthesis information (XYZ coordinates indicating simple position information of the 3DCG model) with a time code added for each frame, and posture information (XYZ coordinates) with a time code added. The 3DCG metafile can also include the model name of the 3DCG model, simple position information (XYZ coordinates) of the 3DCG model at the time of shooting start point, zoom/focus/iris (plus gain/white balance) information, etc.

The personal computer 23 is configured with a recording media drive 51, a synchronization processing unit 52, a tracking unit 53, a synthesis processing unit 54, a rendering unit 55, and a display unit 56.

When the recording media 24 (FIG. 1) on which the main line video file is recorded is set by the recording media drive 36 of the camera 21, the recording media drive 51 reads the main line video file from the recording media 24 and supplies it to the synchronization processing unit 52.

The synchronization processing unit 52 performs a synchronization process to synchronize the 3DCG metafile transmitted from the smartphone 22 with the video contained in the main line video file supplied from the recording media drive 51 according to their respective time codes. The synchronization processing unit 52 then supplies the synchronized main line video file to the tracking unit 53 and the rendering unit 55, and supplies the synchronized 3DCG metafile to the composition processing unit 54 and the rendering unit 55.

The tracking unit 53 tracks the self-position of the camera 21 with high accuracy based on the posture information contained in the 3DCG metafile supplied from the synchronization processing unit 52 and feature points extracted from the image contained in the main line image file supplied from the synchronization processing unit 52. The tracking unit 53 then supplies the synthesis processing unit 54 with tracking information indicating the results of tracking the self-position of the camera 21 with high accuracy (for example, information indicating the highly accurate position, orientation, posture, etc. of the camera 21).

The compositing processing unit 54 performs compositing processing to highly accurately combine the 3DCG model imported from the model storage unit 25 with the video contained in the main line video file supplied from the synchronization processing unit 52 based on the tracking information supplied from the tracking unit 53. For example, the compositing processing unit 54 determines the highly accurate position, orientation, posture, etc. of the 3DCG model to be composited with the video so as to correspond to the scenery in the video and the position, orientation, posture, etc. of the subject 12. The compositing processing unit 54 then generates 3DCG model composition information indicating the highly accurate position, orientation, posture, etc. of the 3DCG model and supplies it to the rendering unit 55.

The rendering unit 55 uses the simple 3DCG model synthesis information contained in the 3DCG metafile supplied from the synchronization processing unit 52 to generate a simple synthetic image in the same manner as the simple rendering unit 44, and supplies it to the display unit 56 for display. In this way, while the simple synthetic image generated in the rendering unit 55 is displayed in the background on the display unit 56, tracking is performed by the tracking unit 53, and synthesis processing is performed by the synthesis processing unit 54.

Furthermore, in the background where the simplified composite image is displayed on the display unit 56, the rendering unit 55 positions the 3DCG model based on the 3DCG model composition information supplied from the composition processing unit 54, sets a virtual camera corresponding to the camera 21 based on the tracking information, and uses the virtual camera to render the 3DCG model with high precision. The rendering unit 55 then generates a highly accurate composite image by superimposing the image of the 3DCG model obtained by rendering on each frame of the image contained in the main line video file supplied from the synchronization processing unit 52, and supplies this to the display unit 56.

The display unit 56 displays the simplified composite image supplied from the rendering unit 55, and then when the simplified composite image is supplied with a high-precision composite image from the display unit 56, the display unit 56 displays the high-precision composite image.

The video processing system 11 is configured as described above, and when shooting, the user can check the simple composite image displayed on the smartphone 22 and shoot with camerawork that results in more appropriate framing. Furthermore, when processing main line video, the user can edit the video while realistically imagining the image of the composite image by checking the simple composite image displayed on the personal computer 23. Then, in the background while the simple composite image is being displayed on the personal computer 23, a higher quality composite image can be generated. This allows the video processing system 11 to generate a more complete composite image.

<Example of video processing>
3 and 4, the image processing executed in the image processing system 11 will be described. Here, an example of processing will be described in which the orientation information of the camera 21 is used, not the orientation information of the smartphone 22.

FIG. 3 is a flowchart explaining the image processing for generating a simple composite image when shooting. For example, the processing starts when the user performs an operation for performing image processing in cooperation with the camera 21 and the smartphone 22.

In step S11, the imaging unit 32 of the camera 21 starts imaging the subject 12, the posture recognition unit 31 starts recognizing the posture of the camera 21, and the time code generation unit 33 starts generating a time code. The data transmission unit 34 then sequentially transmits the video with the time code and posture information of the camera 21 added to it to the smartphone 22.

In step S12, the simplified tracking unit 42 tracks the self-position of the camera 21 based on the attitude information of the camera 21 added to the image transmitted in step S11 and feature points extracted from the image transmitted in step S11. The simplified tracking unit 42 then supplies tracking information indicating the results of the simplified tracking of the self-position of the camera 21 to the simplified synthesis processing unit 43.

In step S13, the simple synthesis processing unit 43 performs synthesis processing to simply synthesize a 3DCG model onto the video transmitted in step S11 based on the tracking information supplied in step S12, and generates simple 3DCG model synthesis information and supplies it to the simple rendering unit 44.

In step S14, the simple rendering unit 44 positions the 3DCG model based on the simple 3DCG model synthesis information supplied in step S13, sets a virtual camera corresponding to the camera 21 based on the tracking information supplied in step S12, and uses the virtual camera to simply render the 3DCG model. The simple rendering unit 44 then superimposes the image of the 3DCG model obtained by rendering onto the image transmitted from the camera 21 to generate a simple synthesized image and supplies it to the display unit 45.

In step S15, the display unit 45 displays the simplified composite image provided in step S14.

In step S16, the metafile creation unit 46 creates a 3DCG metafile that includes at least the posture information of the camera 21 that was added to the video transmitted in step S11 and the simple 3DCG model synthesis information generated in step S13, for example, and to which a time code has been added.

After step S16, the process returns to step S11, and the same process is repeated until, for example, the user performs an operation to end the video processing.

FIG. 4 is a flowchart explaining the image processing for generating a highly accurate composite image during main line image processing. For example, the process is started when the user inserts the recording medium 24 on which the main line image file is recorded into the personal computer 23, connects the smartphone 22 and the personal computer 23 for communication, and then performs an operation to start image processing.

In step S21, the metafile creation unit 46 in the smartphone 22 transmits the 3DCG metafile created in step S16 of FIG. 3 to the personal computer 23.

In step S22, the recording media drive 51 reads the main line video file from the recording media 24 and supplies it to the synchronization processing unit 52.

In step S23, the synchronization processing unit 52 performs a synchronization process to synchronize the 3DCG metafile transmitted in step S21 with the video contained in the main line video file supplied in step S22 according to their respective time codes. The synchronization processing unit 52 then supplies the synchronized main line video file to the tracking unit 53 and the rendering unit 55, and supplies the synchronized 3DCG metafile to the composition processing unit 54 and the rendering unit 55.

In step S24, the rendering unit 55 uses the simple 3DCG model synthesis information contained in the 3DCG metafile supplied in step S23 to generate a simple synthesis image in the same manner as the simple rendering unit 44, and supplies it to the display unit 56 for display.

Meanwhile, steps S25 to S27 are carried out in parallel with step S24.

In step S25, the tracking unit 53 tracks the self-position of the camera 21 with high accuracy based on the posture information included in the 3DCG metafile supplied in step S23 and feature points extracted from the image included in the main line image file supplied in step S23. The tracking unit 53 then supplies the synthesis processing unit 54 with tracking information indicating the results of tracking the self-position of the camera 21 with high accuracy.

In step S26, the synthesis processing unit 54 performs synthesis processing to synthesize a 3DCG model with high accuracy onto the image contained in the main line image file supplied in step S23 based on the tracking information supplied in step S25, and generates 3DCG model synthesis information and supplies it to the rendering unit 55.

In step S27, the rendering unit 55 positions the 3DCG model based on the 3DCG model synthesis information supplied in step S26, sets a virtual camera corresponding to camera 21 based on the tracking information supplied in step S25, and uses the virtual camera to render the 3DCG model with high precision. The rendering unit 55 then superimposes the image of the 3DCG model obtained by rendering onto each frame of the video included in the main line video file supplied from the synchronization processing unit 52 to generate a highly accurate synthesized video, and supplies it to the display unit 56.

In step S28, the display unit 56 displays the high-precision composite image provided in step S27, replacing the simplified composite image displayed in step S24.

After step S28, the process returns to step S21, and the same process is repeated until, for example, the user performs an operation to end the video processing.

By performing the above-described image processing, the image processing system 11 can generate a more complete composite image by displaying a simplified composite image during shooting and generating a composite image in the background that displays the simplified composite image during main line image processing.

<Examples of using the video processing system>
An example of using a 2D model in a video processing system will be described with reference to FIG.

As described above with reference to FIG. 1, the image processing system 11 can synthesize a 3DCG model for a subject shown in an image when shooting and processing main line images. Furthermore, the image processing system 11 can synthesize a 2D model for a subject in the same way as a 3DCG model.

As shown in the upper part of Figure 5, camera 21 photographs subject 13 while transmitting the image acquired by the photograph to smartphone 22, which then performs image processing to simply combine a 2D model with subject 13 shown in the image, and displays the resulting simplified composite image. Note that in the example shown in Figure 5, a mosaic image is shown as the 2D model, but other images such as stamps can also be used as the 2D model.

Then, as shown in the lower part of FIG. 5, the personal computer 23 acquires the main line video file from the camera 21 via the recording medium 24, and acquires a 2D metafile by communicating with the smartphone 22. The 2D metafile is a file that contains various meta information used to superimpose a 2D model onto the subject 12 through simple video processing performed by the smartphone 22. The personal computer 23 then plays back the simple composite video using the 2D metafile, while in the background performing video processing to superimpose the 2D model onto the subject 12 with high precision, generating a higher quality composite video. Of course, in addition to superimposing a 2D model onto the subject 13, a 2D model may also be superimposed into a space other than the subject 13 in the video.

In this way, by using the video processing system 11, it is possible to generate a more complete composite image on the personal computer 23, similar to the example of using the 3DCG model described above with reference to FIG. 1.

For example, the 2D metafile contains at least simple 2D model synthesis information (XYZ coordinates indicating simple position information of the 2D model) with a time code added for each frame. The 2D metafile can also contain the model name of the 2D model, zoom/focus/iris (+gain/white balance) information, etc. It is not necessary to change the orientation of the 2D model in accordance with the movement of the camera 21.

<UI image display example>
A UI (User Interface) image for instructing the execution of video processing using the video processing system 11 will be described with reference to FIGS. 6 and 7. FIG.

Figure 6 shows an example of a UI image that instructs the execution of video processing during shooting.

A in FIG. 6 shows a UI image displayed on the display unit 35 of the camera 21, displaying a "Yes" GUI (Graphical User Interface) button and a "No" GUI button along with the message "Do you want to link posture information and images to perform simple composition?". B in FIG. 6 shows a UI image displayed on the display unit 45 of the smartphone 22, displaying a "Yes" GUI button and a "No" GUI button along with the message "Do you want to receive posture information, images, and 2D/3DCG models to perform simple composition?".

For example, when a user causes a UI image as shown in the figure to be displayed on display unit 35 and display unit 45, and performs a user operation on each of the "Yes" GUI buttons, transmission of posture information and video from camera 21 to smartphone 22 begins. Then, smartphone 22 begins receiving posture information and video, and imports the 2D/3DCG model from model storage unit 25. Note that the import of the 2D/3DCG model may be performed in advance.

FIG. 7 shows an example of a UI image that instructs the execution of video processing during main line video processing.

7A shows a UI image displayed on the display unit 35 of the camera 21, displaying a "Yes" GUI button and a "No" GUI button along with the message "Do you want to send video to edit the main line video?". 7B shows a UI image displayed on the display unit 45 of the smartphone 22, displaying a "Yes" GUI button and a "No" GUI button along with the message "Do you want to send a 2D/3DCG metafile to edit the main line video?". 7C shows a UI image displayed on the display unit 56 of the personal computer 23, displaying a "Yes" GUI button and a "No" GUI button along with the message "Do you want to receive a 2D/3DCG metafile, video, and 2D/3DCG model to edit the main line video?".

For example, when a user causes a UI image as shown in the figure to be displayed on display unit 35, display unit 45, and display unit 56, and performs a user operation on each of the "Yes" GUI buttons, transmission of video from camera 21 to personal computer 23 begins, and transmission of a 2D/3DCG metafile from smartphone 22 to personal computer 23 begins. Then, personal computer 23 begins receiving the video and 2D/3DCG metafile, and imports the 2D/3DCG model from model storage unit 25. Note that the import of the 2D/3DCG model may be performed in advance.

In the video processing system 11, such a UI image can be used to instruct the execution of video processing.

In this embodiment, CG synthesis processing refers to synthesizing CG with the image captured by the camera 21 in a later editing stage. The CG also needs to move in accordance with the movement of the camera 21. Tracking refers to tracking feature points in the image captured by the camera 21, such as objects in the image, to perform 2D tracking or 3D camera tracking. 2D tracking refers to tracking objects in the image using the feature points of the image captured by the camera 21 and an acceleration sensor, and is used, for example, when it is desired to have a mosaic always follow a person in the image. 3D camera tracking refers to analyzing how the camera 21 was moved to capture the image based on an acceleration sensor and feature points, and generating a virtual camera. Generally, 3D camera tracking links the CG and the virtual camera, so that the synthesized CG moves in the same way in accordance with the movement of the image, allowing for seamless synthesis.

The gyro provided in the camera 21 may be a 6-axis or 8-axis sensor, or may be another sensor. Also, instead of the smartphone 22, a tablet may be used, or an external gyro sensor may be used in combination with an external monitor. Furthermore, if the camera 21 has a function for performing 3D tracking and synthesis processing, there is no need to attach an external smartphone 22, and the above-mentioned processing of the smartphone 22 can be performed by the camera 21. Also, instead of generating a simple composite image at the same time as shooting with the camera 21, a simple composite image may be generated using images already shot by the camera 21.

Furthermore, if the smartphone 22 has the same processing capability as the personal computer 23, the main line image processing may be performed by the smartphone 22, in which case there is no need to transmit the main line image file and the 3DCG metafile to the personal computer 23. Furthermore, if the camera 21 has the same processing capability as the smartphone 22 and the personal computer 23, all image processing may be performed inside the camera 21, in which case there is no need to transmit the image from the camera 21 to the outside.

<Example of computer configuration>
Next, the above-mentioned series of processes (video processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

FIG. 8 is a block diagram showing an example of the configuration of one embodiment of a computer on which a program that executes the series of processes described above is installed.

The program can be pre-recorded on the hard disk 105 or ROM 103 as a recording medium built into the computer.

Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as so-called packaged software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory, etc.

In addition to being installed in a computer from the removable recording medium 111 as described above, the program can also be downloaded to the computer via a communication network or broadcasting network and installed in the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer via a wired connection via a network such as a LAN (Local Area Network) or the Internet.

The computer has a built-in CPU (Central Processing Unit) 102, to which an input/output interface 110 is connected via a bus 101.

When a command is input by the user via the input/output interface 110, such as by operating the input unit 107, the CPU 102 executes a program stored in the ROM (Read Only Memory) 103 accordingly. Alternatively, the CPU 102 loads a program stored on the hard disk 105 into the RAM (Random Access Memory) 104 and executes it.

As a result, the CPU 102 performs processing according to the above-mentioned flowchart, or processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 102 outputs the processing results from the output unit 106 via the input/output interface 110, or transmits them from the communication unit 108, or even records them on the hard disk 105, as necessary.

The input unit 107 is composed of a keyboard, mouse, microphone, etc. The output unit 106 is composed of an LCD (Liquid Crystal Display), speaker, etc.

In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may also be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

Also, for example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

Furthermore, for example, the above-mentioned program can be executed in any device. In that case, it is sufficient that the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

Furthermore, for example, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

In addition, the processing of the steps that describe a program executed by a computer may be executed chronologically in the order described in this specification, or may be executed in parallel, or individually at the required timing, such as when a call is made. In other words, as long as no contradictions arise, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps that describe this program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

Note that the multiple present technologies described in this specification can be implemented independently and individually, provided no contradictions arise. Of course, any multiple present technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Also, part or all of any of the present technologies described above can be implemented in combination with other technologies not described above.

<Examples of configuration combinations>
The present technology can also be configured as follows.
(1)
a first image processing unit that performs a first image processing for simply synthesizing CG (Computer Graphics) with an image during shooting to generate and display a simplified synthesized image, and creates a metafile including meta information used to synthesize the CG with the image;
and a second image processing unit that generates a highly accurate composite image by performing a second image processing for synthesizing CG with the image with high precision after shooting while the simple composite image generated using the metafile is displayed in the background.
(2)
Further equipped with a camera for taking pictures,
The image processing system according to (1) above, wherein the first image processing unit and the second image processing unit are operated based on posture information indicating a posture of the camera.
(3)
The first image processing unit includes:
a simplified tracking unit that simply tracks the camera based on the shooting direction of the camera identified based on the posture information and feature points extracted from the video;
a simplified synthesis processing unit that generates simplified synthesis information required for synthesizing the CG image on the video based on a result of the tracking;
The video processing system described in (2) above has a simple rendering unit that positions the CG based on the simple synthesis information, sets a virtual camera corresponding to the camera identified based on the posture information, and uses the virtual camera to simply render the CG to generate the simple composite image.
(4)
the camera has a time code generation unit that generates a time code that identifies a frame of the video captured by the camera;
The second video processing unit has a synchronization processing unit that synchronizes the metafile and the video according to the time code added to the metafile and the time code added to each frame of the video.The video processing system described in (3) above.
(5)
The second image processing unit includes:
a tracking unit that tracks the camera with high accuracy based on the shooting direction of the camera specified based on the posture information and feature points extracted from the video;
a synthesis processing unit that generates synthesis information required for synthesizing the CG image on the video based on a result of the tracking;
and a rendering unit that positions the CG based on the synthesis information, sets a virtual camera corresponding to the camera identified based on the posture information, and uses the virtual camera to render the CG with high accuracy to generate the synthetic image.
(6)
The video processing system described in (5) above, in which tracking is performed by the tracking unit, the synthesis information is generated by the synthesis processing unit, and the synthetic video is generated by the rendering unit in the background while the rendering unit generates and displays the simple synthetic video using the simple synthesis information contained in the metafile.
(7)
The video processing system
During shooting, a first image processing is performed to simply combine CG (Computer Graphics) with the image to generate and display a simplified composite image, and a metafile is created that includes meta information used to combine the CG with the image;
After shooting, in the background where the simple composite image generated using the metafile is displayed, a second image processing is performed to composite CG with the image with high accuracy, thereby generating a highly accurate composite image.
(8)
The computer in the video processing system
During shooting, a first image processing is performed to simply combine CG (Computer Graphics) with the image to generate and display a simplified composite image, and a metafile is created that includes meta information used to combine the CG with the image;
After shooting, in the background where the simple composite image generated using the metafile is displayed, a second image processing is performed to synthesize CG with high precision onto the image, thereby generating a highly accurate composite image.

Note that this embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of this disclosure. Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

11 Image processing system, 12 and 13 Subject, 21 Camera, 22 Smartphone, 23 Personal computer, 24 Recording media, 25 Model memory unit, 31 Posture recognition unit, 32 Shooting unit, 33 Time code generation unit, 34 Data transmission unit, 35 Display unit, 36 Recording media drive, 41 Posture recognition unit, 42 Simple tracking unit, 43 Simple synthesis processing unit, 44 Simple rendering unit, 45 Display unit, 46 Metafile creation unit, 51 Recording media drive, 52 Synchronization processing unit, 53 Tracking unit, 54 Synthesis processing unit, 55 Rendering unit, 56 Display unit

Claims (8)

a first image processing unit that performs a first image processing for simply synthesizing CG (Computer Graphics) with an image during shooting to generate and display a simplified synthesized image, and creates a metafile including meta information used to synthesize the CG with the image;
and a second image processing unit that generates a highly accurate composite image by performing a second image processing for synthesizing CG with the image with high precision after shooting while the simple composite image generated using the metafile is displayed in the background. Further equipped with a camera for taking pictures,
The image processing system according to claim 1 , wherein the first image processing section and the second image processing section are operated based on attitude information indicating an attitude of the camera. The first image processing unit includes:
a simplified tracking unit that simply tracks the camera based on the shooting direction of the camera identified based on the posture information and feature points extracted from the video;
a simplified synthesis processing unit that generates simplified synthesis information required for synthesizing the CG image on the video based on a result of the tracking;
and a simple rendering unit that positions the CG based on the simple synthesis information, sets a virtual camera corresponding to the camera identified based on the posture information, and uses the virtual camera to simply render the CG to generate the simple synthetic image. the camera has a time code generation unit that generates a time code that identifies a frame of the video captured by the camera;
The video processing system of claim 3 , wherein the second video processing unit has a synchronization processing unit that synchronizes the metafile and the video in accordance with the time code added to the metafile and the time code added to each frame of the video. The second image processing unit includes:
a tracking unit that tracks the camera with high accuracy based on the shooting direction of the camera specified based on the posture information and feature points extracted from the video;
a synthesis processing unit that generates synthesis information required for synthesizing the CG image on the video based on a result of the tracking;
a rendering unit that positions the CG based on the synthesis information, sets a virtual camera corresponding to the camera identified based on the posture information, and uses the virtual camera to render the CG with high accuracy to generate the synthetic image. The video processing system according to claim 5, wherein tracking by the tracking unit, generation of the synthesis information by the synthesis processing unit, and generation of the synthetic video by the rendering unit are performed in the background while the rendering unit generates and displays the simple synthetic video using the simple synthesis information contained in the metafile. The video processing system
During shooting, a first image processing is performed to simply combine CG (Computer Graphics) with the image to generate and display a simplified composite image, and a metafile is created that includes meta information used to combine the CG with the image;
After shooting, in the background where the simple composite image generated using the metafile is displayed, a second image processing is performed to composite CG with the image with high accuracy, thereby generating a highly accurate composite image. The computer in the video processing system
During shooting, a first image processing is performed to simply combine CG (Computer Graphics) with the image to generate and display a simplified composite image, and a metafile is created that includes meta information used to combine the CG with the image;
After shooting, in the background where the simplified composite image generated using the metafile is displayed, a second image processing is performed to synthesize CG with high precision onto the image, thereby generating a highly accurate composite image.

Images