KR20250000353A - 5G-based metaverse motion synchronization system and metaverse motion synchronization method using the same - Google Patents

Abstract

The present invention relates to a 5G-based metaverse motion synchronization system (1) and a metaverse motion synchronization method using the same.
According to a preferred embodiment of the present invention, a metaverse motion synchronization method is provided in a metaverse motion synchronization system (1) including a motion capture server (200) and a digital human server (100) connected to a 5G network, the method comprising: a) a step in which the motion capture server (200) collects digital human motion information including at least one of face motion information representing facial movement and body motion information representing body and hand movement as movement information of a user model through a motion capture device; b) a step in which the digital human server (100) configures a virtual model based on a preset virtual model configuration method; c) a step in which the digital human server (100) receives digital human motion information from the motion capture server (200); and d) a step in which the digital human server (100) matches the digital human motion information with the virtual model based on a preset motion synchronization method to generate digital human information.

Description

5G-based metaverse motion synchronization system and metaverse motion synchronization method using the same {5G-based metaverse motion synchronization system and metaverse motion synchronization method using the same}

The present invention relates to a 5G-based metaverse motion synchronization system and a metaverse motion synchronization method using the same, and more particularly, to a 5G-based metaverse motion synchronization system and a metaverse motion synchronization method using the same, which enables natural digital human information generation through rapid and accurate synchronization by using a 5G network-based preset matching mechanism in the process of generating digital human information by matching motion information with a virtual model configured based on a preset virtual model configuration method.

With the development of industry and the spread of infectious diseases, the demand for online and non-face-to-face environments is increasing, and expectations for so-called metaverse content that combines virtual reality and real life are spreading.

These metaverse contents have a great advantage in that they can provide a lot of information in a short period of time, transcending time and space, by providing various contents in a non-face-to-face environment.

Metaverse content based on conventional technology mainly targets specific content and only delivers one-sided content to users, so technology that creates digital humans based on virtual reality but allows for diverse user participation and utilizes them is attracting attention. In particular, research is being conducted on how to make digital human technology that replaces the role of humans perform forms and movements similar to those of actual humans.

As an example of prior art, there is Korean Patent No. 10-2373608 entitled “Electronic device and method for forming a digital human image, and a program stored in a computer-readable recording medium to perform the same.”

The above conventional technology enables the formation of a more sophisticated and realistic digital human image by generating a digital human image by considering the intimacy and age of the subject, facial movement, wind, and external force.

The above conventional technology has the advantage of being able to form a sophisticated and realistic digital human image by considering the real-time facial shape and real-time posture of the subject, but there is a problem in that the digital human image is expressed awkwardly in the metaverse environment because motion synchronization is not properly achieved when high-speed and stable data transmission is not achieved.

In particular, the above-mentioned conventional technology only assigns weights to feature points that take into account real-time facial shape and posture, and does not consider any difference in the collection and transmission time of each motion information (data) when facial or body movements occur simultaneously, so there is a problem in that the realism of the actual implemented object is low.

The present invention is intended to solve the problems of the above-mentioned prior art, and to provide a method and system for synchronizing each motion with optimal system resources when the movements (motions) of digital humans for implementing natural metaverse content are performed simultaneously.

In addition, the present invention seeks to provide a synchronization method that can actively cope with changes in a network environment.

According to a preferred embodiment of the present invention for solving the above-mentioned problem, a metaverse motion synchronization method is provided in a metaverse motion synchronization system including a motion capture server and a digital human server connected to a 5G network, the method comprising: a) a step in which the motion capture server collects digital human motion information including at least one of face motion information representing facial movement and body motion information representing body and hand movement as movement information of a user model through a motion capture device; b) a step in which the digital human server configures a virtual model based on a preset virtual model configuration method; c) a step in which the digital human server receives digital human motion information from the motion capture server; and d) a step in which the digital human server matches the digital human motion information with the virtual model based on a preset motion synchronization method to generate digital human information.

According to another preferred embodiment of the present invention, the preset motion synchronization method is characterized in that synchronization is achieved based on a difference between a first time when the motion capture server collects digital human motion information from the motion capture device and a second time when the digital human server receives the digital human motion information.

According to another preferred embodiment of the present invention, the digital human motion information includes at least one reference motion information, and the preset motion synchronization method is characterized in that synchronization is automatically performed every time the reference motion information is transmitted to either the motion capture server or the digital human server.

According to another preferred embodiment of the present invention, the preset motion synchronization method is characterized in that synchronization is achieved based on face motion information collected from the motion capture device and body motion information collected from the motion capture device in response to the face motion information.

According to another preferred embodiment of the present invention, the preset motion synchronization method is characterized in that synchronization is achieved based on body motion information including hand motion information collected from the motion capture device and body motion information including body motion information collected from the motion capture device in response to the hand motion information.

According to another preferred embodiment of the present invention, the digital human motion information further includes voice information of a user model collected from the motion capture server, and the preset motion synchronization method is characterized in that synchronization is achieved based on the voice information and face motion information collected from the motion capture device in response to the voice information.

According to another preferred embodiment of the present invention, the motion capture server is configured to be connected to at least one of a Wi-Fi and a 5G network and to collect digital human motion information from the motion capture device, and in the preset motion synchronization method, a preset weight is applied according to the type of network connected to the motion capture server to perform pre-motion synchronization.

According to another preferred embodiment of the present invention, a metaverse motion synchronization system is provided, which includes a motion capture server and a digital human server connected to a 5G network, the motion capture server being connected to a motion capture device and a network to collect digital human motion information including at least one of face motion information representing facial movement as movement information of a user model and body motion information representing movement of a body and hands; and a digital human server performing a function of configuring a virtual model based on a preset virtual model configuring method, a function of receiving digital human motion information from the motion capture server, and a function of generating digital human information by matching the digital human motion information with the virtual model based on a preset motion synchronization method.

According to the present invention, there is an effect of implementing realistic metaverse content by displaying realistic and natural digital humans by using a 5G-based wireless network.

In particular, the present invention has an advantage in that it can minimize the phenomenon in which the motion of a digital human is awkwardly implemented due to differences in data transmission time depending on the type of motion and the type of wireless network.

Figure 1 is a configuration diagram of a 5G-based metaverse motion synchronization system according to one embodiment of the present invention.
Figure 2 is a configuration diagram of a digital human server according to one embodiment of the present invention.
Figure 3 is a flow chart of a 5G-based metaverse motion synchronization method according to an embodiment of the present invention.
Figure 4 is a schematic diagram showing the configuration of a virtual model creation studio and a network connection with a digital human server according to one embodiment of the present invention.
FIG. 5 is a schematic diagram showing the configuration of a motion capture device and network connection with a motion capture server according to one embodiment of the present invention.
Figure 6 is a schematic diagram for explaining metaverse motion synchronization according to one embodiment of the present invention.
Figures 7 and 8 are flowcharts of a 5G-based metaverse motion synchronization method according to another embodiment of the present invention.
FIG. 9 is a flow chart of the creation of digital human content generated according to a 5G-based metaverse motion synchronization method according to another embodiment of the present invention.
Figure 10 is a configuration diagram of a 5G-based metaverse motion synchronization system according to another embodiment of the present invention.
Figure 11 is a schematic diagram of digital human content streaming in a 5G-based metaverse motion synchronization system according to another embodiment of the present invention.
FIG. 12 is a flow chart of digital human content streaming performed in a 5G-based metaverse motion synchronization system according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, when describing the present invention, if the technical idea of the present invention is obscured or unclear due to a detailed description of the known configuration, the description of the known configuration will be omitted.

The 'user model' according to the present invention is a real model that is a target for collecting user image information, motion information and/or voice information included in digital human information that serve as the basis for constructing a virtual model. The user image information, motion information and voice information may be provided by one user model in two or more pieces of information, and different user models may provide each piece of information. For example, one user model may provide user image information that serves as the basis for constructing a virtual model, and another user model may provide motion information and voice information.

Below, a 5G-based metaverse motion synchronization system and a metaverse motion synchronization method using the same are described in detail based on drawings.

FIG. 1 is a schematic diagram showing the configuration of a 5G-based metaverse motion synchronization system according to an embodiment of the present invention, FIG. 2 is a schematic diagram showing the configuration of a digital human server according to an embodiment of the present invention, FIG. 4 is a schematic diagram showing the configuration of a virtual model creation studio according to an embodiment of the present invention and a network connection with a digital human server, and FIG. 5 is a schematic diagram showing the configuration of a motion capture device according to an embodiment of the present invention and a network connection with a motion capture server.

A metaverse motion synchronization system (1) according to an example of the present invention includes a digital human server (100) and a motion capture server (200) connected to a 5G network and the digital human server (100), as shown in FIG. 1.

The above digital human server (100) configures a virtual model based on a preset virtual model configuration method, receives digital human motion information from a motion capture server (200), and generates digital human information by matching the digital human motion information with the virtual model based on a preset motion synchronization method.

A digital human server (100) according to an example of the present invention, as illustrated in FIG. 2, includes a memory unit (104) for storing user image information collected in a virtual model creation studio (120) described below, motion information collected through a motion capture device in a motion information collection studio (240), as well as data (information) temporarily generated during the process of configuring a virtual model and generating digital human information, a communication unit (105) configured to include a communication module, a virtual model configuration/management unit (106) for configuring and managing a virtual model, a digital human information synchronization/management unit (107) for generating, synchronizing, and managing digital human information, and an operation unit (102) for performing overall operation of the server.

The above virtual model creation studio (120) may include, as illustrated in FIG. 4, a full-body capture device (122), and the full-body capture device (122) may include at least a plurality of cameras (124) and lights (126), and may include a communication module (129, 129') that performs a function of controlling the plurality of cameras (124) and lights (126), obtaining user image information, and transmitting the information to the digital human server (100) via a network. The communication module (149, 149') may be a 5G modem, a 5G wireless dongle, a Wifi AP, etc.

For example, the full body capture device (122) may be configured to capture 360-degree full body images of a user model by evenly disposing 16 to 64 volumetric cameras or 32 to 128 photogrammetric cameras on the outside of a steel frame (not shown in the drawing) in which a user model can be accommodated. For example, the volumetric camera may be a camera capable of generating depth values, such as an RGB-D camera, an RGB camera combined with a Kinect sensor, etc.

According to an example of the present invention, a motion capture server (200) can collect motion information of a user model (UM2) through a motion capture device (242, 242', 244', 246') in the motion information collection studio (240).

The above motion information collection studio (240), as illustrated in FIG. 5, includes a motion capture device (242, 242', 244', 246'), and may include a communication module (249, 249') that performs a function of controlling the motion capture device (242, 242', 244', 246') and obtaining at least one of face motion information (UF) representing the movement of the face of the user model (UM1) and body motion information (UBB, UBH, UB) representing the movement of the hand or the body excluding the hand of the user model (UM1) and transmitting the obtained information to the motion capture server (200) via a network. The communication module (249, 249') may be a 5G modem, a 5G wireless dongle, a Wifi AP, or the like, and may also be formed by combining a Wifi module and a 5G module. For example, body motion information (UBB, UBH, UB) may be received by a Wifi AP or Wifi Gateway and transmitted to a capture server (200) through a 5G network via a 5G wireless dongle.

For example, the motion capture device may be configured in the form of a camera (142) to capture the entire body of a user model (UM1) and recognize and store facial and body movements separately. In this case, the camera (142) may be a CCTV or a digital camera, or may be a camera equipped on a smartphone (depth camera, wide-angle/telephoto camera, etc.).

As another example, the motion capture device may be configured in the form of a facial motion capture (detection) device (242') that collects movement of the user model's face and/or voice information of the user model described below, a sensor (144') that is attached to the body of the user model in multiple units to collect movement information of the body of the user model, a glove (146'), etc. The sensor may be, for example, a 9 DoF IMU sensor having a 3-axis accelerometer, a 3-axis magnetometer, and a 3-axis gyroscope.

For example, the motion capture device that collects the above body motion information (UBB) may be configured as a smart suit in which 16 or more magnetic sensors are provided at locations corresponding to the joints of the user model, and each sensor is configured to capture the body motion of the user model.

For example, a motion capture device that collects body motion information (UBH) according to the movement of the hand may be configured as a smart glove in which 24 or more magnetic sensors are provided at positions corresponding to the finger joints of the user model, and each sensor is configured to capture body motion representing the movement of the user model's hand.

The motion capture device, such as the above camera (142) or facial motion capture device (142'), captures the gaze processing, facial expression, etc. of the user model's (UM2) face and identifies characteristic points according to changes in the gaze or facial expression. The motion capture server (200) can collect facial motion information (UF) of the user model through changes in these characteristic points.

The motion capture device, such as the above camera (142) or facial motion capture device (142'), is equipped with a microphone and recording function and can collect voice information (UV, see FIG. 6) of the user model (UM2) and transmit the collected voice information (UV) to the motion capture server (200).

The voice information (UV) of the above user model may include the user model's voice color, tone, or speech pattern information, and may also be composed of specific words to form reference motion information for metaverse motion synchronization.

The above motion capture server (200) is connected to a motion capture device and a network to collect digital human motion information including facial motion information (UF) representing facial movement as movement information of a user model, body motion information (UB, UBH, UBB) representing movement of the body and hands, and voice information (UV), and transmit the collected information to the digital human server (100) through a network.

The above network can be configured as a 5G-based network, and when configured as a 5G-based network compared to existing low-speed networks, there is a high possibility of data loss or time delay in the process of efficiently transmitting not only the virtual reality content information but also the digital human information data. Therefore, it is desirable to configure a 5G-based network in that it allows faster and larger data transmission.

However, the above motion capture server (200) and motion capture device may be configured to be additionally connected to a separate wireless network such as Wi-Fi or Bluetooth to supplement spatiotemporal constraints and 5G networks.

Metaverse motion synchronization according to one embodiment of the present invention is intended to implement natural virtual reality in a situation where time differences may occur in the transmitted motion information due to differences in data transmission capacity and transmission time of a wireless network and changes in the network usage environment (e.g., disaster, speed delay due to user concentration, etc.). Therefore, when matching virtual models and digital human motion information, motion synchronization that takes time differences into account is performed so that digital human information is generated.

The above motion capture server (200) can measure/store the collection time at which the face motion information (UF), body motion information (UB, UBH, UBB) and voice information (UV) are collected through a motion capture device, and when voice and face motion, face motion and body motion, and body motion and voice are performed simultaneously, the time at which each piece of information is captured through a motion capture device can also be stored.

The above digital human server (100) and motion capture server (200) can each measure the reception and transmission times of digital human motion information, and metaverse motion synchronization can be achieved based on the difference between the reception and transmission times of the digital human motion information.

FIG. 3 is a flowchart of a 5G-based metaverse motion synchronization method according to an embodiment of the present invention, FIG. 6 is a schematic diagram for explaining metaverse motion synchronization according to an embodiment of the present invention, and FIGS. 7 and 8 are flowcharts of a 5G-based metaverse motion synchronization method according to another embodiment of the present invention.

A metaverse motion synchronization method according to a preferred embodiment of the present invention may include a process in which a motion capture server (200) collects digital human motion information (step a), a digital human server (100) configures a virtual model (step b), the motion capture server (200) transmits the digital human motion information to the digital human server (100) (step c), and the digital human server (100) matches the virtual model and the digital human motion information to generate digital human information (step d).

As described above, as illustrated in FIG. 3, in step a, the motion capture server (200) can collect digital human motion information including at least one of face motion information (UF) representing facial movement as movement information of a user model and body motion information (UB, UBH, UBB) representing movement of the body and hands, through a motion capture device.

The digital human motion information of the user model (UM2) collected through a motion capture device in the above motion information collection studio (240) is transmitted to the motion capture server (200) through a wired or wireless network.

Preferably, the digital human motion information can be transmitted to the motion capture server (200) via a 5G network, and in case the 5G network is temporarily not operating due to internal or external environmental problems, it can be transmitted via a separately connected wireless network such as Wi-Fi. At this time, the motion capture server (200) can obtain information about changes in the network and the type of network transmitting the digital human motion information.

The above motion capture device can capture movement information (expression and/or movement) of a user model (UM2), and the digital human motion information can include at least one of face motion information (UF) indicating movement of a face and body motion information indicating movement of a body and hands, and can further include voice information (UV) of the above-described user model (UM2).

The above facial motion information (UF) enables the user model's (UM2) emotions or psychological state to be conveyed through the user model's (UM2) gaze or facial expression, and the body motion information enables the user model's (UM2) movements to be conveyed.

In step b, the digital human server (100) can configure a virtual model based on a preset virtual model configuration method.

The above preset virtual model configuration method can collect user image information by photographing a user model (UM1) through a full body capture device equipped in a virtual model creation studio (120) connected to the digital human server (100) through a network, generate virtual model basic modeling information based on the user image information, and configure a virtual model by post-processing the generated virtual model basic modeling information.

Generating virtual model basic modeling information based on the above user image information is to collect user images of the 360-degree full body of the user model (UM1), and generate virtual model basic modeling information by performing an automated modeling process while combining each image. For example, the virtual model basic modeling information can be understood as a 360-degree external shape of the user model (UM1) without a sense of three-dimensionality.

The above virtual model basic modeling information can be post-processed to form a three-dimensional virtual model with a three-dimensional effect by texturing, which gives a three-dimensional effect by adding color and texture to the virtual model basic modeling information, and rigging, which allows the virtual model to move by combining it with digital human motion information by giving joints and bones to the virtual model basic modeling information.

The virtual model configuration of step b above may be performed simultaneously with the collection of digital human motion information of step a or prior to step a, and steps a and b do not imply an order in generating digital human information.

The above motion capture server (200) can be connected to a motion capture device through a network and transmit the collected digital human motion information to the digital human server (100) through the network. After the digital human server (100) receives the digital human motion information from the motion capture server (200) through step c, in step d, the digital human server (100) can match the digital human motion information with the virtual model based on a preset motion synchronization method to generate digital human information.

Metaverse motion synchronization according to an example of the present invention must consider the time difference resulting from the process of information transmission through the aforementioned network, and in addition, in the process of collecting digital human motion information, creating a virtual model, and combining the virtual model and the digital human motion information to create digital human information, post-processing such as data transmission, texturing, and rigging is required. Therefore, in cases where multiple motions occur simultaneously (e.g., face motion information and body motion information, face motion information and voice information, etc.), it is desirable to synchronize them to ensure accurate matching.

For example, the above preset motion synchronization method enables synchronization to be achieved based on the difference between a first time when the motion capture server (200) collects digital human motion information from the motion capture device and a second time when the digital human server (100) receives the digital human motion information.

When generating digital human information, if a 5G network is used, the transmission time is very short even when transmitting a large amount of data, so digital human motion information can be quickly matched with the virtual model.

However, if the number of images or sensors of the user model is large, the data size of the digital human motion information constituting the digital human information is very large, or if there is a problem with the 5G network and smooth data transmission is not possible, or if the speed of the wireless network such as Wifi that can be used instead of the 5G network is slow or the data processing amount or transmission capacity per hour is very low, the transmission of the digital human motion information may be delayed, resulting in the composition of digital human information with low realism when matching the virtual model and the digital human motion information.

Accordingly, by measuring the difference between the time (first time) at which the digital human motion information is collected from the motion capture server (200) and the time (second time) at which the digital human motion information is collected from the digital human server (100), and taking this time difference into consideration when matching with the virtual model, natural digital human information that minimizes interruption can be generated.

The above digital human server (100) can receive digital human motion information from the motion capture server (200) along with collection and/or transmission time information of the digital human motion information collected and/or transmitted by the motion capture server (200), and can accurately achieve metaverse motion synchronization by reflecting the time difference between one motion (action) and the next motion or at the start of a new motion to match it with the virtual model.

As another example, in a preset motion synchronization method, the first time may be set to a time (moment) at which the motion capture device collects digital human motion information or a time at which the motion capture server (200) collects digital human motion information from the motion capture device and transmits it to the digital human server (100), and the second time may be set to a time (moment) at which the digital human server (100) receives the digital human motion information and matches it with the generated virtual model.

The above preset motion synchronization method can be configured to be automatically performed depending on whether a certain amount of time has passed since the previous motion synchronization time, whether the reference motion information is included in the digital human motion information, and whether a forced motion synchronization signal has been received by the digital human server (100).

For example, the preset motion synchronization method can be configured to measure the time difference between the first time and the second time at regular time intervals (e.g., 1 hour) and reflect the time difference in motion synchronization.

As another example, the above-described preset motion synchronization method can be configured to measure a time difference between the first time and the second time when a specific motion is included in the digital human motion information, and to reflect the time difference in the motion synchronization. To this end, a specific motion among the motion information of the user model (UM2) collected from the motion capture device can be set as reference motion information, and the digital human motion information can include at least one or more reference motion information.

For example, the above specific motion (reference motion information) can be configured as a greeting motion, and if the digital human motion information includes a greeting motion, motion synchronization can be performed automatically.

The above-mentioned reference motion information is intended to automatically perform motion synchronization, and may be set to automatically perform synchronization whenever the above-mentioned reference motion information is transmitted to either the motion capture server (200) or the digital human server (100). That is, when the user model (UM2) performs (operates) a specific motion (e.g., greeting) that is preset, if the motion is transmitted to the motion capture server (200) and/or the digital human server (100), the metaverse motion synchronization according to an example of the present invention may be automatically performed.

As another example, as illustrated in FIG. 7, when a digital human server (100) receives digital human motion information from a motion capture server (200), it first determines whether reference motion information is included, and if reference motion information is included, motion synchronization is performed automatically, and if not, it determines whether a certain amount of time (e.g., 1 hour) has passed since the previous motion synchronization time.

If a certain amount of time has passed since the previous motion synchronization time, motion synchronization is performed automatically. If not, it determines whether a forced motion synchronization signal has been received.

The above-mentioned forced motion synchronization signal is intended to automatically perform motion synchronization when the signal is transmitted in a situation where motion synchronization must be performed forcibly, thereby allowing digital human information to be naturally configured. For example, when the status of the network transmitting digital human motion information changes, the forced motion synchronization signal may be configured to be automatically transmitted from the motion capture server (200) to the digital human server (100). The digital human server (100) may be configured to automatically perform motion synchronization by reflecting the time difference between the first time and the second time when the forced motion synchronization signal is received, and to not automatically perform motion synchronization otherwise.

According to another preferred embodiment of the present invention, the preset motion synchronization method may be set to perform synchronization based on body motion information (UBH) including hand motion information collected from the motion capture device, as illustrated in FIG. 6, and body motion information (UBB) including body motion information collected from the motion capture device in response to the hand motion information.

For example, the above preset motion synchronization method may be set to synchronize by reflecting the time difference between the time (third time) at which body motion information (UBH) including hand motion information collected from the motion capture device is received by the digital human server (100) and the time (fourth time) at which body motion information (UBB) including body motion information collected from the motion capture device in response to the hand motion information is received by the digital human server (100).

When synchronization is achieved by reflecting the time difference between the third and fourth times, body motion information (UBH) including the motion information of the hand and body motion information (UBB) including the motion information of the body (excluding the hand) corresponding to the motion information of the hand can be achieved simultaneously, so that natural digital human information can be generated (and displayed).

As another example, the third time period may be set as a time when body motion information (UBH) including hand motion information is combined with a virtual model to generate digital human information, and the fourth time period may be set as a time when body motion information (UBB) (excluding the hand) collected in response to the hand motion information is combined with a virtual model to generate digital human information. Table 1 shows the generation time (third time period) of digital human information generated based on body motion information (UBH) including hand motion information and the generation time (fourth time period) of digital human information generated based on body motion information (UBB) excluding hand motion information according to the present embodiment, and the difference therebetween.

In this case, when matching the body motion information (UBH) including the motion information of the hand with the virtual model by delaying by the average time difference of 0.798 seconds, metaverse motion synchronization can be achieved in which the body motion information (UBH) including the motion information of the hand and the body motion information excluding the hand (UBB) collected in response to it generate natural digital human information.

Digital human information generation time based on hand and body motion information excluding hands Number of Tests 3rd hour 4th hour Time difference (seconds) 1 02:50.03 02:51.38 1.35 2 03:25.33 03:25.78 0.45 3 03:50.32 03:51.65 1.33 4 04:28.74 04:29.14 0.40 5 05:12.25 05:12.71 0.46 Average Time Difference (sec) 0.798

According to another preferred embodiment of the present invention, when the digital human motion information further includes voice information (UV) of a user model collected from the motion capture server (200), the preset motion synchronization method may be set to synchronize by reflecting a time difference between a time (fifth time) at which the voice information (UV) is received and a time (sixth time) at which the face motion information (UF) collected from the motion capture device corresponding to the voice information (UV) is received.

When synchronization is achieved by reflecting the time difference between the above-mentioned 5th and 6th hours, the voice information (UV) and the facial motion information (UF) corresponding to the voice information (UV) can be achieved simultaneously, so that natural digital human information can be generated (and displayed).

As another example, the fifth time period may be set as a time at which voice information (UV) is combined with a virtual model to generate digital human information and is displayed (e.g., voice output), and the sixth time period may be set as a time at which face motion information (UF) collected in response to the voice information (UV) is combined with a virtual model to generate digital human information and is displayed (e.g., digital human information display). Table 2 shows the display time (sixth time period) of digital human information generated based on face motion information (UF) according to the present embodiment and the output time (fifth time period) of voice information (UV) generated in response to the face motion information (UF) and the difference therebetween.

In this case, when matching the facial motion information (UF) corresponding to the voice information (UV) with the virtual model by delaying it by the average time difference of 0.766 seconds, metaverse motion synchronization can be achieved in which the facial motion information (UF) and the voice information (UV) collected in response to it generate natural digital human information.

Digital human information display time according to voice information and facial motion information Number of Tests 6th hour 5th hour Time difference (seconds) 1 04.28 05.13 0.85 2 04.93 05.79 0.86 3 12.02 12.54 0.52 4 08.13 08.94 0.81 5 10.04 10.83 0.79 Average Time Difference (sec) 0.766

According to another preferred embodiment of the present invention, the preset motion synchronization method may be set to be synchronized based on face motion information (UF) collected from the motion capture device and body motion information (UB, UBH, UBB) collected from the motion capture device in response to the face motion information (UF).

For example, the above preset motion synchronization method can be set to synchronize by reflecting the time difference between the time at which the face motion information (UF) collected from the motion capture device is received and the time at which the body motion information (UB, UBH, UBB) collected from the motion capture device in response to the face motion information (UF) is received, and since this synchronization can be performed similarly to the synchronization of the body motion information (UBH) including the hand and the body motion information (UBB) excluding the hand, a detailed description thereof is omitted.

According to another preferred embodiment of the present invention, as illustrated in FIG. 8, the motion capture server (200) is configured to collect digital human motion information from the motion capture device by being connected to at least one of a Wi-Fi and a 5G network, and in the preset motion synchronization method, a preset weight may be applied according to the type of network connected to the motion capture server (200) so that pre-motion synchronization may be performed. At this time, if the type of network does not change or the speed of the network does not change, the pre-motion synchronization may be set not to be performed, or if the speed of the network changes beyond a preset range even if the type of the network does not change, the pre-motion synchronization may be set to be performed.

The above weights can be understood as being applied in advance to minimize time differences, as there are differences in the speed of transmitting data such as digital human motion information and the amount of data transmitted per hour depending on the type of network.

For example, the weight can be set to 1 for a 5G network, 0.5 to 0.8 for a Wi-Fi network, and 0.2 to 0.3 for a Bluetooth network.

For example, if the motion capture server (200) collects digital human motion information including facial motion information (UF) and voice information (UV) through a 5G network and a problem occurs in the 5G network and is changed to a Wi-Fi network, the speed and the amount of data that can be transmitted per hour decrease compared to the 5G network, so that the time difference according to data transmission becomes larger. Therefore, by applying a weight of 0.5 to 0.8 depending on the speed of the Wi-Fi network and the amount of data that can be transmitted per hour, the difference in transmission time between facial motion information (UF) and voice information (UV) can be minimized, thereby enabling rapid and accurate metaverse motion synchronization.

FIG. 9 is a flowchart regarding the creation of digital human content generated according to a 5G-based metaverse motion synchronization method according to another embodiment of the present invention, FIG. 10 is a configuration diagram of a 5G-based metaverse motion synchronization system according to another embodiment of the present invention, FIG. 11 is a schematic diagram regarding digital human content streaming performed in a 5G-based metaverse motion synchronization system according to another embodiment of the present invention, and FIG. 12 is a flowchart regarding digital human content streaming performed in a 5G-based metaverse motion synchronization system according to another embodiment of the present invention.

A metaverse motion synchronization system (1) according to another embodiment of the present invention can generate digital human content with motion synchronization that combines the digital human information and virtual reality content information based on the aforementioned motion synchronization, thereby allowing natural combination of the digital human information and the virtual reality content information.

The above digital human content creation is accomplished by including a digital human server (100) that creates, synchronizes, and manages digital human information by matching a virtual model with digital human motion information, a motion capture server (200) that collects motion information of the digital human, a content management server (300) that manages virtual reality content, and a metaverse cloud server (400) that creates digital human content by combining the digital human information and virtual reality content information.

As illustrated in FIG. 10, a 5G-based metaverse motion synchronization system (1) according to another embodiment of the present invention may be configured to stream digital human content generated in the metaverse cloud server (400), and to this end, the 5G-based metaverse motion synchronization system according to the present embodiment may further include a metaverse streaming server (500) and a user terminal (600) through which the digital human content is streamed.

The above content management server (300) manages a plurality of virtual reality content information, transmits at least one virtual reality content information to the metaverse cloud server (400) upon request of the metaverse cloud server (400), and can manage user models, digital human information, and virtual reality content information.

As another example, the content management server (300) may be configured to relay the metaverse cloud server (400) and the metaverse streaming server (500) to transmit digital human content from the metaverse cloud server (400) to the metaverse streaming server (500), in which case, video content including digital human content generated in the metaverse cloud server (400) can be managed.

The above-mentioned plurality of virtual reality content information may be stored in advance in a storage medium (not shown) provided in the content management server (300), but may preferably be stored in a separate virtual reality content information DB (not shown) connected to the content management server (300) through a network, and may be configured in the form of streamable video content and transmitted from various types of external streaming servers such as YouTube.

For example, the above virtual reality content information relates to video content that allows for direct or indirect experience of virtual reality in various fields including educational content, exhibition content, and performance content, and can be created and stored in various content servers.

For example, the metaverse cloud server (400) can create digital human content by combining virtual reality content information received from the content management server (300) and digital human information generated by combining user motion information, and transmit the digital human content to the metaverse streaming server (500).

The above metaverse streaming server (500) can perform a function of streaming the digital human content by providing it to each user terminal (600), and can further perform a function of streaming the digital human content for each user by providing it to each user terminal (600) by each preset user group.

In addition to the real-time streaming service of the digital human content, the above metaverse streaming server (500) can also possess various forms of video content and provide video content including digital human content for each user to the content management server (300).

The above metaverse streaming server (500) can be configured to differentiate the range of digital human content streamed for each user group as well as the above digital human content, and details will be described later.

The above metaverse streaming server (500) can be configured to include various streaming protocols to correspond to various types of user terminals (600) for each user.

The above digital human content streaming module may include one or more of streaming protocols such as RTSP (Real Time Streaming Protocol), RTMP (Real Time Messaging Protocol), or adaptive HTTP streaming protocols such as HLS (Http Live Streaming), and may additionally configure a progressive download function.

The above user terminal (600) can be understood as a display device capable of displaying digital human content, and can be a portable terminal equipped with a display device such as a smart phone or tablet, or a computer, or a VR (Virtual Reality) device capable of displaying virtual reality content such as an HMD (Head Mount Display). It is preferable to configure the user terminal (600) to be movable so that various digital human content can be utilized anytime and anywhere, and for this purpose, it is preferable to configure a network wirelessly.

As illustrated in FIG. 9, a motion synchronization method performed in a 5G-based metaverse motion synchronization system (1) according to another embodiment of the present invention includes a step (a) in which a motion capture server (200) collects digital human motion information, a step (b) in which a digital human server (100) configures a virtual model based on a preset virtual model configuration method, a step (c) in which the digital human server (100) receives digital human motion information from the motion capture server (200), and a step (d) in which the digital human server (100) matches the digital human motion information with the virtual model to generate digital human information.

Next, the digital human server (100) may include a step (e) of transmitting digital human information to the metaverse cloud server, and a step (f) of the metaverse cloud server combining virtual reality content information received from the content management server and the digital human information to generate digital human content, and the metaverse cloud server may transmit the digital human content to the metaverse streaming server (g), and real-time streaming of the digital human content may be displayed through a network on a user terminal (600).

In the above step a (virtual model configuration step) and step b (digital human motion information reception step), multiple virtual models can be created, and multiple digital human motion information matching each virtual model can be received. To this end, as illustrated in FIG. 10, the digital human server (100) can configure virtual models through multiple virtual model creation studios (120, 120'), and the motion capture server (200) can collect motion information for each user through multiple motion information collection studios (240, 240').

In the above step f (step of generating digital human content), when the digital human information generated in the above step d (step of generating digital human information) is combined with virtual reality content information received from a content management server, synchronization of the digital human information and virtual reality content information can be achieved similarly to the metaverse motion synchronization method according to an example of the present invention.

In other words, methods such as motion synchronization settings based on the difference in the reception time of the aforementioned data, whether or not reference motion information is included, automatic motion synchronization based on motion synchronization time (cycle) or forced motion synchronization signal reception, and pre-motion synchronization reflecting preset weights based on changes in the network environment can be used in the process of creating digital human content, and more realistic real-time digital human content streaming can be achieved.

For example, the metaverse cloud server (400) can obtain delay time information on virtual reality content information received from the content management server (300) through the difference in time between the time the content management server (300) transmits the virtual reality content information and the time the metaverse cloud server (400) receives the virtual reality content information.

Next, delay time information on the digital human information can be obtained through the difference in time between the time the digital human information is transmitted from the digital human server (100) and the time it is received from the metaverse cloud server (400).

The above metaverse cloud server (400) can generate digital human content in which motion synchronization is achieved when the virtual reality content information and the digital human information are combined by reflecting the delay time information for the virtual reality content information and the delay time information for the digital human information.

The metaverse streaming server (500) receives a plurality of user-specific digital human contents in which a plurality of user-specific digital human motion information is reflected from the metaverse cloud server (400) and transmits them to each user terminal (600), and when the metaverse streaming server (500) transmits the same to each user terminal (600), the metaverse streaming server can perform motion synchronization for each user so that natural movements (motions) can be made between the multiple user-specific digital human contents during the process of displaying the multiple user-specific digital human contents on one metaverse space. The motion synchronization for each user in the streaming of the multiple user-specific digital human contents can be performed based on the difference between the time when each user-specific digital human content is transmitted from the metaverse cloud server (400) and the time when the metaverse streaming server (500) receives it.

Preferably, the 5G-based metaverse motion synchronization system (1) according to each embodiment of the present invention is configured such that each server (100, 200, 300, 400, 500) and user terminal (600) are interconnected through a 5G network, and a separate wired/wireless network such as Wi-Fi, Bluetooth, or wired/wireless Lan can be selectively connected to respond according to a communication situation.

As illustrated in FIG. 12, a metaverse synchronization system (1) according to another example of the present invention may be configured to collect digital human motion information (user motion information) of each user from the user terminal (600) and directly transmit the same to the digital human server (100), or transmit the digital human motion information (user motion information) of each user collected from the user terminal (600) to the digital human server (100) via the motion capture server (200).

For example, the user terminal (600) may be equipped with a motion capture device capable of collecting each user's motion information. The motion capture device equipped in the user terminal (600) may be a camera and/or various sensors (gravity sensor, acceleration sensor, gyro sensor, etc.).

The above user terminal (600) can provide digital human motion information of users to each preset user group through the motion capture device, and can stream digital human content with motion synchronization for each user through a display means.

In this case, the metaverse cloud server (400) can create user-specific digital human content by combining the virtual reality content information received from the content management server (300) and the user-specific digital human information generated by combining the user-specific motion information received from the user terminal (600), and transmit the user-specific digital human content to the metaverse streaming server (500).

At this time, the metaverse cloud server (400) can generate natural user-specific digital human content through motion synchronization in the process of combining the virtual reality content information and user-specific digital human information.

The above user-specific digital human content can be understood as a combination of user-specific digital human information and virtual reality content information.

For example, one virtual reality content information may be received in one situation (e.g., model selection contest, concert, exhibition, education, etc.), and digital human information for each user may be arranged in the same virtual reality content information with only different locations and/or times. As another example, each virtual reality content information may be separately configured to be customized for each user's digital human information and combined with each other, so that unique digital human content may be created for each user.

The above preset user group may include a first user group that provides first user motion information that matches the virtual model to generate user-specific digital human information, and a second user group that provides second user motion information that enables changes to the first user motion information provided in the first user group.

For example, the digital human server (100) can generate digital human information for each user by matching the second user motion information with the virtual model within a preset range.

For example, the second user motion information is movement information of a second user included in a second user group collected through a motion capture device, and may include at least one of face motion information indicating facial movement and body motion information indicating body and hand movement.

The above user group can be understood as being divided into grades or stages in order to provide diversity in directly or indirectly experiencing various types and/or forms of content within virtual reality.

Virtual reality content information may be model selection contests, education, exhibitions, performances, etc. For example, in the case of a performance, performers performing in person at a performance venue and audience members watching the performance at the venue are required, and in the case of a model selection contest, multiple models performing walks and various performances (movements) directly, as well as an emcee and/or audience members are required.

At this time, the first user group and the second user group can be understood as performers (or models) and spectators (or hosts), respectively, and since the actions performed at the performance venue (e.g., performance-spectator) and the range of permitted movements (e.g., on-stage-off-stage) are different, it is desirable to also limit (distinguish) the motion information that constitutes digital human information for generating digital human content by combining with virtual reality content information.

For example, since the user(s) corresponding to the first user group are people who perform actual performances, all of the user's motion information (first user motion information) can be provided as is and matched with the aforementioned virtual model to generate user-specific digital human information.

To this end, the motion information collection studio (240) may be configured to allow users of the first user group to visit in person to collect digital human motion information for each user, and to transmit the collected digital human motion information for each user of the first user group to the digital human server (100) via the motion capture server (200).

For example, the second user motion information may be configured to include only body motion information for the upper body among the body movements.

That is, since the user(s) corresponding to the second user group are people watching a performance or competition, it is necessary to prevent them from moving from under the stage to on the stage or to restrict them from making sounds exceeding a certain volume, and only some of the motion information of the user (the second user motion information) can be provided to match it with the virtual model described above to generate digital human information for each user.

For example, the second user motion information provided by a user corresponding to the second user group may be an action of clapping or waving a placard to cheer on a performer (e.g., a user of the first user group). In this case, in the case of a live performance where streaming of digital human contents for each user is performed in real time, such an action may be made visible to the users of the first user group, thereby enabling a change in the first user motion information provided by the first user group. In addition, it is desirable to enable natural movements through motion synchronization so as to induce various forms of interaction between the performer and the audience.

For example, the user terminal (600) of the second user group may include a VR device or a two-dimensional or three-dimensional display means, and as illustrated in FIG. 11, the appearance (digital human information, U1) and the performance stage (virtual reality content information, MVSC, Metaverse Virtual Space Contents) of the performer (user of the first user group) may be configured to be streamed as user-specific digital human content (MDHC, Metaverse Digital Human Contents).

The user-specific digital human content (MDHC) provided to the user terminal (600) of the second user group may be the appearance (digital human information, U2) of another user of the second user group who is sitting in the audience rather than on the performance stage.

For example, the user terminal (600) of the first user group may include a VR device or a two-dimensional or three-dimensional display means, and may be configured to stream the performance stage or the appearance of the audience (user of the second user group) as viewed from the performer's point of view as user-specific digital human content (MDHC). In addition, the user terminal (600) of the first user group may be configured to select and stream image/video information viewed from each position of the virtual camera so that the performer's appearance can be confirmed according to the first user's selection.

For example, the user-specific digital human content includes digital human information viewed from two or more viewpoints, and at least one user terminal (600) among the user terminals (600) of the first user group and the second user group can be configured to selectively stream digital human information (U1, U1-1, U1-2) viewed from the two or more viewpoints.

The digital human information viewed from the above two or more viewpoints may be image/video information viewed from at least one virtual camera (VCAM1, VCAM2) configured within the virtual reality content.

For example, as shown in FIGS. 4 and 5, multiple virtual cameras (VCAM1, VCAM2) can be configured and configured to stream digital human information viewed from various viewpoints from user terminals (600) of each user group.

For example, the user-specific digital human information (U1, U1-1, U1-2) formed by first user motion information provided by users of a first user group may be configured to be captured from the viewpoint of a plurality of virtual cameras (VCAM1) and the corresponding images may be selectively streamed from user terminals (600) of the first user group, and the user-specific digital human information (U2, U2-1, U2-2, U2-3) formed by second user motion information provided by users of a second user group may be configured to be captured from the viewpoint of a plurality of virtual cameras (VCAM2) and the corresponding images may be selectively streamed from user terminals (600) of the first user group and/or the second user group.

According to another embodiment of the present invention, the preset user group may further include a third user group that does not provide user motion information.

For example, as illustrated in FIG. 11, the user terminal (600) of the third user group may be configured to simply stream digital human content generated based on user motion information provided by the first user group and the second user group.

The above third user group may be understood as general viewers, different from the first and second user groups corresponding to the performers and spectators described above, respectively, and may be users who can stream live performances or recorded videos of performances.

For example, users of the third user group can be restricted from selectively streaming digital human content including digital human information of various viewpoints captured by the aforementioned virtual cameras (VCAM1, VCAM2).

In this case, just as general viewers have no choice but to watch the performance video broadcast by the broadcasting station as is, it is desirable for the user terminal (600) of the third user group to stream the video (digital human content) planned in advance by the planner who composes the digital human content in order to differentiate it from the users (e.g., viewers) of the second user group. Through this differentiation of streaming scope (authority) according to participation, viewing, and viewing by user group, active performance viewing can be induced rather than simple viewing.

The metaverse motion synchronization system (1) according to the present embodiment can be configured to perform synchronization when generating digital human content for each preset user group and when streaming the generated digital human content, and the metaverse cloud server (400) and the metaverse streaming server (500) can be configured to store data transmission/reception times for motion synchronization, and perform motion synchronization according to the time difference.

The 5G-based metaverse motion synchronization method according to each embodiment of the present invention is implemented in the form of a program command that can be executed on various computers and recorded on a computer-readable recording medium.

The above recording medium may include program commands, data files, data structures, etc., alone or in combination, and may be, for example, a magnetic recording medium such as a hard disk or a floppy disk, an optical recording medium such as a CD or DVD, a magneto-optical medium such as a floptical disk, a ROM, a RAM, a flash memory, etc.

The program commands recorded on the above recording medium may be specially designed or configured for the present invention, or may be known and available. For example, the program commands may include machine language code or high-level language code.

Although the present invention has been described in detail above with reference to specific embodiments, the embodiments are merely examples to facilitate understanding of the present invention. Therefore, embodiments that are substituted, added, and modified within the scope that does not depart from the technical spirit of the present invention are also included in the scope of protection of the present invention defined by the following claims.

1: Metaverse motion synchronization system 100: Digital human server
120: Virtual Model Creation Studio 200: Motion Capture Server
240: Motion information collection studio 300: Content management server
400: Metaverse Cloud Server 500: Metaverse Streaming Server
600 : User terminal

Claims (8)

A metaverse motion synchronization method performed in a metaverse motion synchronization system (1) including a motion capture server (200) and a digital human server (100) connected to a 5G network.
a) A step in which the motion capture server (200) collects digital human motion information including at least one of face motion information indicating facial movement and body motion information indicating body and hand movement as movement information of a user model through a motion capture device;
b) A step in which the digital human server (100) configures a virtual model based on a preset virtual model configuration method;
c) a step in which the digital human server (100) receives digital human motion information from the motion capture server (200); and
d) A metaverse motion synchronization method, comprising a step of generating digital human information by matching the digital human motion information with the virtual model based on a preset motion synchronization method by the digital human server (100).
In paragraph 1,
The above preset motion synchronization method is,
A metaverse motion synchronization method characterized in that synchronization is achieved based on the difference between a first time when the motion capture server (200) collects digital human motion information from the motion capture device and a second time when the digital human server (100) receives the digital human motion information.
In paragraph 2,
The above digital human motion information includes at least one reference motion information,
The above preset motion synchronization method is,
A metaverse motion synchronization method characterized in that the above-mentioned standard motion information is automatically synchronized every time it is transmitted to either the motion capture server (200) or the digital human server (100).
In paragraph 1,
The above preset motion synchronization method is,
A metaverse motion synchronization method characterized in that synchronization is achieved based on face motion information collected from the motion capture device and body motion information collected from the motion capture device in response to the face motion information.
In paragraph 1,
The above preset motion synchronization method is,
A metaverse motion synchronization method characterized in that synchronization is performed based on body motion information including hand motion information collected from the motion capture device and body motion information including body motion information collected from the motion capture device in response to the hand motion information.
In paragraph 1,
The above digital human motion information further includes voice information of the user model collected from the motion capture server (200).
The above preset motion synchronization method is,
A metaverse motion synchronization method characterized in that synchronization is achieved based on the above voice information and the facial motion information collected from the motion capture device in response to the above voice information.
In paragraph 1,
The above motion capture server (200) is configured to collect digital human motion information from the motion capture device by being connected to at least one of a Wi-Fi and a 5G network.
In the above preset motion synchronization method,
A metaverse motion synchronization method characterized in that pre-motion synchronization is performed by applying preset weights according to the type of network connected to the above motion capture server (200).
A metaverse motion synchronization system (1) including a motion capture server (200) and a digital human server (100) connected to a 5G network,
A motion capture server (200) connected to a motion capture device and a network to collect digital human motion information including at least one of face motion information representing facial movement as movement information of a user model and body motion information representing movement of the body and hands; and
A metaverse motion synchronization system, comprising: a digital human server (100) that performs a function of configuring a virtual model based on a preset virtual model configuration method, a function of receiving digital human motion information from the motion capture server (200), and a function of generating digital human information by matching the digital human motion information with the virtual model based on a preset motion synchronization method.

Images