WO2025152573A1 - Information display method based on dynamic digital human avatar, and electronic device - Google Patents

Abstract

Disclosed in the embodiments of the present application are an information display method based on a dynamic digital human avatar, and an electronic device. The method comprises: obtaining video content in a plurality of real angles of view, wherein the video content in the plurality of real angles of view is obtained by means of a plurality of camera devices synchronously photographing, from different angles of view, the process of a real person in target clothes showing the target clothes; respectively extracting, from a plurality of video frames included in the video content, person images included in the plurality of video frames; using image-based rendering technology to generate person images at corresponding time points for a plurality of intermediate angles of view between adjacent real angles of view; and generating a multi-angle-of-view video on the basis of the person images at a plurality of time points in the plurality of real angles of view and the plurality of intermediate angles of view, such that the multi-angle-of-view video is matched on a client into a preset virtual 3D space scene model. By means of the embodiments of the present application, a more realistic visual experience can be provided for a user at a lower cost.

Description

Method and electronic device for information display based on dynamic digital human image

Cross-references

This application refers to Chinese Patent Application No. 202410069436X, filed on January 17, 2024, entitled “Method and electronic device for information display based on dynamic digital human image”, which is incorporated into this application in its entirety by reference.

Technical Field

The present application relates to the field of information processing technology, and in particular to a method and electronic device for displaying information based on a dynamic digital human image.

Background Art

With the rapid development of science and technology, especially the maturity of computer vision, augmented reality (AR) and virtual reality (VR) technologies, virtual-reality fusion technology is attracting more and more attention. This technology provides users with an immersive experience by integrating real-world elements and computer-generated virtual elements, and it has opened up new application areas and opportunities for various industries. Especially in industries such as entertainment, advertising and e-commerce, the rise of these technologies has not only brought new experiences to consumers, but also opened up new markets and marketing opportunities for companies. For example, in e-commerce platforms, for the clothing industry, functions such as "model catwalk" and "fitting room" can be provided to bring consumers an immersive visual experience.

However, in the process of realizing the above functions, new technical and design challenges are also raised. For example, in the "model catwalk" scene, how to provide a more realistic virtual environment and "digital human", etc. Among them, regarding "digital human", in the existing technology, there are usually the following implementation methods:

Method 1: Traditional 3D portrait modeling technology: Use high-precision 3D scanning equipment to scan real models, and then manually or semi-automatically perform texture mapping and detail carving to generate a human body model. However, this method usually requires expensive scanning equipment and highly skilled experts to process the scanned data. It takes a long time from scanning to the completed model. In addition, it is mainly suitable for static modeling. Dynamic expressions and motion capture require additional work.

Method 2: AI-based portrait synthesis: Using neural networks and a large amount of training data, directly synthesize or convert portrait perspectives and postures. However, this method requires a large amount of labeled data for training, and in some complex scenes and angles, the generated results may not be realistic enough. In addition, real-time applications may require high-performance hardware support and high computing requirements.

Method 3, stereo image rendering solution based on depth image: Use color image and corresponding depth image to generate new viewpoint image. Through depth information, this method can estimate the 3D structure of the scene and render the scene from a new viewpoint. However, the output quality of this method is heavily dependent on the quality of the depth image, and low-quality or inaccurate depth information may also cause artifacts or distortion in the rendered image. In addition, the acquisition cost of depth image is relatively high, and it needs to be obtained through some hardware, lidar, etc., which may increase cost and complexity.

Therefore, how to provide users with a more realistic visual experience at a lower cost in the process of information display based on digital humans has become a technical problem that needs to be solved by technical personnel in this field.

Summary of the invention

The present application provides a method and electronic device for displaying information based on a dynamic digital human image, which can provide users with a more realistic visual experience at a lower cost.

This application provides the following solutions:

A method for displaying information based on a dynamic digital human image, comprising:

Obtaining video contents from multiple real perspectives, wherein the video contents from multiple real perspectives are obtained by synchronously shooting a process in which a real person wears a target garment and displays the target garment from different perspectives using multiple camera devices;

Extracting the character images contained in the plurality of video frames respectively from the video content;

Using image-based rendering technology, based on pixel offset relationship information between character images at adjacent real perspectives at the same time point, character images at corresponding time points are generated for multiple intermediate perspectives between the adjacent real perspectives;

A multi-perspective video is generated based on the character images of the multiple real perspectives and the multiple intermediate perspectives at multiple time points, so that the multi-perspective video can be matched to a preset virtual 3D space scene model on the client, so as to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene, and provide an interactive effect for simulating continuous perspective switching.

The video contents of the multiple real perspectives are obtained by synchronously shooting a process in which a real person wearing target clothing walks in a target space to display the target clothing through multiple camera devices from different perspectives.

The step of generating character images at corresponding time points for a plurality of intermediate perspectives between the adjacent real perspectives includes:

By using image-based rendering technology, according to the pixel offset relationship information between the character images of adjacent real perspectives at the same time point, the pixel positions of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points are estimated to generate the character images of the intermediate perspectives at multiple time points.

The estimating pixel positions of a plurality of intermediate perspectives between the adjacent real perspectives at corresponding time points includes:

Taking the character images corresponding to adjacent real perspectives at the same time point as input, a dense optical flow field between the character images of adjacent real perspectives is fitted through a deep learning model, and the dense optical flow field is used to estimate the pixel positions of the character images of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points.

Among them, it also includes:

The multi-view video is compressed for transmission to a client.

The step of compressing the multi-view video includes:

The video frames corresponding to the multiple perspectives are spliced in units of time points to obtain a frame sequence formed by multiple combined frames; wherein the multiple video frames corresponding to the multiple perspectives at the time point are divided into multiple sets, and the multiple video frames in each set are spliced into a combined frame, and the video frames of adjacent perspectives are located at the same position of adjacent combined frames;

A general video encoder is used to encode a frame sequence formed by the plurality of combined frames, and inter-frame compression processing is performed on the plurality of combined frames.

The resolution of each combined frame is lower than the maximum resolution supported by the terminal device.

Among them, it also includes:

After encoding and inter-frame compression processing are performed on a frame sequence formed by a plurality of combined frames, the frame sequence is further sliced so as to be transmitted in units of the obtained segments after slicing and independently decoded and played in units of the segments at the receiving end.

Wherein, when encoding a frame sequence formed by a plurality of combined frames, the method further includes:

According to the number of combined frames included in each slice, the key frame interval in the inter-frame coding process is controlled so as to reduce the number of frames encoded as key frames in the same slice.

Wherein, when encoding a frame sequence formed by a plurality of combined frames, the method further includes:

For combined frames other than key frames, the number of frames encoded as bidirectional reference frames in the same slice is increased by lowering the judgment threshold for bidirectional reference frames.

A method for displaying information based on a dynamic digital human image, comprising:

In response to a viewing request initiated by a user, a multi-view video is obtained, wherein the multi-view video is generated in the following manner: a process of a real person wearing a target clothing and displaying the target clothing is synchronously photographed by multiple camera devices from different perspectives to obtain video contents of multiple real perspectives, character images contained in multiple video frames contained in the video contents of the multiple real perspectives are respectively extracted, character images are generated for multiple intermediate perspectives between adjacent real perspectives using an image-based rendering technology, and the multi-view video is generated according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points;

Decoding the multi-view video;

Matching the multi-view video to a preset virtual 3D space scene model to provide a dynamic digital human image displaying the target clothing in the virtual 3D space scene;

In response to the interactive operation of continuous perspective switching, an interactive effect simulating continuous perspective switching is provided by switching to character image video content of other perspectives.

A method for generating a dynamic digital human image, comprising:

Obtaining video contents from multiple real perspectives, wherein the video contents from multiple real perspectives are obtained by synchronously shooting a process in which a real person performs a target action from different perspectives using multiple camera devices;

Extracting the character images contained in the plurality of video frames respectively from the video content;

Using image-based rendering technology, based on pixel offset relationship information between character images at adjacent real perspectives at the same time point, character images at corresponding time points are generated for multiple intermediate perspectives between the adjacent real perspectives;

A multi-perspective video is generated according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points, so as to display the corresponding dynamic digital human image through the multi-perspective video.

A method for displaying clothing information based on a dynamic digital human image, comprising:

In response to a request for displaying the target clothing through a dynamic digital human, a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video are obtained, wherein the multi-view video is used to display the process of the target person displaying the target clothing when wearing the target clothing from multiple perspectives;

Match the multi-view video to the virtual 3D space scene model to provide a dynamic digital human image in the virtual 3D space scene model. The target clothing is displayed in the virtual 3D space scene, and an interactive effect for simulating continuous perspective switching is provided.

A device for displaying information based on a dynamic digital human image, comprising:

A video content obtaining unit, configured to obtain video contents of multiple real perspectives, wherein the video contents of the multiple real perspectives are obtained by synchronously shooting a process in which a real person wears a target garment and displays the target garment from different perspectives using multiple camera devices;

A character image extraction unit, used to extract character images contained in the plurality of video frames contained in the video content respectively;

A perspective synthesis unit, configured to generate character images at corresponding time points for a plurality of intermediate perspectives between adjacent real perspectives based on pixel offset relationship information between character images at the same time point from adjacent real perspectives by using an image-based rendering technology;

A multi-perspective video storage and unit is used to generate a multi-perspective video based on the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points, so as to match the multi-perspective video to a preset virtual 3D space scene model on the client, to provide content of a dynamic digital human image displaying the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching.

A device for displaying information based on a dynamic digital human image, comprising:

A multi-view video acquisition unit is used to acquire a multi-view video in response to a viewing request initiated by a user, wherein the multi-view video is generated in the following manner: a process of a real person wearing a target clothing and displaying the target clothing is synchronously photographed by multiple camera devices from different perspectives to obtain video contents of multiple real perspectives, character images contained in multiple video frames contained in the video contents of the multiple real perspectives are respectively extracted, character images are generated for multiple intermediate perspectives between adjacent real perspectives using an image-based rendering technology, and the multi-view video is generated according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points;

A decoding unit, configured to decode the multi-view video;

An adding unit, used for matching the multi-view video to a preset virtual 3D space scene model, so as to provide a content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene;

The perspective switching interaction unit is used to respond to the interactive operation of continuous perspective switching and provide an interactive effect simulating continuous perspective switching by switching to character image video content of other perspectives.

A device for generating a dynamic digital human image, comprising:

A video content obtaining unit, used to obtain video contents of multiple real perspectives, wherein the video contents of the multiple real perspectives are obtained by synchronously shooting a process in which a real person performs a target action from different perspectives using multiple camera devices;

A character image extraction unit, used to extract character images contained in the plurality of video frames contained in the video content respectively;

A perspective synthesis unit, configured to generate character images at corresponding time points for a plurality of intermediate perspectives between adjacent real perspectives based on pixel offset relationship information between character images at the same time point from adjacent real perspectives by using an image-based rendering technology;

A dynamic digital human asset generation unit is used to generate a plurality of digital human assets according to the plurality of real perspectives and the plurality of intermediate perspectives. The character image at a certain time point generates a multi-view video so as to display the corresponding dynamic digital human image through the multi-view video.

A device for displaying clothing information based on a dynamic digital human image, comprising:

A request receiving unit, configured to obtain, in response to a request for displaying a target garment through a dynamic digital human, a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video, wherein the multi-view video is used to display a process of displaying the target garment by the target person in a state of wearing the target garment from multiple viewpoints;

A display unit is used to match the multi-view video to the virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous viewpoint switching.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

An electronic device, comprising:

one or more processors; and

A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the aforementioned methods.

According to the specific embodiments provided in this application, this application discloses the following technical effects:

Through the embodiment of the present application, when it is necessary to provide users with functions such as "model show", multiple camera devices can be used to synchronously shoot the process of real people wearing target clothing and displaying the target clothing from different perspectives to obtain video content from multiple real perspectives. After that, the character images contained in the multiple video frames contained in the video content can be extracted respectively, and then the image-based rendering technology is used to generate character images at corresponding time points for multiple intermediate perspectives between the adjacent real perspectives according to the pixel offset relationship information between the character images of adjacent real perspectives at the same time point. Then, a multi-perspective video can be generated based on the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points, so that the character image video content of one of the perspectives is added to the preset virtual 3D space scene model on the client for rendering and display, so as to provide the content of the dynamic digital human displaying the target clothing in the virtual 3D space scene, and provide an interactive effect for simulating continuous perspective switching. In this way, there is no need to explicitly model the model, and only the images taken from multiple perspectives can be used to achieve a 3D-like real-life digital human effect, avoiding the complex modeling process and high modeling cost, so as to obtain a low-cost, high-fidelity, real-time interactive dynamic real-life digital human effect. In addition, since the format of the real-life digital human asset established in the embodiment of the present application is a video in a common HEVC format, it can be parsed by most mobile devices, and avoids the complex rendering process and large computing overhead.

In addition, for multiple video contents corresponding to multiple perspectives, when performing video encoding, multiple video frames corresponding to the multiple perspectives can be spliced in units of time points to obtain a frame sequence formed by multiple combined frames, and the multiple video frames corresponding to the multiple perspectives at the time point are divided into multiple sets, and multiple video frames in each set are spliced into a combination, and the video frames of adjacent perspectives are located at the same position of adjacent combined frames. Afterwards, a general video encoder can be used to encode the frame sequence formed by the multiple combined frames, and the multiple combined frames can be encoded by encoding the multiple combined frames. Inter-frame compression processing is performed to eliminate or reduce redundant information between video frames of adjacent perspectives. In this way, since the video frames of multiple perspectives are grouped and spliced, the resolution of the spliced combined frame is not too high, which is convenient for real-time decoding in most terminal devices; in addition, since the grouping method and the arrangement method are controlled when performing grouping and splicing, the video frames of adjacent perspectives are located in the same position of adjacent combined frames, that is, the video frames of adjacent perspectives are located in different but adjacent combined frames, and the positions in different combined frames are the same, and the video frames of adjacent perspectives have the characteristic of high similarity. Therefore, the adjacent combined frames spliced in this way have a relatively high similarity, and then a general inter-frame compression algorithm can be used to eliminate or reduce the redundant information between the video frames of adjacent perspectives, and a higher compression rate can be obtained. In other words, in the embodiment of the present application, an ideal compression rate can be obtained by a general video encoder, and accordingly, decoding can be completed by using a general decoder at the decoding end, so that it can be supported on more terminal devices.

Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a system architecture provided in an embodiment of the present application;

FIG2 is a flow chart of a first method provided in an embodiment of the present application;

FIG3 is a schematic diagram of a frame reordering method provided in an embodiment of the present application;

FIG4 is a flow chart of a second method provided in an embodiment of the present application;

FIG5 is a flow chart of a third method provided in an embodiment of the present application;

FIG6 is a flow chart of a fourth method provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

First of all, it should be noted that in the embodiments of the present application, it is mainly possible to provide users with scenes such as "model show" in application systems such as commodity information service systems. The effect to be achieved in this scene is usually: users can view through the client that "digital human" is wearing certain clothes, walking in a certain space scene, or making certain display actions, and it is usually necessary to provide users with an interactive effect of continuous perspective switching. Among them, digital human (Digital Human/Meta Human) is a digital human image close to human image created by digital technology. That is to say, when users are viewing this "model show" interface through mobile phones and other terminal devices, they can switch to an "arbitrary" perspective by sliding on the mobile phone screen, just like the user is personally at the "show" scene and can walk to different positions to watch the model show process.

In the prior art, some commodity information service systems also have products related to "model shows", but they mainly use modeling to generate "3D digital people", and then match 3D clothing models to this "3D digital people" to simulate the effect of real people "walking on the catwalk". In this process, interactive effects such as multi-view viewing can be provided, but because the "3D digital people" are generated by 3D modeling, they look obviously "cartoon-like", not realistic enough, and their movements such as walking and posture are not natural enough; in addition, it is difficult to fully restore the characteristics of clothing through 3D clothing models, etc.

In the embodiments of the present application, in order to provide users with a more realistic visual experience in scenes such as the "model show" and "fitting room" in the product information service system, so that users can see more realistic character images (rather than cartoon images that appear to lack relative realism), an implementation solution is provided for providing users with functions such as "model show" by using "real digital people".

Specifically, multi-view videos can be used to replace 3D modeling in the prior art. That is, videos with multiple viewpoints can be obtained by shooting with multiple cameras, etc. Then, the multi-view videos can be used to provide users with a "model show" function, making the figures and clothing in the "model" show more realistic. In addition, since the assets actually produced exist in the form of multi-view videos rather than models, they can be rendered and displayed in more terminal devices.

In the process of realizing the "model show" through the above-mentioned multi-view video, there are still the following problems: As mentioned above, in the "model show" function, it is usually necessary to provide users with an interactive effect of continuous perspective switching, for example, allowing users to switch perspectives continuously by sliding the screen. However, in the embodiment of the present application, since a specific "digital person" is represented by a multi-view video, and the perspective is discrete, if the perspective needs to be switched during the client display process, it can only be switched between these fixed perspectives, for example, from perspective 1 to perspective 2. If the distance between the perspectives is relatively large, the user may feel an obvious jump phenomenon. Of course, in theory, if the number of camera devices is large enough and the distribution is dense enough so that video content from more perspectives can be obtained, then during the playback process at the playback end, even if switching between discrete perspectives, the interactive effect of continuous perspective switching can be simulated to a certain extent. However, the more camera devices there are, the higher the cost.

Therefore, in order to achieve the simulation of continuous perspective switching at a lower cost, the embodiment of the present application also adopts an image-based rendering technology, through which the images actually captured by two adjacent perspectives can be used to estimate the pixel positions at multiple intermediate perspectives and generate images at intermediate perspectives. In other words, there is no need to actually increase the number of camera devices, but the image-based rendering technology can be used to supplement the images at multiple intermediate perspectives where camera devices are not actually deployed, so as to generate image content at more perspectives, thereby achieving the purpose of better simulating continuous perspective switching.

Of course, in the specific implementation, it may be necessary to provide a variety of different scenes in the "Model Show" function, such as indoor scenes, outdoor scenes, technological scenes, etc. If these scenes are built directly in the real space, the cost may still be relatively high, and the same model character will be required to re-complete the show in multiple different scenes and re-shoot, etc. Therefore, the time cost will also be relatively high.

To this end, in the embodiment of the present application, the generation of real digital people and the creation of scenes can be performed independently. After obtaining multi-perspective video content through camera equipment, the model characters in the video can be "cut out", and the subsequent image synthesis of the intermediate perspective can be performed based on the character image obtained after cutting out. In addition, a variety of different 3D virtual space models can be provided. When displayed on the client, the multi-perspective character image video can be placed in the 3D virtual space scene model for display. In this way, the same multi-perspective character image video can be placed in different 3D virtual space scene models for display.

In addition, since the embodiments of the present application involve multi-view video, and in order to simulate continuous view switching, the number of view angles (including real view angles and intermediate view angles supplemented by algorithms) may be very large (may be one view angle every 3 degrees or 5 degrees, so there are 120 or 72 view angles, etc.), at this time, the compression and transmission of such multi-view video are also issues that need to be considered. In response to this problem, the embodiments of the present application also provide a corresponding solution, which will be described in detail later.

From the perspective of system architecture, referring to FIG1 , the embodiment of the present application may involve an acquisition end, a service end, and a client end. Among them, at the acquisition end, multiple camera devices may be used to shoot processes such as model task catwalks to obtain video content from multiple real perspectives. At the service end, portrait cutouts, image synthesis of intermediate perspectives, multi-perspective video compression, and other processes may be performed. The compressed multi-perspective video may be transmitted to the client, and slice processing may also be performed during transmission to shorten the waiting delay of the client. At the client end, the multi-perspective video may be downloaded and a 3D space scene model may be rendered, and a "digital human" (a character image video content of a certain default perspective) may be rendered according to the multi-perspective video, and then the "digital human" may be added to the 3D space scene model. In this process, some light and shadow fusion processing may also be performed, for example, including achieving shadow consistency under different perspectives, etc., to enhance the display effect. During the display process, the user's perspective switching operation may be responded to, and the effect of continuous perspective switching may be simulated by switching to the video content of other adjacent perspectives. In addition, the zoom operation performed by the user may also be responded to, etc.

The specific implementation scheme provided in the embodiments of the present application is described in detail below.

Embodiment 1

First, from the perspective of the server, this embodiment 1 provides a method for displaying information based on a dynamic digital human image. Referring to FIG. 2 , the method may specifically include:

S201: Obtain video contents from multiple real perspectives, where the video contents from multiple real perspectives are obtained by synchronously shooting a process in which a real person wears a target garment and displays the target garment from different perspectives using multiple camera devices.

Among them, the so-called real perspective is the perspective at which camera equipment is actually deployed for shooting. In specific implementation, for the "model show" function, first, multiple camera equipment can be arranged 180/360 degrees around real people (that is, real model characters) in real space scenes such as studios to ensure that each perspective can be clearly photographed. When the real model characters are wearing the specific clothing required to be displayed, all camera equipment are used to perform synchronous shooting, so that the movements and postures of the characters are captured at the same time. In this way, the specific model characters can walk or make certain display actions in the above-mentioned space scene while wearing the specific clothing required to be displayed. In this process, the above-mentioned multiple camera devices can be synchronously shot to obtain video content from multiple perspectives. In an embodiment of the present application, the number of specific camera devices does not need to be too large, for example, it can be on the order of more than a dozen, and so on.

S202: Extracting character images contained in the plurality of video frames respectively from the plurality of video frames contained in the video content.

After acquiring video content from multiple perspectives, since it is necessary to generate a "digital human" in order to facilitate integration with multiple different scene models, the character images contained therein can first be extracted from the video frames of multiple perspectives. Specifically, since each perspective corresponds to a video content, each video content includes multiple video frames, and each video frame can include a character image, the character image can be extracted from it using the "cutout" technology. In this way, multiple character images can be extracted from each perspective, corresponding to the same person. The "multiple character images" here refer to the character images extracted from the video frames at multiple different time points in each perspective. Since the video frame has time information, the extracted character images will also have corresponding time point information. These character images can be arranged according to the time point to form a character image sequence, and a character image sequence can be obtained from each perspective.

S203: Using image-based rendering technology, according to pixel offset relationship information between character images at the same time point in adjacent real perspectives, character images at corresponding time points are generated for multiple intermediate perspectives between the adjacent real perspectives.

After obtaining the character image sequence corresponding to each perspective, in order to better simulate the continuous perspective switching effect, character image sequences at more perspectives can be synthesized by interpolation. Specifically, the perspective synthesis process can be performed between every two adjacent real perspectives. For example, assuming that there are 10 real perspectives, and perspective 1 is adjacent to perspective 2, then character images can be synthesized for multiple intermediate perspectives between perspective 1 and perspective 2, that is, it is estimated that if a camera device is provided at the intermediate perspective, what the captured character image will look like. In addition, image synthesis can also be performed for intermediate perspectives between perspective 2 and perspective 3, between perspective 3 and perspective 4, and other adjacent perspectives. In this way, multiple images at intermediate perspectives can be estimated between every two adjacent real perspectives.

Among them, the selection of the intermediate viewing angle can be determined according to actual needs. For example, after testing, if a viewing angle is set every 3 degrees, the user will not obviously feel the jump phenomenon during the switching between viewing angles. Therefore, the interval of the intermediate viewing angle can be set to 3 degrees; of course, if in order to better simulate continuous viewing angle switching, the interval of the intermediate viewing angle can be set smaller, or, if in order to avoid too high a transmission bit rate, the interval of the intermediate viewing angle can be set larger, and so on.

After determining the interval of the intermediate perspectives, the pixel positions of each intermediate perspective can be estimated based on the pixel positions of the character images corresponding to the two adjacent perspectives and the pixel offset relationship information, and then the character images at the specific intermediate perspective can be synthesized. Among them, since there are multiple character images at each perspective, each corresponding to a different time point, when synthesizing the images of the intermediate perspectives, it can also be performed in units of time points, that is, multiple character images corresponding to each time point are generated for the specific intermediate perspective, so that each intermediate perspective can correspond to a character image sequence.

In specific implementation, there can be multiple ways to estimate the pixel position of the intermediate perspective based on the character images of two adjacent perspectives. For example, in one method, the character images corresponding to the adjacent real perspectives at the same time point can be used as input, and the dense optical flow field between the character images corresponding to the adjacent perspectives can be fitted through a deep learning model. After that, this dense optical flow field can be used to estimate the pixel positions of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points. Among them, optical flow is the instantaneous speed of pixel movement of a spatial moving object on the observation imaging plane. The optical flow method is a method that uses the changes in pixels in the time domain in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. In space, motion can be described by a motion field, while on an image plane, the motion of an object is often described by a motion field. The different manifestations of the grayscale distribution of different images in the image sequence, thus, the motion field in the space is transferred to the image and represented as an optical flow field. The optical flow field is a two-dimensional vector field, which reflects the changing trend of the grayscale of each point on the image, and can be regarded as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information it contains is the instantaneous motion velocity vector information of each image point. Among them, dense optical flow is an image registration method that performs point-by-point matching on an image or a specified area. It calculates the offset of all points on the image to form a dense optical flow field. Through this dense optical flow field, pixel-level image registration can be performed. Therefore, in an embodiment of the present application, this optical flow field information can be first estimated, and then, based on the optical flow field, the pixel position of the character image of each intermediate perspective at the corresponding time point can be estimated, and then the corresponding character image can be generated for the intermediate perspective.

S204: Generate a multi-perspective video based on the character images of the multiple real perspectives and the multiple intermediate perspectives at multiple time points, so that the multi-perspective video can be matched to a preset virtual 3D space scene model on the client for rendering and display, so as to provide content of a dynamic digital human image displaying the target clothing in the virtual 3D space scene, and provide an interactive effect for simulating continuous perspective switching.

After obtaining a sequence of character images of multiple intermediate perspectives, the sequences composed of multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points can be organized into video formats, respectively, to obtain multi-perspective videos. That is to say, in an embodiment of the present application, a multi-perspective video may include videos corresponding to multiple real perspectives, and videos corresponding to multiple intermediate perspectives. This multi-perspective video can be used to express a "digital person", which is generated by shooting, cutting out, and synthesizing intermediate perspectives of real people, rather than by 3D modeling. Therefore, while maintaining the 3D effect, the "digital person" can also look more real and reduce the cartoon feeling, and a more realistic and natural display effect can be achieved for goods such as clothing.

After obtaining the multi-view video in the above manner, the multi-view video can also be compressed and encoded so as to be provided to the client for display. When the client displays it, the specific multi-view video can be decoded. Since the image has been cut out before, the multi-view video can have features such as a transparent background. Therefore, it can be placed in a pre-generated 3D space scene model, thereby presenting a visual effect of a specific "digital human" in the 3D space scene model.

Specifically, when performing compression coding, since multi-view video is involved, the compression coding method can also be specially processed in the embodiment of the present application. This is because, compared with ordinary videos, multi-view videos usually have the following characteristics: In terms of resolution: multi-view videos may include video data corresponding to dozens or even hundreds of viewpoints. If each viewpoint corresponds to a high-definition video, the overall video data will be huge. Assuming there are 120 viewpoints, the overall resolution will exceed 32K or even higher, which is a large load for most devices; in terms of video bit rate, high resolution means higher video bit rate, which will make real-time transmission and smooth playback more difficult. The bit rate of ordinary 720P video may be 2-5Mbps, but the bit rate of multi-view video will increase by dozens or even hundreds of times. In terms of video volume, multi-view video contains data from multiple viewpoints, which leads to a rapid increase in the volume of video files. One hour of multi-view video may require tens or even hundreds of GB of storage space. These technical challenges make the storage, compression and transmission of multi-view videos particularly difficult.

In the prior art, there are some solutions for compressing and transmitting multi-view videos. For example:

Method 1: You can simply stitch multi-view videos together. Specifically, you can stitch all the views at the same time point together. The images are stitched together into the same frame and then compressed for transmission. However, this will result in a spliced video with a resolution that is too high, making it difficult to decode and play in real time and difficult to transmit.

Method 2 is to transmit through streaming media. However, on the one hand, the compression efficiency is not ideal. On the other hand, in order to enable the client to switch perspectives, the multi-perspective videos need to be sliced and processed separately before streaming. When the user needs to switch from perspective A to perspective B at a certain moment, the data of perspective B in the corresponding time slice can be pulled for playback. However, the delay of segmented streaming is relatively large, and whether smooth perspective switching can be performed depends on the size of the slice, because the previous slice must be played before switching to the next slice of the next perspective for playback.

Method three, use a codec designed for multi-view video for video encoding and decoding. This method provides better compression performance, but it increases the complexity of encoding and decoding and requires higher computing power. In addition, since MV-HEVC is a relatively new standard, a dedicated decoder is required to complete the decoding. Therefore, not all devices support this format, especially ordinary mobile phones or terminal devices such as computers usually cannot support it. Even if they can support it, there will be freezes and other phenomena when opening and playing the video.

In view of the above situation, in the embodiments of the present application, corresponding solutions are also provided for the storage and compression of multi-view videos. Specifically, before encoding a specific multi-view video, the video contents corresponding to the multiple viewpoints can be spliced first, that is, the video contents corresponding to different viewpoints at the same time point can be spliced, but it is not simply to splice all the video contents of all viewpoints into the same frame, but to group the multiple video frames corresponding to the multiple viewpoints at the same time point to obtain multiple sets, and splice the multiple video frames in the same set into the same frame (for the convenience of distinction, the frame obtained after such splicing can be called a "combined frame"). In other words, the same time point can correspond to multiple different combined frames. In this way, the resolution of the combined frame can be made not too high.

Specifically, when grouping multi-view video frames, the number of groups to be divided and the number of view angles included in each group can be determined based on the number of specific view angles, the resolution of a single video frame at each view angle, and the maximum resolution supported by the terminal device, so that the resolution of each combined frame is lower than the maximum resolution supported by the terminal device. For example, assuming that there are 72 view angles in total, and the resolution of the video frame of each view angle is 720P, most of the terminal devices on the market can usually support real-time decoding of 4K resolution. At this time, the above 72 view angles can be divided into 12 groups, and each combined frame will include 6 video frames corresponding to the view angles at the same time point. The resolution of each combined frame is 720P×6＝4320P, which is close to the resolution of a general 4K image. Therefore, most of the terminal devices can achieve real-time decoding of image frames of this resolution.

After determining the specific number of groups, in order to achieve higher compression efficiency during the encoding process, the grouping method of different perspectives and the arrangement method in the specific combined frame can also be determined. There are many specific grouping methods and arrangement methods. For example, in the simplest way, the 1st to 6th perspectives can be grouped as a group, the 7th to 12th perspectives can be grouped as a group, and so on. In the specific combined frame, it can be divided into 3×2 (three rows and two columns) blocks, and each perspective can be arranged in these blocks in order of number, and so on. However, considering that the video frames shot at the same time point from different perspectives often have a relatively high similarity in content, especially the adjacent perspectives, the similarity between the two will be higher. From the perspective of information encoding, there will be a lot of redundant information in the highly similar content between adjacent perspectives, which is content that can be compressed during the encoding process. In other words, the existence of redundant information is conducive to improving the compression efficiency of encoding. Therefore, in the encoding process, if this redundant information can be fully utilized, it will be helpful to improve the compression efficiency. It will be of great help.

In the process of video encoding, specific information compression technologies can be divided into two types: intra-frame compression and inter-frame compression. Intra-frame compression is compression in the spatial domain (on the XY axis of space), and the similarity between the data of the current frame is mainly referred to during the compression process; while inter-frame compression uses the inter-frame redundancy between different video frames in the video sequence, such as the similarity between the previous and next frames, to reduce the amount of data through a prediction method. Generally, compared with intra-frame compression, inter-frame compression can usually achieve a higher compression rate.

However, if the simple perspective grouping and arrangement method described in the above example is followed, the video frames of adjacent perspectives with the highest redundancy are in the same combined frame. Therefore, when the combined frame is compressed, this redundant information can only be used in the intra-frame compression process and cannot be fully utilized in the inter-frame encoding process.

To this end, in an embodiment of the present application, a more optimal perspective grouping and arrangement method is also provided. Specifically, the video frames of adjacent perspectives can be located at the same position of adjacent combined frames. That is to say, the video frames of adjacent perspectives will be divided into different but adjacent groups, and will be located at the same position of adjacent combined frames.

For example, suppose there are 36 views (the number of views is reduced for ease of description), which are divided into 6 groups, each group has 6 video frames of views, that is, every 6 video frames of views form a combined frame. As shown in FIG3, suppose each combined frame includes 3×2 blocks, each block is used to place a video frame of a view, and the position number of each block is 0, 1, 2, 3, 4, 5 respectively; in addition, suppose that the 36 views are represented by A1, A2, A3...A36 respectively. It can be seen from FIG3 that the views A1, A2, A3, A4, A5, A6 are located at the 0th position of the combined frames 1 to 6, and the views A7, A8, A9, A10, A11, A12 are located at the 1st position of the combined frames 1 to 6, and so on. That is to say, the perspectives A1, A7, A13, A19, A25, and A31 are the first group, which are spliced into group frame 1; A2, A8, A14, A20, A26, and A32 are the second group, which are spliced into group frame 2, and so on. It can be seen that in each combined frame, the perspective numbers form an arithmetic progression, and the difference between the perspective numbers is the number of groups, which is 6 in this example. In this way, the perspectives at the same position between adjacent combined frames can be adjacent, which makes the image content at the same position between adjacent groups of frames have a high degree of similarity. When performing inter-frame compression coding, the redundant information generated by the high content similarity between adjacent perspectives can be fully utilized, which is conducive to obtaining a higher compression rate.

Of course, the above example only shows the splicing of video frames of each perspective at one time point, and the video frames of each perspective at other time points can also be grouped and arranged in the above manner. In this way, 6 combined frames can be spliced at each time point. After the splicing is completed at each time point in the above manner, the obtained combined frames can be formed into a frame sequence. For example, if each combined frame is represented as "combined frame mn", where m represents the number of the time point and n represents the number of each combined frame corresponding to the same time point, the formed frame sequence can be: (combined frame 11, combined frame 12, combined frame 13, combined frame 14, combined frame 15, combined frame 16, combined frame 21, combined frame 22, combined frame 23, combined frame 24, combined frame 25, combined frame 26, combined frame 31, combined frame 32...).

Through the above grouping and arrangement, the video frames of adjacent perspectives are dispersed into different but adjacent combined frames, and are located at the same position of adjacent frames. There is usually a high degree of similarity between video frames of adjacent perspectives. Therefore, the adjacent combined frames, at least the video frames at each position, have a high degree of similarity, that is, there is a large amount of redundant information, which can be compressed and optimized in the process of inter-frame coding. object. Therefore, when encoding the combined frame, the redundant information between the video frames of adjacent perspectives can be fully utilized through inter-frame coding to obtain higher compression efficiency. Moreover, since this high compression rate is achieved through inter-frame coding technology, and the general video encoder has the ability of inter-frame coding, the encoding can be achieved using a general video encoder without relying on an encoder dedicated to multi-perspective coding. Correspondingly, the decoding can be performed using a general video decoder at the playback end, thereby further supporting decoding and playback in more terminal devices.

That is to say, through the embodiment of the present application, a method of grouping and splicing video frames of multiple perspectives is adopted. Compared with the method of directly splicing all perspectives into the same frame, the resolution of each combined frame can be reduced, and the decoding pressure of the terminal device can be reduced; in addition, the arrangement of different perspectives in different combined frames is specially processed so that the content similarity between different combined frames will be relatively high. In this way, the redundant information between video frames of adjacent perspectives can be eliminated or reduced by compressing the combined frames between frames, so as to obtain a higher compression rate, so that a general encoder (for example, HEVC (High Efficiency Video Coding, high-efficiency video coding) etc.) can complete the encoding process. Correspondingly, a general decoder can also be used for decoding, so that a specific video can be decoded and played in most terminal devices, avoiding complex rendering processes and large computing overheads as well as dependence on dedicated codecs.

After encoding and inter-frame compression of a frame sequence formed by multiple combined frames, it can be used for transmission to a client for decoding and display at the client. In an optional implementation, the frame sequence can be sliced before transmission so that it can be transmitted in units of fragments obtained after slicing, and the receiving end can decode and play the fragments independently. In this way, the receiving end only needs to receive the first fragment to decode and play, without waiting for all frame sequences to be transmitted, thereby shortening the waiting delay.

Among them, the specific slice duration can be determined according to actual needs. If the slice is smaller, the delay at the receiving end will be smaller. For example, the duration of each slice can be 1S, or it can also be 0.5S, and so on. Among them, in the embodiment of the present application, since the video frames of multiple perspectives are grouped and spliced, after determining the slice duration, the number of combined frames that need to be included in each slice can be determined according to the playback frame rate of the playback end. For example, still taking 72 perspectives divided into 12 groups as an example, each time point corresponds to 12 combined frames. In addition, assuming that the playback frame rate of the playback end is 30 frames/S, the slice duration when the combined frame is sliced is 1S, then each segment needs to contain 30×72/6=360 combined frames. That is to say, the combined frames included in each segment need to meet the number of frames required to be played by the playback end within a duration of 1S, wherein the playback end specifically needs to decode the combined frames, select the video frames corresponding to a certain perspective, and play them, and the 30 frames played by the playback end within 1S are usually 30 video frames under the same perspective, and it is possible to play any perspective. Therefore, when slicing the combined frames, if each segment is 1S, it is necessary to make 30 video frames exist in each perspective in the same segment. If the number of perspectives is 72, the number of video frames is 30×72. Since these video frames are grouped and spliced into combined frames, the number of combined frames is 30×72/6=360. Of course, under the above assumptions, if the segment duration is changed to 0.5S, each segment can include 180 combined frames, and so on.

In addition, if the above-mentioned segmented transmission is performed, the compression rate can be further improved by controlling the number of key frames and bidirectional reference frames during inter-frame coding. Specifically, the encoder encodes multiple images into GOPs (Group of Pictures) one by one, and the decoder reads GOPs one by one during playback. After decoding, the picture is read and then rendered for display. GOP is a group of continuous pictures, consisting of an I frame and several B/P frames. It is the basic unit for video image encoders and decoders to access. Its arrangement order will be repeated until the end of the image. Among them, I frame is an internal coding frame (also called a key frame), P frame is a forward prediction frame (forward reference frame), and B frame is a bidirectional interpolation frame (bidirectional reference frame). Specifically, I frame is usually a complete picture, while P frame and B frame record the changes relative to I frame. Among them, P frame and B frame do not have complete picture data. P frame only has the data of the picture difference with the previous frame, and B frame records the difference between this frame and the previous and next frames. Among them, the amount of information required to be recorded by B frame is relatively small, so it usually has a higher compression rate. If a GOP includes fewer I frames and more B frames, the overall compression rate will be higher.

In practical applications, which frames will be encoded as I frames, P frames, B frames, etc. are usually determined by the encoder according to the algorithm. In the embodiment of the present application, in order to further control the compression rate of the video, the number of I frames can be reduced and the number of B frames can be increased by intervening in the encoder. In specific implementation, the key frame interval in the inter-frame encoding process can be controlled according to the number of combined frames included in each slice, so as to reduce the number of frames encoded as key frames in the same slice. For example, assuming that each slice includes 360 combined frames, the key frame interval can be set to 360 frames or 180 frames, that is, only one or two frames in the same slice will be encoded as I frames. In addition, for combined frames other than key frames, the number of frames encoded as bidirectional reference frames in the same slice can also be increased by lowering the judgment threshold of bidirectional reference frames. That is to say, for B frames, the encoder usually determines whether the current frame can be encoded as a B frame by calculating the similarity between the current frame and the previous and next frames and comparing it with a certain threshold. In the embodiment of the present application, the threshold can be lowered, and more frames can be encoded as B frames to improve the compression rate.

It should be noted here that since the decoding of P frames and B frames depends on I frames, and the decoding of B frames depends on the previous frame and the next frame, theoretically, if the number of I frames is small and the number of B frames is large, although there will be better performance in terms of compression rate, it may affect the image quality during decoding. However, in the embodiment of the present application, since the multi-perspective video frames are grouped and spliced, and the video frames of adjacent perspectives are located in the same position of adjacent combined frames, each two adjacent combined frames have a high degree of similarity. Under this premise, even if the number of I frames and B frames is controlled in the above manner, it usually does not affect the image quality of the decoding end. After testing, the solution provided by the embodiment of the present application has significantly reduced resolution and bit rate compared with the solution of encoding after simple splicing, and PSNR (Peak Signal-to-Noise Ratio, which represents the ratio of the maximum possible power of the signal to the destructive noise power that affects its representation accuracy, and is one of the indicators for measuring image quality) has been improved, as shown in Table 1:

Table 1

Of course, in practical applications, if higher picture quality is required, the number of I frames can be appropriately increased to reduce the number of frames. Reduce the number of B frames, for example, each slice can include 2 or more I frames, and so on.

The above provides a detailed introduction to the compression and transmission of multi-view videos. In actual applications, assuming that a user initiates access to the "model show" through a client, the specific multi-view video can be transmitted to the client, which can download it and use a general video decoder to decode it. Afterwards, the video frames from one of the perspectives (usually one of the perspectives can be set as the default perspective) can be added to a preset 3D space scene model. Among them, the 3D space scene model can be pre-generated by 3D modeling, that is, the client can render the 3D space scene model and the multi-view video separately, and add the video content corresponding to one of the perspectives to the 3D space scene model for display.

In specific implementation, in order to achieve a better mutual integration effect between the video content and the 3D space scene model, the scene perspective and the character perspective can be rendered synchronously, and the perspective can also be switched synchronously. In addition, the light and shadow integration of the character and the scene can be achieved, etc. For example, lighting information can be added to the 3D space scene, and then the character's shadow position, light position, etc. can be calculated based on the lighting information, and shadow information, light information, etc. can be added to the 3D space scene, so that the character and the scene can be better integrated in this way.

In summary, through the embodiments of the present application, when it is necessary to provide users with functions such as "model show", multiple camera devices can be used to synchronously shoot the process of real people wearing target clothing and displaying the target clothing from different perspectives to obtain video content from multiple real perspectives. After that, the character images contained in the multiple video frames contained in the video content can be extracted respectively, and then the image-based rendering technology is used to generate character images at corresponding time points for multiple intermediate perspectives between the adjacent real perspectives according to the pixel offset relationship information between the character images of adjacent real perspectives at the same time point. Then, a multi-perspective video can be generated based on the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points, so that the character image video content of one of the perspectives can be added to the preset virtual 3D space scene model on the client for rendering and display, so as to provide the content of the dynamic digital human displaying the target clothing in the virtual 3D space scene, and provide an interactive effect for simulating continuous perspective switching. In this way, there is no need to explicitly model the model, and only the images taken from multiple perspectives can be used to achieve a 3D-like real-life digital human effect, avoiding the complex modeling process and high modeling cost, so as to obtain a low-cost, high-fidelity, real-time interactive dynamic real-life digital human effect. In addition, since the format of the real-life digital human asset established in the embodiment of the present application is a video in a common HEVC format, it can be parsed by most mobile devices, and avoids the complex rendering process and large computing overhead.

In addition, for multiple video contents corresponding to multiple perspectives, specifically when performing video encoding, the multiple video frames corresponding to the multiple perspectives can be spliced in units of time points to obtain a frame sequence formed by multiple combined frames, and the multiple video frames corresponding to the multiple perspectives at the time point are divided into multiple sets, and the multiple video frames in each set are spliced into a combined frame, and the video frames of adjacent perspectives are located at the same position of adjacent combined frames. Afterwards, a general video encoder can be used to encode the frame sequence formed by the multiple combined frames, and the redundant information between the video frames of adjacent perspectives can be eliminated or reduced by performing inter-frame compression processing on the multiple combined frames. In this way, since the video frames of multiple perspectives are grouped and spliced, the resolution of the spliced combined frame is not too high, which is convenient for real-time decoding in most terminal devices; in addition, since the grouping method and the arrangement method are controlled when performing grouping and splicing, the video frames of adjacent perspectives are located at the same position of adjacent combined frames, that is, the adjacent perspectives are The video frames of the two angles are located in different but adjacent combined frames, and have the same position in different combined frames, and the video frames of adjacent viewing angles have a relatively high similarity. Therefore, the adjacent combined frames spliced in this way have a relatively high similarity, and then a general inter-frame compression algorithm can be used to eliminate or reduce the redundant information between the video frames of adjacent viewing angles, thereby obtaining a higher compression rate. In other words, in the embodiment of the present application, an ideal compression rate can be obtained by a general video encoder, and correspondingly, decoding can be completed by using a general decoder at the decoding end, so that it can be supported on more terminal devices.

Embodiment 2

The second embodiment corresponds to the first embodiment and provides a method for displaying information based on a dynamic digital human image from the perspective of a client. Referring to FIG. 4 , the method may include:

S401: In response to a viewing request initiated by a user, a multi-view video is obtained, wherein the multi-view video is generated in the following manner: a process of a real person wearing a target clothing and displaying the target clothing is synchronously photographed by multiple camera devices from different perspectives to obtain video contents of multiple real perspectives, character images contained in multiple video frames contained in the video contents of the multiple real perspectives are respectively extracted, character images are generated for multiple intermediate perspectives between adjacent real perspectives using an image-based rendering technology, and the multi-view video is generated according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points;

S402: Decoding the multi-view video;

S403: Matching the multi-view video to a preset virtual 3D space scene model to provide a dynamic digital human image displaying the target clothing in the virtual 3D space scene;

S404: In response to the interactive operation of continuous perspective switching, an interactive effect simulating continuous perspective switching is provided by switching to character image video content of other perspectives.

Embodiment 3

The above embodiments 1 and 2 mainly take the example of providing users with functions such as "model show" in the commodity information service system, and introduce the specific implementation scheme. In actual applications, the method of producing dynamic real-life digital people provided in the embodiment of this application can also be used in other application scenarios. To this end, this embodiment 3 also provides a method for generating a dynamic digital human image, see Figure 5, the method may include:

S501: Obtain video contents from multiple real perspectives, where the video contents from multiple real perspectives are obtained by synchronously shooting a process in which a real person performs a target action from different perspectives using multiple camera devices.

The specific target action can be determined according to the needs of the actual scene, for example, it can be performing a certain dance move, etc.

S502: Extracting character images contained in the plurality of video frames respectively from the plurality of video frames contained in the video content.

S503: Using image-based rendering technology, according to pixel offset relationship information between character images at the same time point in adjacent real perspectives, character images at corresponding time points are generated for multiple intermediate perspectives between the adjacent real perspectives.

S504: Generate a multi-perspective video according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points, so as to display the corresponding dynamic digital human image through the multi-perspective video.

The data related to the dynamic digital human image produced by the above method can exist in the form of multi-view video, which is composed of files in ordinary video format corresponding to multiple viewpoints. Therefore, this kind of real digital human has The terminal-side friendly feature allows for display in a variety of scenarios, including display after integration with a certain 3D space scene model, thereby presenting the perspective effect of a specific digital person performing corresponding actions in a specific 3D space scene. In addition, users can perform continuous perspective switching or perspective zooming, etc.

Embodiment 4

In the fourth embodiment, the method of matching a multi-view video to a virtual 3D space scene model is mainly protected. Specifically, the fourth embodiment provides a method for displaying clothing information based on a dynamic digital human image. Referring to FIG. 6 , the method may include:

S601: In response to a request for displaying a target garment through a dynamic digital human, a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video are obtained, wherein the multi-view video is used to display a process of displaying the target garment by the target person in a state of wearing the target garment from multiple viewpoints;

S602: Matching the multi-view video to the virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene, and providing an interactive effect for simulating continuous viewpoint switching.

In the fourth embodiment, the multi-view video can be generated by the method provided in the above embodiment, or can be generated by other methods, for example, the multi-view video can be generated directly by densely deploying more cameras, etc. Specifically, when matching the multi-view video to the virtual 3D space scene model, it can be achieved through a 3D rendering engine. Of course, in the specific implementation, some functional customization can be performed on the basis of a general 3D rendering engine, for example, achieving shadow consistency of multiple different perspectives, etc.

For the contents not described in detail in Embodiments 2 to 4, please refer to Embodiment 1 and other parts of this specification, which will not be repeated here.

It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

Corresponding to the first embodiment, the embodiment of the present application further provides a device for displaying information based on a dynamic digital human image, which may include:

A video content obtaining unit, configured to obtain video contents of multiple real perspectives, wherein the video contents of the multiple real perspectives are obtained by synchronously shooting a process in which a real person wears a target garment and displays the target garment from different perspectives using multiple camera devices;

A character image extraction unit, used to extract character images contained in the plurality of video frames contained in the video content respectively;

A perspective synthesis unit, configured to generate character images at corresponding time points for a plurality of intermediate perspectives between adjacent real perspectives based on pixel offset relationship information between character images at the same time point from adjacent real perspectives by using an image-based rendering technology;

A multi-view video storage and unit is used to generate a multi-view video according to the multiple real view angles and the character images of the multiple intermediate view angles at multiple time points, so as to match the multi-view video to a preset virtual 3D space scene model on the client side, so as to provide a dynamic digital human image to perform the target clothing in the virtual 3D space scene. The content displayed and provide interactive effects for simulating continuous perspective switching.

The video contents of the multiple real perspectives are obtained by synchronously shooting a process in which a real person wearing target clothing walks in a target space to display the target clothing through multiple camera devices from different perspectives.

Specifically, the perspective synthesis unit may be used for:

By using image-based rendering technology, according to the pixel offset relationship information between the character images of adjacent real perspectives at the same time point, the pixel positions of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points are estimated to generate the character images of the intermediate perspectives at multiple time points.

More specifically, the perspective synthesis unit may be used for:

Taking the character images corresponding to adjacent real perspectives at the same time point as input, a dense optical flow field between the character images of adjacent real perspectives is fitted through a deep learning model, and the dense optical flow field is used to estimate the pixel positions of the character images of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points.

In addition, the device may also include:

The video compression unit is used to compress the multi-view video for transmission to a client.

The video compression unit may specifically include:

A splicing processing subunit is used to splice video frames corresponding to multiple viewpoints in units of time points to obtain a frame sequence formed by multiple combined frames; wherein, for the same time point, the multiple video frames corresponding to the multiple viewpoints at the time point are divided into multiple sets, and the multiple video frames in each set are spliced into a combined frame, and the video frames of adjacent viewpoints are located at the same position of adjacent combined frames;

The inter-frame compression subunit is used to encode the frame sequence formed by the multiple combined frames using a universal video encoder, and perform inter-frame compression processing on the multiple combined frames.

The resolution of each combined frame is lower than the maximum resolution supported by the terminal device.

In addition, the device may also include:

The slicing processing unit is used to slice the frame sequence formed by multiple combined frames after encoding and inter-frame compression, so that the frame sequence can be transmitted in units of the obtained segments after slicing, and independently decoded and played in units of segments at the receiving end.

Furthermore, it may also include:

The key frame quantity control unit is used to control the key frame interval in the inter-frame coding process according to the quantity of combined frames included in each slice, so as to reduce the number of frames encoded as key frames in the same slice.

The bidirectional reference frame quantity control unit is used to increase the number of frames encoded as bidirectional reference frames in the same slice by lowering the judgment threshold of the bidirectional reference frames for the combined frames other than the key frames.

Corresponding to the second embodiment, the embodiment of the present application further provides a device for displaying information based on a dynamic digital human image, which may include:

The multi-view video acquisition unit is used to respond to a viewing request initiated by a user and acquire a multi-view video, wherein the multi-view video is generated by: synchronously shooting a process in which a real person wears a target garment and displays the target garment from different perspectives through multiple camera devices to obtain video contents of multiple real perspectives, and Extracting character images contained in multiple video frames contained in video contents of multiple real perspectives respectively, generating character images for multiple intermediate perspectives between adjacent real perspectives by using image-based rendering technology, and generating the multi-perspective video according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points;

A decoding unit, configured to decode the multi-view video;

An adding unit, used for matching the multi-view video to a preset virtual 3D space scene model, so as to provide a content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene;

The perspective switching interaction unit is used to respond to the interactive operation of continuous perspective switching and provide an interactive effect simulating continuous perspective switching by switching to character image video content of other perspectives.

Corresponding to the third embodiment, the embodiment of the present application further provides a device for generating a dynamic digital human image, which may include:

A video content obtaining unit, used to obtain video contents of multiple real perspectives, wherein the video contents of the multiple real perspectives are obtained by synchronously shooting a process in which a real person performs a target action from different perspectives using multiple camera devices;

A character image extraction unit, used to extract character images contained in the plurality of video frames contained in the video content respectively;

A perspective synthesis unit, configured to generate character images at corresponding time points for a plurality of intermediate perspectives between adjacent real perspectives based on pixel offset relationship information between character images at the same time point from adjacent real perspectives by using an image-based rendering technology;

The dynamic digital human asset generation unit is used to generate a multi-view video based on the character images of the multiple real perspectives and the multiple intermediate perspectives at multiple time points, so as to display the corresponding dynamic digital human image through the multi-view video.

Corresponding to the fourth embodiment, the embodiment of the present application further provides a device for displaying clothing information based on a dynamic digital human image, and the device may include:

A request receiving unit, configured to obtain, in response to a request for displaying a target garment through a dynamic digital human, a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video, wherein the multi-view video is used to display a process of displaying the target garment by the target person in a state of wearing the target garment from multiple viewpoints;

A display unit is used to match the multi-view video to the virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous viewpoint switching.

In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

And an electronic device, comprising:

one or more processors; and

A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

7 exemplarily shows the architecture of the electronic device, which may include a processor 710, a video display adapter 711, a disk drive 712, an input/output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input/output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a communication bus 730.

Among them, the processor 710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to realize the technical solution provided in this application.

The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and an information display processing system 725, etc. can also be stored. The above-mentioned information display processing system 725 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

The input/output interface 713 is used to connect the input/output module to realize information input and output. The input/output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

The network interface 714 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710, the video display adapter 711, the disk drive 712, the input/output interface 713, the network interface 714, and the memory 720).

It should be noted that, although the above device only shows a processor 710, a video display adapter 711, a disk drive 712, an input/output interface 713, a network interface 714, a memory 720, a bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiment described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

The above is a detailed introduction to the method and electronic device for information display based on dynamic digital human images provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims (13)

A method for displaying information based on a dynamic digital human image, characterized by comprising: Obtaining video contents from multiple real perspectives, wherein the video contents from multiple real perspectives are obtained by synchronously shooting a process in which a real person wears a target garment and displays the target garment from different perspectives using multiple camera devices; Extracting the character images contained in the plurality of video frames respectively from the video content; Using image-based rendering technology, based on pixel offset relationship information between character images at adjacent real perspectives at the same time point, character images at corresponding time points are generated for multiple intermediate perspectives between the adjacent real perspectives; A multi-perspective video is generated based on the character images of the multiple real perspectives and the multiple intermediate perspectives at multiple time points, so that the multi-perspective video can be matched to a preset virtual 3D space scene model on the client, so as to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene, and provide an interactive effect for simulating continuous perspective switching. The method according to claim 1, characterized in that The video contents of the multiple real perspectives are obtained by synchronously shooting with multiple camera devices from different perspectives a process in which a real person walks in a target space wearing target clothing to display the target clothing. The method according to claim 1, characterized in that The step of generating character images at corresponding time points for a plurality of intermediate perspectives between the adjacent real perspectives includes: By using image-based rendering technology, according to the pixel offset relationship information between the character images of adjacent real perspectives at the same time point, the pixel positions of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points are estimated to generate the character images of the intermediate perspectives at multiple time points. The method according to claim 3, characterized in that The estimating pixel positions of a plurality of intermediate perspectives between the adjacent real perspectives at corresponding time points includes: Taking the character images corresponding to adjacent real perspectives at the same time point as input, a dense optical flow field between the character images of adjacent real perspectives is fitted through a deep learning model, and the dense optical flow field is used to estimate the pixel positions of the character images of multiple intermediate perspectives between the adjacent real perspectives at corresponding time points. The method according to claim 1, further comprising: The multi-view video is compressed for transmission to a client. The method according to claim 5, characterized in that The compressing the multi-view video includes: The video frames corresponding to the multiple perspectives are spliced in units of time points to obtain a frame sequence formed by multiple combined frames; wherein the multiple video frames corresponding to the multiple perspectives at the time point are divided into multiple sets, and the multiple video frames in each set are spliced into a combined frame, and the video frames of adjacent perspectives are located at the same position of adjacent combined frames; A general video encoder is used to encode a frame sequence formed by the plurality of combined frames, and inter-frame compression processing is performed on the plurality of combined frames. The method according to claim 6, characterized in that The resolution of each combined frame is lower than the maximum resolution supported by the end device. The method according to claim 6, further comprising: After encoding and inter-frame compression processing are performed on a frame sequence formed by a plurality of combined frames, the frame sequence is further sliced so as to be transmitted in units of the obtained segments after slicing and independently decoded and played in units of the segments at the receiving end. A method for displaying information based on a dynamic digital human image, characterized by comprising: In response to a viewing request initiated by a user, a multi-view video is obtained, wherein the multi-view video is generated in the following manner: a process of a real person wearing a target clothing and displaying the target clothing is synchronously photographed by multiple camera devices from different perspectives to obtain video contents of multiple real perspectives, character images contained in multiple video frames contained in the video contents of the multiple real perspectives are respectively extracted, character images are generated for multiple intermediate perspectives between adjacent real perspectives using an image-based rendering technology, and the multi-view video is generated according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points; Decoding the multi-view video; Matching the multi-view video to a preset virtual 3D space scene model to provide a dynamic digital human image displaying the target clothing in the virtual 3D space scene; In response to the interactive operation of continuous perspective switching, an interactive effect simulating continuous perspective switching is provided by switching to character image video content of other perspectives. A method for generating a dynamic digital human image, characterized by comprising: Obtaining video contents from multiple real perspectives, wherein the video contents from multiple real perspectives are obtained by synchronously shooting a process in which a real person performs a target action from different perspectives using multiple camera devices; Extracting the character images contained in the plurality of video frames respectively from the video content; Using image-based rendering technology, based on pixel offset relationship information between character images at adjacent real perspectives at the same time point, character images at corresponding time points are generated for multiple intermediate perspectives between the adjacent real perspectives; A multi-perspective video is generated according to the multiple real perspectives and the character images of the multiple intermediate perspectives at multiple time points, so as to display the corresponding dynamic digital human image through the multi-perspective video. A method for displaying clothing information based on a dynamic digital human image, characterized by comprising: In response to a request for displaying the target clothing through a dynamic digital human, a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video are obtained, wherein the multi-view video is used to display the process of the target person displaying the target clothing when wearing the target clothing from multiple perspectives; The multi-view video is matched to the virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous viewpoint switching. A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented. An electronic device, comprising: one or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 11 are executed.