RU2844015C1 - Method and system for generating three-dimensional scenes - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: method for volumetric rendering of three-dimensional scenes, including training a predetermined machine learning algorithm from a plurality of machine learning algorithms to determine the boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene, wherein training includes: receiving from a predetermined camera of a plurality of cameras a sequence of training two-dimensional images, formation based on training two-dimensional images sequences, formation for a given training object from a plurality of interpenetrating training objects on a given corrected training two-dimensional image from a sequence of corrected training two-dimensional images of the SMPL posture estimate, obtaining, for a given training object from a plurality of interpenetrating training objects, a canonical pose SMPL containing canonical vertices which form the surface of the training object in a predetermined pose.

EFFECT: high accuracy of determining boundaries of objects when forming a high-resolution three-dimensional scene.

20 cl, 10 dwg

Description

Region techniques, To which refers to invention

[001] The present technology relates to the field of video processing in general and, in particular, to methods and systems for processing video content for the purpose of generating three-dimensional scenes.

Level techniques

[002] Generating three-dimensional scenes from image sequences, such as those forming video content, has a wide range of applications in many fields, such as generating three-dimensional maps for autonomous vehicles, navigation in robotics, and creating content for virtual or augmented reality applications.

[003] In general, video content representing objects displayed in a three-dimensional scene is recorded using a camera. Such video content is then analyzed to extract objects in it and to create three-dimensional models of these objects and the space surrounding them using volume rendering techniques.

[004] A portion of the video content may represent scenes containing objects that closely interact with each other (also referred to herein as "interpenetrating objects"), i.e., at least partially overlap or cover each other. For example, the video content may represent a boxing match or a football match, the participants of which may touch each other, for example, with their limbs, forming contact surfaces between them. With such close interaction of objects in the video content, determining the boundaries between them in order to track the movement of the overlapped and/or covered parts of the objects and form a realistic three-dimensional scene based on them may be a complex task.

[005] A number of methods have been proposed in the known technical solutions to solve the above technical problem.

[006] In the paper by Guo et al., “Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition,” Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, a method for generating human avatars based on monocular videos shot in the wild is disclosed. In particular, this method makes it possible to solve the problems of scene decomposition and surface reconstruction directly in 3D space by jointly modeling a person and a background in a scene with parameterization on two separate neural fields. To this end, the method involves determining a time-consistent representation of a person in a canonical space and globally optimizing the background model, the canonical shape and texture of the human body, and the parameters of the human pose in each frame. The volume rendering fine-grained sampling strategy and new targets provide clear separation between the dynamic human body and the static background, resulting in detailed and reliable 3D reconstructions of the human body.

[007] In the paper by Qing et al., “Novel View Synthesis of Human Interactions From Sparse Multi-view Videos,” ACM SIGGRAPH 2022 Conference Proceedings, July 2022, a system for generating free-view video images of human performers using a very sparse RGB camera system is disclosed. This system reconstructs a multi-layer neural representation of a dynamic scene with multiple participants from multi-view videos, each layer of which represents a moving object or a static background.

[008] In the paper by Shuo et al., “Fast Virtual View Synthesis for an 8K 3D Light-Field Display Based on Cutoff-Nerf and 3D Voxel Rendering,” Optics Express Vol. 30, Issue 24, pp. 44201-44217, November 2022, a two-step virtual view synthesis method based on cutoff-NeRF neural network and 3D voxel rendering is disclosed, which enables fast synthesis of novel high-density, smooth parallax, and 7680×4320 resolution 3D views for a 3D light field display. In the first stage, we propose to use the image-based cutoff-NeRF neural network, which provides an implicit representation of the scene content distribution and improves the quality of the virtual view. In the second stage, we propose to use the voxel-based 3D image construction and coding algorithm, which provides a quantitative estimate of the scene content distribution obtained as a result of training the cutoff-NeRF neural network, to quickly construct high-resolution virtual views and output high-resolution 3D images.

Disclosure inventions

[009] The aim of the present technology is to eliminate at least some of the disadvantages of known technical solutions.

[010] The developers of the present technology have determined that the boundary between objects in each image from a sequence of images of video content, used in the formation of a three-dimensional scene, can be determined using specially trained neural networks. In accordance with at least some non-limiting embodiments of the present technology, the neural network is trained to calculate the value of the signed distance field (SDF) constructed around an object previously associated with the given neural network, for a given point of a given image from a sequence of images.

[011] The signed field of distances around an object in a given image is defined in such a way that the value of the signed field of distances at each point of the image located inside the contour of this object is equal to the distance from this point to the contour of the object, taken with a minus sign. And vice versa, the value of the signed field of distances at each point located outside the contour of the object is equal to the distance from this point to the contour of the object, taken with a plus sign.

[012] Thus, when several neural networks are trained together, each of them is trained to calculate the value of the signed field of distances for a given point in relation to its own object previously associated with this network. In addition, for a more accurate determination of the boundary between at least two interpenetrating objects, the developers of this technology have developed a special loss function capable of assigning a penalty to identical predictions of neural networks.

[013] This allows to minimize or even eliminate the probability of two neural networks calculating the same negative value of the signed distance field for a given point (i.e. predicting the location of this point within the contours of objects previously associated with these networks), which ensures an accurate determination of the contour of the object in which the given point is located. Accordingly, the points in the image determined in this way allow to more accurately determine the boundary between a pair of interpenetrating objects and, as a result, to build a more accurate and realistic three-dimensional scene.

[014] The developers of this technology have also developed a method for determining, based on the values of the sign field of distances obtained by neural networks trained in the manner described above, the color values for interpenetrating objects, which are subsequently used in the process of volumetric rendering of a three-dimensional scene.

[015] In particular, in accordance with a first broad aspect of the present technology, a computer method for volume rendering of three-dimensional scenes is implemented. The method involves training a given machine-learning algorithm (MLA) from a plurality of machine-learning algorithms to determine the boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene. The training involves receiving from a given camera from a plurality of cameras a sequence of training two-dimensional images representing a plurality of interpenetrating training objects, forming on the basis of the sequences of training two-dimensional images obtained from the plurality of cameras a sequence of corrected training two-dimensional images, forming for a given training object from the plurality of interpenetrating training objects on a given corrected training two-dimensional image from the sequence of corrected training two-dimensional images a pose estimate according to a linear parametric model of the surface of a human body (SMPL, Skinned Multi-Person Linear) containing pose vertices forming the surface of the training object in the corresponding pose on the corrected training two-dimensional image, and an SMPL pose vector containing the values of a plurality of SMPL pose parameters characterizing the pose of the training object on the corrected training two-dimensional image, obtaining for a given training object from the plurality of interpenetrating training objects a canonical SMPL pose containing the canonical vertices forming the surface of the training object in a predetermined pose, constructing a closed three-dimensional space around the SMPL pose estimates associated with a plurality of interpenetrating training objects, forming on an inner surface of the closed three-dimensional space a plurality of viewpoints, each of which is oriented toward the center of the closed three-dimensional space, constructing from each viewpoint of the plurality of viewpoints a plurality of rays through the SMPL pose estimates for the plurality of interpenetrating training objects, determining for a given point on a given ray a canonical point in the canonical SMPL pose associated with each of the plurality of interpenetrating training objects, forming for a given machine learning algorithm from the plurality of machine learning algorithms a training data set containing a plurality of training digital objects, wherein the given training digital object for a given point on a given ray contains (a) coordinates of the canonical point associated with this training object, and (b) a SMPL pose vector associated with this training object, jointly training the plurality of machine learning algorithms using the training data sets to determine the boundary of the given object from the plurality of interpenetrating objects by transmitting the training digital objects associated with the given point to each machine learning algorithm of the plurality of machine learning algorithms, whereby each of the plurality of machine learning algorithms generates a predicted value of a signed distance field for a canonical point, and applying a loss function for the signed distance field capable of assigning a penalty to the predicted value of the signed distance field generated by a given machine learning algorithm for a given point if the predicted value of the signed distance field generated by this machine learning algorithm is equal to the predicted value of the signed distance field generated by another machine learning algorithm of the plurality of machine learning algorithms.

[016] In some embodiments of the method, the closed three-dimensional space is a sphere constructed around the training three-dimensional scene, and each of the plurality of viewpoints is located on the inner surface of this sphere.

[017] In some embodiments of the method, the plurality of viewpoints represent two opposite viewpoints oriented in the direction of the center of the closed three-dimensional space.

[018] In some embodiments of the method, a plurality of rays emanating from a given viewpoint from a plurality of viewpoints are spaced from each other by the same distance.

[019] In some embodiments of the method, it additionally provides for finding on a given ray a set of points containing a given number of points uniformly distributed along this ray.

[020] In some embodiments of the method, the determination of the canonical point is carried out using a skinning algorithm based on linear blending (Linear Blend Skinning) using the SMPL model.

[021] In some embodiments of the method, the following formula is used in this algorithm:

where x ^c is the canonical point associated with the training object;

x ^d - a given point on the ray;

B _i is the transformation matrix associated with the joint j _i of the training object and formed based on the SMPL pose estimate associated with the training object;

n _b - the number of joints in the training object;

w _i ^d is a function characterizing the distribution of weights in the skinning process, determined based on the coordinates of the SMPL pose estimate vertex associated with the training object, closest to the given point.

[022] In some embodiments of the method, the loss function for the signed distance field is expressed by the following formula:

,

where P is the number of training objects in the set of interpenetrating training objects;

- the number of unique pairs of training objects in a set of interpenetrating training objects;

s _i ^k - the predicted value of the signed distance field for a point, calculated by the first algorithm from the set of machine learning algorithms associated with the first training object from the set of interpenetrating training objects;

s _i ^p - the predicted value of the signed distance field for a point, calculated by the second algorithm from the set of machine learning algorithms associated with the second training object from the set of interpenetrating training objects.

[023] In some embodiments of the method, the transmission of training digital objects associated with a given point to each machine learning algorithm of a plurality of machine learning algorithms additionally initiates the formation by each of the plurality of machine learning algorithms of a corresponding vector representation of a signed distance field for a canonical point, and the training of the plurality of machine learning algorithms involves joint training of the plurality of machine learning algorithms and a second plurality of machine learning algorithms for the purpose of determining color values for a plurality of interpenetrating objects displayed in a three-dimensional scene. Training the second plurality of machine learning algorithms involves receiving, at the current training iteration, from each of the plurality of machine learning algorithms a predicted value of the signed distance field for a canonical point associated with a training object, determining, based on the predicted values of the signed distance field, current predicted density values for the corresponding canonical points, forming for another machine learning algorithm from the second plurality of machine learning algorithms another training data set containing another plurality of training digital objects, wherein the specified training digital object for a specified point on a specified ray contains (a) the coordinates of a canonical point associated with a training object associated with the other machine learning algorithm, (b) a vector representation of the signed distance field for a canonical point associated with the training object, and (c) a label representing the color value of a pixel on the adjusted training two-dimensional image associated with the specified ray, transmitting training digital objects associated with a specified point from each other training data set to each machine learning algorithm from the second plurality of machine learning algorithms, as a result of which each of the second plurality machine learning algorithms generate a predicted color value for a canonical point associated with a training object, determining, based on the predicted color values generated by each of the second plurality of machine learning algorithms and the current predicted density values generated by the plurality of machine learning algorithms for each point of a given ray, an intermediate integral color value for the ray at the current training iteration, and applying a loss function for color capable of assigning a penalty to the intermediate integral color value if this value differs from the color value from the corresponding label.

[024] In some embodiments of the method, determining the intermediate integral color value involves determining the intermediate integral color value using the following formula:

,

where c _i ^p is the predicted color value for the canonical point associated with object p , generated by the machine learning algorithm from the second set of machine learning algorithms;

w _i ^p - the weight value determined for the predicted color value based on the current predicted density values;

bg _r - background color associated with the beam.

[025] In some embodiments of the method, it additionally provides for determining the weight value using the following formula:

,

where Δ x _i is the length of the segment between a given point and the next point on the ray;

σ _i ^p is the current predicted density value generated by a machine learning algorithm from a set of machine learning algorithms associated with the training object, based on the corresponding value of the signed distance field.

[026] In some embodiments of the method, determining the current predicted density values for the canonical point involves applying a normalized integral Laplace distribution function to each of the predicted values of the signed distance field.

[027] In some embodiments of the method, before forming another training data set, it additionally provides for selecting from the set of points of the given beam a set of points for training a second set of machine learning algorithms in the areas of maximum density of the training object.

[028] In some embodiments of the method, such selection involves the use of a significance sampling algorithm capable of forming points on the beam that represent each of the plurality of interpenetrating training objects through which the beam passes.

[029] In some embodiments of the method, it further provides for combining beam points representing each of a plurality of interpenetrating training objects through which the beam passes.

[030] In some embodiments of the method, the importance sampling algorithm is an opacity function.

[031] In some embodiments of the method, it further provides for the use of a plurality of machine learning algorithms to determine the boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene. Such application involves obtaining from a given camera from a plurality of cameras a working (i.e. related to the process of using the machine learning algorithm) sequence of two-dimensional images representing a plurality of interpenetrating objects, forming on the basis of the working sequences of two-dimensional images obtained from a plurality of cameras, a sequence of working corrected two-dimensional images, forming for a given object from a plurality of interpenetrating objects on each working corrected two-dimensional image from the sequence of working corrected two-dimensional images a working estimate of the pose SMPL, containing working pose vertices forming the surface of the object in the corresponding pose on the working corrected two-dimensional image, and a working pose vector SMPL, containing the values of a plurality of pose parameters SMPL, characterizing the pose of the object on the working corrected two-dimensional image, obtaining for a given object from a plurality of interpenetrating objects a working canonical pose SMPL, containing working canonical vertices forming the surface of the given object in a predetermined pose, constructing a closed three-dimensional space containing a plurality of viewpoints formed on its an inner surface, around working SMPL pose estimates associated with a plurality of interpenetrating objects, constructing from each viewpoint of the plurality of viewpoints a plurality of rays through the working SMPL pose estimates, determining for a given point on a given ray a working canonical point for the working canonical pose of the SMPL associated with each of the plurality of interpenetrating objects, generating for a given machine learning algorithm associated with an object from the plurality of interpenetrating objects a given working digital object containing (a) coordinates of the working canonical point associated with the corresponding object, and (b) a SMPL pose vector associated with the corresponding object, transmitting the working digital object to a given machine learning algorithm from the plurality of machine learning algorithms, as a result of which this machine learning algorithm determines a working value of a signed distance field for the canonical point, and determining whether the surface of the given object passes through the given point or not, based on the working values of the signed distance field at the canonical point generated by the plurality of machine learning algorithms.

[032] In some embodiments of the method, the transfer of a given working digital object to a given machine learning algorithm from a plurality of machine learning algorithms additionally initiates the formation by this machine learning algorithm of a working vector representation of a signed distance field for a canonical point associated with the corresponding object, and also additionally provides for the use of a second plurality of machine learning algorithms in order to determine a color value for each of the plurality of interpenetrating objects by determining, based on all working values of the signed distance field at the canonical point generated by the plurality of machine learning algorithms, the density values at this canonical point, the formation of a working digital object for determining a color containing (a) the coordinates of the working canonical point associated with the corresponding object, and (b) the working vector representation of the signed distance field for this canonical point, the transfer to another machine learning algorithm from the second plurality of machine learning algorithms associated with the corresponding object, the working color digital object, as a result of which this other machine learning algorithm generates a color value for the canonical point of this object, the determination based on the density values generated by the plurality of machine learning algorithms and the color values generated by the second plurality of algorithms machine learning for the canonical point, the integral color value for each ray, and the application of the integral color value for volumetric rendering of interpenetrating objects in a 3D scene.

[033] In accordance with the second broad aspect of the present technology, a server for volume rendering of three-dimensional scenes is implemented. The server comprises at least one processor and at least one physical memory storing executable commands, the execution of which by at least one processor on the server ensures training of a given machine learning algorithm from a plurality of machine learning algorithms to determine the boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene by receiving from a given camera from a plurality of cameras a sequence of training two-dimensional images representing a plurality of interpenetrating training objects, forming, on the basis of the sequences of training two-dimensional images obtained from the plurality of cameras, a sequence of corrected training two-dimensional images, forming for a given training object from a plurality of interpenetrating training objects on a given corrected training two-dimensional image from the sequence of corrected training two-dimensional images an SMPL pose estimate containing pose vertices forming the surface of the training object in the corresponding pose on the corrected training two-dimensional image, and an SMPL pose vector containing the values of a plurality of SMPL pose parameters characterizing the pose of the training object on the corrected a training two-dimensional image, obtaining for a given training object from a plurality of interpenetrating training objects a canonical pose SMPL containing canonical vertices forming the surface of the training object in a predetermined pose, constructing a closed three-dimensional space around the SMPL pose estimates associated with the plurality of interpenetrating training objects, forming on the inner surface of the closed three-dimensional space a plurality of viewpoints, each of which is oriented in the direction of the center of the closed three-dimensional space, constructing from each viewpoint from the plurality of viewpoints a plurality of rays through the SMPL pose estimates for the plurality of interpenetrating training objects, determining for a given point on a given ray a canonical point in the canonical pose SMPL associated with each of the plurality of interpenetrating training objects, forming for a given machine learning algorithm from a plurality of machine learning algorithms a training data set containing a plurality of training digital objects, wherein the given training digital object for a given point on a given ray contains (a) the coordinates of the canonical point associated with this training object, and (b) a pose vector SMPL, associated with this training object, jointly training a plurality of machine learning algorithms using training data sets to determine the boundary of a given object from a plurality of interpenetrating objects by transmitting training digital objects associated with a given point to each machine learning algorithm of the plurality of machine learning algorithms, as a result of which each of the plurality of machine learning algorithms generates a predicted value of a signed distance field for a canonical point, and applying a loss function for the signed distance field capable of assigning a penalty to the predicted value of the signed distance field generated by the machine learning algorithm for the given point, if the predicted value of the signed distance field generated by this machine learning algorithm is equal to the predicted value of the signed distance field generated by another machine learning algorithm of the plurality of machine learning algorithms.

[034] In some embodiments of the server, the transmission of training digital objects associated with a given point to each of the plurality of machine learning algorithms further initiates the formation by each of the plurality of machine learning algorithms of a vector representation of a signed distance field for the canonical point, and the provision by the training server of the plurality of machine learning algorithms during the execution of executable commands provides for joint training of the plurality of machine learning algorithms and a second plurality of machine learning algorithms for the purpose of determining color values for a given object from a plurality of interpenetrating objects displayed in a three-dimensional scene by receiving at the current iteration of training from each of the plurality of machine learning algorithms a predicted value of a signed distance field for a canonical point associated with the training object, determining based on the predicted values of the signed distance field current predicted values of the density for the corresponding canonical points, forming for another machine learning algorithm from the second plurality of machine learning algorithms another training data set containing another plurality of training digital objects, wherein the given training digital object for a given point on a given ray contains (a) coordinates of a canonical point associated with a training object associated with another machine learning algorithm, (b) a vector representation of a signed distance field for a canonical point associated with the corresponding training object, and (c) a label representing a color value of a pixel in a corrected training two-dimensional image associated with a given ray, transmitting training digital objects associated with a given point from each other training data set to each machine learning algorithm of a second plurality of machine learning algorithms, as a result of which each of the second plurality of machine learning algorithms generates a predicted color value for the canonical point associated with the training object, determining, based on the predicted color values generated by each of the second plurality of machine learning algorithms and the current predicted density values generated by the plurality of machine learning algorithms for each point of the given ray, an intermediate integrated color value for the given ray at the current training iteration, and applying a loss function for a color capable of assigning a penalty to the intermediate integrated color value if this value differs from the color value from the corresponding label.

[035] In some embodiments of the server, the executable commands further provide for applying on the server a plurality of machine learning algorithms to determine a boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene by receiving from a given camera of a plurality of cameras a working sequence of two-dimensional images representing the plurality of interpenetrating objects, generating, based on the working sequences of two-dimensional images received from the plurality of cameras, a sequence of working corrected two-dimensional images, generating for a given object from the plurality of interpenetrating objects on each working corrected two-dimensional image from the sequence of working corrected two-dimensional images a working pose estimate SMPL containing working pose vertices forming the surface of the object in the corresponding pose on the working corrected two-dimensional image, and a working pose vector SMPL containing the values of a plurality of pose parameters SMPL characterizing the pose of the object on the working corrected two-dimensional image, obtaining for a given object from the plurality of interpenetrating objects a working canonical pose SMPL containing the working canonical vertices forming the surface of the object in a predetermined pose, constructing a closed three-dimensional space containing a plurality of viewpoints formed on its inner surface around the working SMPL pose estimates associated with a plurality of interpenetrating objects, constructing from each viewpoint of the plurality of viewpoints a plurality of rays through the working SMPL pose estimates, determining for a given point on a given ray a working canonical point for the working canonical pose of the SMPL associated with each of the plurality of interpenetrating objects, generating for a given machine learning algorithm associated with a corresponding object of the plurality of interpenetrating objects a given working digital object containing (a) coordinates of the working canonical point associated with the corresponding object, and (b) a SMPL pose vector associated with the corresponding object, transmitting the working digital object to a given machine learning algorithm from a plurality of machine learning algorithms, as a result of which this machine learning algorithm determines a working value of a signed distance field for the canonical point, and determining whether the surface of the given object passes through the given point or not, based on the working values of the signed distance field at the canonical point formed by the plurality of machine learning algorithms.

[036] As used herein, a "server" is a computer program running on appropriate hardware that is capable of receiving requests (e.g., from client devices) over a network and of executing or causing these requests to be executed. Such hardware may be implemented as a single physical computer or a single physical computer system, which is not essential to the present technology. As used herein, the term "server" does not imply that any particular task or all tasks (e.g., received commands or requests) are received, executed, or started on the same server (i.e., by the same software and/or hardware), but rather that any number of software or hardware may participate in receiving, transmitting, executing, or causing the execution of any tasks or requests, or the results of any tasks or requests, and that all of these software and hardware may represent a single server or multiple servers, with the term "at least one server" including both of these cases.

[037] In the context of the present description, a "client device" is any computer hardware capable of running software suitable for performing a given task. Thus, non-limiting examples of client devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network equipment such as routers, switches and gateways. It should be noted that a device acting as a client in this context does not lose the ability to act as a server for other client devices. The use of the expression "client device" does not exclude the use of multiple client devices to receive, transmit, perform or initiate the performance of any tasks or requests, the results of any tasks or requests, or the steps of any method described herein.

[038] As used herein, the term "database" means any structured collection of data, regardless of its specific structure, software for managing the database, and computer hardware for storing such data, using it, or otherwise enabling its use. A database may be hosted on the same hardware as the process that enables the storage or use of the information stored in the database, or on separate hardware, such as a dedicated server or group of servers.

[039] As used herein, the term "information" includes information of any kind or type capable of being stored in a database. Such information includes, but is not limited to, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (reviews, comments, questions, messages, etc.), documents, spreadsheets, word lists, etc.

[040] In the context of this description, the expression "component" means software (in the context of specific hardware) necessary and sufficient to perform the specific functions referred to.

[041] In the context of the present description, the expression "computer storage medium" is intended to refer to media of any kind and type, including RAM, ROM, disks (CD-ROM, DVD, floppy disks, hard disks, etc.), USB drives, solid state drives, tape drives, etc.

[042] As used herein, the numerals "first," "second," "third," etc., serve merely to distinguish between the nouns to which they refer, and do not describe any particular relationships between those nouns. Thus, it should be understood that, for example, the terms "first server" and "third server" do not imply the existence of any particular order, type, chronology, hierarchy, or ranking of servers, and the use of these terms (in and of themselves) does not necessarily imply the presence of any "second server" in any particular situation. Furthermore, as occurs in other contexts herein, references to a "first" element and a "second" element do not exclude the possibility that the two elements may in fact be the same element. For example, in some cases the “first” and “second” servers may represent the same software and/or hardware, and in other cases, different software and/or hardware.

[043] Each embodiment of the present technology relates to at least one of the above-mentioned objectives and/or one of the above-mentioned aspects, but not necessarily to all of them. It should be understood that some aspects of the present technology associated with an attempt to achieve the above-mentioned objective may not correspond to this objective and/or may correspond to other objectives not explicitly described herein.

[044] Additional and/or alternative features, aspects and advantages of embodiments of the present technology are contained in the following description, in the attached drawings and in the claims.

Brief description drawings

[045] For a more complete understanding of the present technology, its aspects and additional features, reference should be made to the following description, which should be used in conjunction with the accompanying drawings.

[046] Fig. 1 is a schematic representation of an example of a computer system suitable for implementing certain non-limiting embodiments of systems and/or methods in accordance with the present technology.

[047] Fig. 2 illustrates a networked computing environment suitable for use with certain non-limiting embodiments of the present technology.

[048] Fig. 3 is a schematic representation of a scene to be rendered by a server within the networked computing environment shown in Fig. 2, in accordance with some non-limiting embodiments of the present technology.

[049] Fig. 4 is a schematic diagram of a training scene in a system for generating a training data set for training a plurality of machine learning algorithms for determining density and a plurality of machine learning algorithms for determining color, executed on a server within the networked computer environment shown in Fig. 2, for the purpose of rendering the scene shown in Fig. 3, in accordance with some non-limiting embodiments of the present technology.

[050] Fig. 5 is a schematic diagram of a training three-dimensional scene comprising pose estimates of training objects according to the SMPL model for the training scene shown in Fig. 4, displayed based on two-dimensional image data representing the training objects and generated in the system shown in Fig. 4, in accordance with some non-limiting embodiments of the present technology.

[051] Fig. 6 is a schematic representation of the canonical pose SMPL of the training object shown in Fig. 4 within a signed distance field grid in accordance with some non-limiting embodiments of the present technology.

[052] Fig. 7 is a schematic representation of the scene shown in Fig. 3 in a system for generating working digital objects for a plurality of machine learning algorithms for determining density and color, executed on a server within the networked computer environment shown in Fig. 2, in accordance with some non-limiting embodiments of the present technology.

[053] Fig. 8 is a schematic representation of a three-dimensional scene generated from image data representing the scene shown in Fig. 3 in the system shown in Fig. 7 in accordance with some non-limiting embodiments of the present technology.

[054] Fig. 9 is a schematic representation of the three-dimensional scene shown in Fig. 8, including rendered three-dimensional models of objects that are present in the scene shown in Fig. 3, in accordance with some non-limiting embodiments of the present technology.

[055] Fig. 10 is a flow chart of a method for training a machine learning algorithm from a plurality of machine learning algorithms for determining density and color running on a server that is part of the networked computing environment shown in Fig. 2 for rendering the three-dimensional scene shown in Fig. 9, in accordance with some non-limiting embodiments of the present technology.

Implementation inventions

[056] The examples and conventional language provided herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically provided examples and conditions. It is obvious that those skilled in the art are capable of devising various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its spirit and scope.

[057] Furthermore, in order to facilitate better understanding, the following description may contain simplified embodiments of the present technology. Those skilled in the art will understand that various embodiments of the present technology may be significantly more complex.

[058] In some cases, supposedly useful examples of modifications of the present technology are also given. They are intended only to facilitate understanding and also do not define the scope or boundaries of the present technology. The presented list of modifications is not exhaustive, and a person skilled in the art can develop other modifications within the scope of the present technology. In addition, if in some cases modifications are not described, this does not mean that they are impossible and/or that the description contains the only possible embodiment of a particular element of the present technology.

[059] In this case, the description of the principles, aspects and embodiments of the present technology, as well as their specific examples, are intended to cover their structural and functional equivalents, regardless of whether they are currently known or will be developed in the future. For example, those skilled in the art should understand that all block diagrams provided herein correspond to conceptual representations of illustrative circuit diagrams implementing the principles of the present technology. It should also be understood that all block diagrams, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that can be represented on a machine-readable physical information carrier and can be executed by a computer or processor, regardless of whether such a computer or processor is shown explicitly or not.

[060] The functions of the various elements shown in the drawings, including all functional blocks designated as a "processor" or "graphics processor", may be implemented using specialized hardware, as well as hardware capable of supporting the execution of corresponding software. If a processor is used, these functions may be performed by one dedicated processor, one shared processor, and/or several separate processors, some of which may be used together. In some embodiments of the present technology, the processor may be a general-purpose processor, such as a central processor, or a processor dedicated to solving a specific task, such as a graphics processor. In addition, explicit use of the term "processor" or "controller" shall not be construed as referring solely to hardware capable of executing software, and may include, but is not limited to, hardware such as a digital signal processor (DSP), a network processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a ROM for storing software, RAM, and/or non-volatile memory. Other hardware, general purpose and/or custom, may also be included.

[061] Software modules or simply modules, the implementation of which is supposed to be based on software, may be represented here as any combination of flow chart elements or other elements indicating the execution of process steps and/or containing a text description. Such modules may be implemented using hardware, shown explicitly or implied.

[062] Taking into account the above principles, some non-limiting examples are discussed below to illustrate various embodiments of aspects of the present technology.

Computer system

[063] Fig. 1 shows a computer system 100 suitable for use in some embodiments of the present technology. The computer system 100 includes various hardware elements, including one or more single-core or multi-core processors, collectively represented by a processor 110, a graphics processor 111, a solid-state drive 120, a random access memory 130, a display interface 140, and an input/output interface 150.

[064] Communication between elements of the computer system 100 may be accomplished via one or more internal and/or external buses 160 (such as a PCI bus, a USB bus, an IEEE 1394 FireWire bus, a SCSI bus, a Serial-ATA bus, etc.) to which various hardware elements are electronically connected.

[065] The input/output interface 150 may be coupled to a touch screen 190 and/or to one or more internal and/or external buses 160. The touch screen 190 may be part of a display. In some embodiments, the touch screen 190 is a display. In the embodiments shown in Fig. 1, the touch screen 190 comprises touch means 194 (e.g., touch elements embedded in a layer of the display and providing for registration of physical interaction between a user and the display) and a touch input/output controller 192 providing for communication with the display interface 140 and/or one or more internal and/or external buses 160. In some embodiments, the input/output interface 150 may be coupled to a keyboard (not shown), a mouse (not shown), or a touch pad (not shown), which provide for user interaction with the computer system 100 in addition to or instead of the touch screen 190. In some embodiments, the computer system 100 may include one or more microphones (not shown). The microphones provide for recording audio, such as user speech. The user speech may be converted into commands for controlling the computer system 100.

[066] It should be noted that in some non-limiting embodiments of the present technology, some elements of the computer system 100 may be missing. For example, the touch screen 190 may be missing, especially in cases (but not limited to) where the computer system is implemented as a smart speaker.

[067] In accordance with embodiments of the present technology, software instructions suitable for loading into random access memory 130 and execution by processor 110 and/or graphics processor 111 are stored on solid state drive 120. For example, software instructions may be part of a library or application.

Network computer Wednesday

[068] Fig. 2 is a schematic diagram of a networked computing environment 200 suitable for use with some embodiments of systems and/or methods in accordance with the present technology. In some non-limiting embodiments of the present technology, the networked computing environment 200 is capable of generating three-dimensional scenes.

[069] In this regard, in some non-limiting embodiments of the present technology, the networked computer environment 200 comprises a server 202 connected to an electronic device 204 via a communication network 208. In non-limiting embodiments of the present technology, the electronic device 204 may be connected to a user 206.

[070] In some non-limiting embodiments of the present technology, server 202 is implemented as a conventional computer server and may include some or all of the elements of computer system 100 shown in Fig. 1. In one non-limiting example, server 202 is implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system, but it may also be implemented on any other suitable types of hardware, software, and/or firmware, or a combination thereof. In the shown non-limiting embodiments of the present technology, server 202 is a single server. In other non-limiting embodiments of the present technology (not shown), the functions of server 202 may be distributed among multiple servers.

[071] In addition, the electronic device 204 may be any computer hardware capable of running software suitable for solving the task at hand, and may also include some or all of the elements of the computer system 100 shown in Fig. 1. Thus, non-limiting examples of the electronic device 204 include personal computers (desktop computers, laptops, netbooks, etc.), smartphones, and tablets.

[072] In some non-limiting embodiments of the present technology, the electronic device 204 may be a head-mounted display (also referred to herein as a head-mounted display). Generally, a head-mounted display is an electronic device that is designed to be worn over the eyes of a user 206 and has a display (also referred to herein as a "viewing window") located on the surface of the device facing the eyes and capable of displaying a variety of video content that may represent various simulated environments. For example, in various non-limiting embodiments of the present technology, the head-mounted display may be embedded in a helmet or goggles worn over the eyes of the user 206. Typically, the head-mounted display includes sensors capable of tracking movements of the body, head, and/or pupils of the eyes of the user 206 and may use data from such sensors to either adjust parameters of the current video content or to receive and play other video content, providing a simulated experience for the user 206.

[073] In addition, in some non-limiting embodiments of the present technology, the head-mounted device, when worn over the eyes of the user 206, is capable of completely blocking the field of view of the user 206, creating a simulated environment for the user by playing corresponding video content. In such embodiments, the head-mounted device may be referred to as a virtual reality (VR) head-mounted device. In other non-limiting embodiments of the present technology, the head-mounted device is capable of only partially blocking the field of view of the user 206 and playing video content in the viewing window of the head-mounted device such that it is superimposed on the real environment viewed by the user 206 at the same time. In such embodiments, the head-mounted device may be referred to as an augmented reality (AR) head-mounted device. Areas of application of the head electronic device may include, in particular, (a) video games, such as those in which the user 206 acts in the first person, (b) art, such as conducting virtual tours of museums, and (c) medicine, such as simulating surgical scenarios.

[074] Specific examples of a head-mounted display device include, but are not limited to, a Meta™ Quest™ head-mounted display device, an Amazon™ Oculus™ head-mounted display device, and an HTC™ Vive™ Pro head-mounted display device. In general, in such embodiments, the electronic device 204 is capable of displaying immersive virtual or augmented reality video content to the user 206.

[075] Furthermore, it is apparent that the electronic device 204 may be one of a plurality of electronic devices similar to the electronic device 204 that are connected to the server 202 for playing video content received from the server 202.

[076] In some non-limiting embodiments of the present technology, the video content displayed by the electronic device 204 is three-dimensional animated video content that includes a three-dimensional scene with animated three-dimensional models of a plurality of objects. There are no limitations on the types of objects that can be displayed in the three-dimensional scene and such objects can be various moving and static objects. The three-dimensional scene, for example, can represent a boxing match, and the objects in the three-dimensional scene can include boxers, a boxing ring, and at least a portion of the spectators surrounding the boxing ring. In another example, the three-dimensional scene can represent a football match, and the objects represented in the three-dimensional scene can include football players and a ball. In yet another example, the three-dimensional scene can represent animals in their natural habitat.

[077] According to some non-limiting embodiments of the present technology, the three-dimensional animated video content for playback on the electronic device 204 may be transmitted by the server 202. Thus, in some non-limiting embodiments of the present technology, the server 202 may be operated by an organization that produces some video content for distribution to end users, such as the user 206, via the communication network 208. In particular, in such embodiments, the server 202 may be used to create new and/or to store already existing three-dimensional animated video content and to transmit this video content to the electronic device 204, for example, upon request from it. The format of the video content transmitted by the server 202 is not limited, including, for example, MP4, MOV or F4V.

[078] Thus, in accordance with some non-limiting embodiments of the present technology, the electronic device 204 is capable of transmitting a three-dimensional animation request 212 to the server 202 over the communication network 208, and the server 202 is capable, in response thereto, of (a) finding three-dimensional animated video content 214 that meets the three-dimensional animation request 212 in the database 218, (b) compressing the three-dimensional animated video content 214 to form a compressed video data packet 216, and (c) transmitting the compressed video data packet 216 to the electronic device 204 for subsequent display of the three-dimensional animated video content 214 to the user 206.

[079] In some non-limiting embodiments of the present technology, the three-dimensional animated video content 214 may be generated by the server 202. However, in other non-limiting embodiments of the present technology, the three-dimensional animated video content 214 may be generated not by the server 202, but by some other third-party server, and transmitted to the server 202 for subsequent storage and distribution to users of the communication network 208, such as user 206.

[080] There are no limitations on the method for causing the electronic device 204 to send the three-dimensional animation request 212 to the server 202. In some non-limiting embodiments of the present technology, the three-dimensional animation request 212 for displaying the three-dimensional animated video content 214 may be sent by the user 206 directly, for example, using a corresponding graphical user interface control implemented on the electronic device 204. In such embodiments, the three-dimensional animation request 212, depending on the field of application of the electronic device 204, may indicate, for example, the three-dimensional animated video content 214 to be displayed from a catalog of existing content in the database 218. In other non-limiting embodiments of the present technology, the user 206 may send the three-dimensional animation request 212 implicitly. For example, a user 206 wearing an electronic device 204, which is the aforementioned head-mounted electronic device, may move his head, causing an image to be constructed of another portion of the currently viewed three-dimensional animated content.

[081] Fig. 3 is a schematic representation of a displayed scene 302 of three-dimensional animated video content 214 in accordance with some non-limiting embodiments of the present technology.

[082] It is obvious that in the presented example, the scene 302 to be reconstructed from the three-dimensional animated video content 214 contains two objects, namely the first object 301 and the second object 303, which in this example are boxers during a boxing match. At the same time, as indicated above, without deviating from the essence and scope of the present technology, scenes representing other events, such as a football match or a dance party, and containing more than two objects can also be reconstructed.

[083] According to some non-limiting embodiments of the present technology, the server 202 is capable of generating three-dimensional animated video content 214 representing the scene 302 based on two-dimensional video content representing it. As described in more detail below, the two-dimensional video content may include one or more sequences of two-dimensional images representing the scene 302 and taken from a corresponding viewpoint. To transform the two-dimensional video content into three-dimensional animated video content 214, the server 202, according to some non-limiting embodiments of the present technology, is capable of using a volumetric rendering algorithm. However, there are no limitations on the method for implementing the volumetric rendering algorithm. For example, in some non-limiting embodiments of the present technology, the volumetric rendering algorithm may be a volumetric ray casting algorithm, described in more detail below.

[084] One of the problems associated with the formation of three-dimensional animated video content based on two-dimensional video content is that the objects in the scene, in particular the first and second objects 301, 303, may have one or more overlapping regions, such as the overlapping region 305. In this regard, the first and second objects 301, 303 are referred to in this document as "interpenetrating objects". In other words, in the context of the present description, the term "interpenetrating objects" is used to denote objects that partially overlap or cover each other in the two-dimensional image representing them.

[085] For example, due to the presence of the overlap region 305, the server 202, in the process of creating three-dimensional animated video content 214, including three-dimensional models of the first and second objects 301, 303, may be unable to determine their actual boundaries and, as a result, may apply inappropriate textures, such as color values, to the three-dimensional model of the first and/or second objects 301, 303 within the overlap region 305. In particular, the inability of the server 202 to determine that the hand of the second object 303 covers the hand of the first object 301 may result in it generating three-dimensional animated video content 214 in which a portion of the hand of the second object 303 in the overlapping region 305 has the texture and color of a portion of the hand of the first object 301, that is, the texture and color of the boxing glove of the first object 301. In other words, the inability to accurately determine the boundaries of interpenetrating objects in the scene 302 may result in the three-dimensional animated video content 214 being displayed with errors and unrealistically, which may negatively affect the experience of the user 206 viewing the three-dimensional animated video content 214.

[086] To solve the identified technical problem, the developers of the present technology have implemented the methods and systems described herein, which are oriented towards training a plurality 210 of machine learning algorithms for determining density in order to find the value of the signed distance field (SDF) for the vertices forming the surfaces of interpenetrating objects in the displayed three-dimensional scene. To train the plurality 210 of machine learning algorithms for determining density, the server 202, in accordance with non-limiting embodiments of the present technology, is capable of using a special loss function capable of assigning a penalty to the same predictions of the plurality 210 of machine learning algorithms for determining density. According to some non-limiting embodiments of the present technology, a given machine learning algorithm of the plurality of machine learning algorithms is associated with a corresponding object of the interpenetrating objects in the displayed three-dimensional scene 302. In the current example, two objects are present in the scene 302, namely the first and second objects 301, 303, so the plurality 210 of machine learning algorithms for determining the density includes two machine learning algorithms, where (a) the first machine learning algorithm of the plurality 210 of machine learning algorithms for determining the density is capable of determining the value of the signed distance field for a given point within the first object 301 and (b) the second machine learning algorithm of the plurality 210 of machine learning algorithms for determining the density is capable of determining the value of the signed distance field for the same point within the second object 303.

[087] In other words, the loss function is capable of assigning a penalty to the predictions of the first and second machine learning algorithms if they simultaneously characterize a given point as being within the first and second objects 301, 303. To this end, as described in detail below, in some non-limiting embodiments of the present technology, the server 202 is capable of training a plurality 210 of machine learning algorithms for determining density jointly, that is, at the same time.

[088] According to some non-limiting embodiments of the present technology, a given machine learning algorithm of the plurality 210 machine learning algorithms for determining density may be implemented as a neural network. In a specific non-limiting example, a given machine learning algorithm of the plurality 210 machine learning algorithms for determining density may be a Multilayer Perceptron (MLP). In this example, each machine learning algorithm for determining density may be implemented as described in detail in the paper by Yariv et al., “Volume Rendering of Neural Implicit Surfaces,” published on arxiv.org in December 2021, the contents of which are incorporated herein by reference in their entirety.

[089] In addition, in some non-limiting embodiments of the present technology, the server 202, based on the values of the signed distance field determined by the plurality 210 machine learning algorithms for determining density, is capable of determining density values for a given point, which can be used to train the plurality 215 machine learning algorithms for determining color in order to determine color values for each of the first and second objects 301, 303. In other words, each of the plurality 215 machine learning algorithms for determining color, like the plurality 210 machine learning algorithms for determining density, is associated with an object in the displayed scene 302. In the illustrated example, the plurality 215 machine learning algorithms includes two machine learning algorithms for determining color: (a) a first machine learning algorithm of the plurality 215 machine learning algorithms for determining color is capable of determining a color value for a given point within the first object 301 and (b) a second machine learning algorithm of the plurality 215 machine learning algorithms for determining color is capable of determining a color value for the same point within the second object 303.

[090] According to some non-limiting embodiments of the present technology, the server 202 is also capable of jointly training a plurality 215 machine learning algorithms for determining a color. In addition, in some non-limiting embodiments of the present technology, the server 202 is capable of jointly training a plurality 210 machine learning algorithms for determining a density and a plurality 215 machine learning algorithms for determining a color. In particular, as described in detail below, the server 202 is capable of generating current input training data for a given machine learning algorithm for determining a color based on current output training data of a machine learning algorithm for determining a density.

[091] Similar to the machine learning algorithms for density determination, a machine learning algorithm from the set of machine learning algorithms for color determination can be implemented as a neural network.

[092] It can be stated that, in general, the server 202 performs two processes with respect to the plurality 210 of machine learning algorithms for determining the density and the plurality 215 of machine learning algorithms for determining the color. The first of the two processes is a training process in which the server 202 trains the plurality 210 of machine learning algorithms 215 for determining the density and color based on a repeated training data set in order to determine the values of the signed field of distances and color for a given point, as described below with reference to Figs. 4-6. The second process is a working process (use process), in which the server 202 executes the trained plurality 210 of machine learning algorithms 215 for determining the density and color in order to determine the values of the signed field of distances and color for working interpenetrating objects, such as the first and second objects 301, 303, as described below with reference to Figs. 7-9.

[093] Based on the values of the signed distance field determined in this manner, the server 202, according to some non-limiting embodiments of the present technology, can more accurately determine the boundaries between the first and second objects 301, 303 within the overlapping region 305. This, in turn, allows for more accurate training of the plurality 215 machine learning algorithms for determining color to determine color values for the first and second objects 301, 303, which makes it possible to generate more realistic three-dimensional images of these objects.

Net connections

[094] In some non-limiting embodiments of the present technology, the communication network 208 is the Internet. In other non-limiting embodiments of the present technology, the communication network 208 may be implemented as any suitable local area network (LAN), wide area network (WAN), private communication network, etc. It should be understood that the embodiments of the communication network 208 are given for illustrative purposes only. The method for implementing the communication links (not separately indicated) between the server 202, the electronic device 204 and the communication network 208 depends, among other things, on the implementation of the server 202 and the electronic device 204. For example, it can be noted that, in particular, in those embodiments of the present technology, where the electronic device 204 is implemented as a wireless communication device, such as a smartphone, the communication link can be implemented as a wireless communication link. Examples of wireless communication lines may include, but are not limited to, a 3G communication network channel, a 4G communication network channel, etc. The communication network 208 may also use a wireless connection to the server 202.

Process training

[095] The training process begins with the formation by the server 202 of training data sets for the sets 210 of algorithms, 215 of machine learning for determining density and color. In this regard, the server 202 is capable of receiving training two-dimensional video content representing a set of training interpenetrating objects.

[096] Fig. 4 is a schematic representation of a displayed training scene 402 in a system for generating a training data set for training a plurality 210 of machine learning algorithms for determining density and a plurality 215 of machine learning algorithms for determining color in accordance with some non-limiting embodiments of the present technology.

[097] It is obvious that the training scene 402, like the scene 302, contains two interpenetrating objects, namely the first training object 401 and the second training object 403, which form a training overlapping region 405 between them. Although the first and second training objects 401, 403 are boxers, like in the scene 302, it should be understood that such similarity is due solely to considerations of clarity and simplicity of description of the present technology and in some non-limiting embodiments of the present technology, the training scene 402 may represent another event with other objects, such as football players during a football match, dancers, etc.

[098] According to some non-limiting embodiments of the present technology, the training two-dimensional video content for forming the training data set is generated by cameras that record the training scene 402, in particular, the first camera 408 and the second camera 410. In some non-limiting embodiments of the present technology, the first and second cameras 408, 410 are located on opposite sides of the training scene 402. It should be understood that there is no limitation on the number of cameras used to generate the training two-dimensional video content and such content can be generated, for example, using only one of the first and second cameras 408, 410 or using more than two cameras, in particular three, five or ten cameras located around the training scene 402.

[099] There are also no limitations on the method of implementing the first and second cameras 408, 410, and in some non-limiting embodiments of the present technology, each of these cameras may comprise a complementary metal-oxide-semiconductor (CMOS) image sensor capable of producing image sequences corresponding to a desired playback speed, in particular, 24 frames per second, 48 frames per second, or 72 frames per second.

[0100] In a specific non-limiting example, each of the cameras may be implemented using a CMOS image sensor manufactured by Sony Semiconductor Solutions Corporation, 4-14-1 Asahi-cho, Atsugi-shi, Kanagawa, 243-0014, Japan. It is obvious that the cameras may be implemented using any other suitable equipment.

[0101] Furthermore, according to some non-limiting embodiments of the present technology, the first and second cameras 408, 410 may be placed within the training scene 402 on the inner surface of the imaginary sphere 420 such that each of the first and second cameras 408, 410 faces the center (not shown) of the imaginary sphere 420, and the first and second training objects 401, 403 are located in one of the cross-sections of the imaginary sphere 420. There are no limitations on the dimensions of the imaginary sphere 420 and, in some non-limiting embodiments of the present technology, the imaginary sphere 420 may have, for example, a diameter of 4 meters.

[0102] Thus, according to some non-limiting embodiments of the present technology, each of the first and second cameras 408, 410 is capable of generating a sequence of training two-dimensional images representing the training scene 402, from the angle of this camera. In some non-limiting embodiments of the present technology, the first and second cameras 408, 410 are synchronized and, as a result, simultaneously generate training two-dimensional images representing the training scene 402.

[0103] In addition, in some non-limiting embodiments of the present technology, the server 202 is capable of receiving training two-dimensional image sequences directly from the first and second cameras 408, 410 via a wired or wireless communication channel. In other non-limiting embodiments of the present technology, training two-dimensional image sequences may be collected on a third-party server (not shown) to which the server 202 is connected, or on a removable medium located on the server 202.

[0104] Also, after receiving the sequences of training two-dimensional images representing the training scene 402 and generated by the first and second cameras 408, 410, the server 202, in accordance with some non-limiting embodiments of the present technology, is capable of generating sequences of corrected training two-dimensional images based on these sequences of training two-dimensional images. For this, the server 202, in accordance with some non-limiting embodiments of the present technology, is capable of projecting training two-dimensional images from each of the sequences of training two-dimensional images onto a common plane 412 passing parallel to the baseline 414.

[0105] Using each corrected training two-dimensional image from the plurality of corrected training two-dimensional images generated in this manner, for example, the corrected training two-dimensional image 406, the server 202 is able to generate a training data set. In this regard, according to some non-limiting embodiments of the present technology, the server 202 is able to analyze the corrected training two-dimensional image 406 by applying a semantic segmentation model (not shown) to it. The server 202 can access the semantic segmentation model, for example, via the communication network 208. According to some non-limiting embodiments of the present technology, the semantic segmentation model can be pre-trained to classify each pixel of the corrected training two-dimensional image 406 as belonging to one or more objects, such as the first and second training objects 401, 403, or to the background. There are no restrictions on the method of implementing the semantic segmentation model, and in some non-restrictive embodiments of the present technology, the semantic segmentation model may be a convolutional neural network (CNN).

[0106] In addition, according to some non-limiting embodiments of the present technology, the server 202 is capable of generating a training three-dimensional scene based on the adjusted training two-dimensional image 406. Fig. 5 schematically shows a training three-dimensional scene 502 generated based on the adjusted training two-dimensional image 406, according to some non-limiting embodiments of the present technology.

[0107] According to some non-limiting embodiments of the present technology, the server 202, based on the classes of objects determined for all pixels of the adjusted training two-dimensional image 406, is capable of generating for each of the first and second training objects 401, 403 a training estimate of a pose according to a linear parametric model of the surface of a human body SMPL, in particular, a first training estimate 501 of a pose SMPL for the first training object 401 and a second training estimate 503 of a pose SMPL for the second training object 403.

[0108] In general, each SMPL pose estimate of the first and second training SMPL pose estimates 501, 503, in particular the first training SMPL pose estimate 501, includes grid elements (e.g., triangular grid elements) that contain a plurality of vertices that form a surface of the first training object 401 in a pose captured in the adjusted training two-dimensional image 406. Furthermore, in some non-limiting embodiments of the present technology, the first and second training SMPL pose estimates 501, 503 include SMPL pose training vectors, that is, the first SMPL pose training vector 505 and the second SMPL pose training vector 507, respectively.

[0109] According to some non-limiting embodiments of the present technology, each of the first and second training pose vectors 505, 507 SMPL, for example the first training pose vector 505 SMPL, comprises pose and shape parameters associated with the first training object 401 and characterizing the pose and shape of the first training object 401 captured in the adjusted training two-dimensional image 406. In particular, in those embodiments in which the first training object 401 is a person, the pose parameters in the first training pose vector 505 SMPL include parameters characterizing relative rotation angles of the joints of the person in the pose captured in the adjusted training two-dimensional image 406. In some non-limiting embodiments of the present technology, the pose parameters include sixty-nine parameters. The shape parameters characterize the degree of stretching or compression of the first training object 401 on the adjusted training two-dimensional image 406 along a plurality of specified directions. For example, one of the plurality of specified directions can be oriented along the longitudinal axis of the first training object 401, passing along its spine, and the corresponding shape parameters can characterize the height of the first training object 401 on the adjusted training two-dimensional image 406. In some non-limiting embodiments of the present technology, the shape parameters include ten parameters.

[0110] According to some non-limiting embodiments of the present technology, to generate the first and second training estimates 501, 503 of the SMPL pose for the first and second training objects 401, 403, respectively, the server 202 is capable of applying to the corrected training two-dimensional image 406 containing the classified pixels, an algorithm based on the SMPL model created using the EasyMocap toolkit. However, any other suitable algorithm can also be used.

[0111] In some non-limiting embodiments of the present technology, as explained in the description below, the server 202 is capable of generating (or otherwise obtaining) for each of the first and second training objects 401, 403 a training canonical pose SMPL, in particular a first training canonical pose 601 SMPL for the first training object 401, schematically illustrated in Fig. 6.

[0112] The first training canonical pose 601 SMPL, like the first training estimate 501 of the pose SMPL, may comprise elements of a grid including vertices forming a surface of the first training object 401, however, unlike the first training estimate 501 of the pose SMPL, the vertices of the first canonical pose 601 SMPL represent the surface of the first training object 401 in a predetermined pose. In the embodiment shown in Fig. 6, the predetermined pose of the first training object 401 is a so-called T-shaped pose, that is, an upright standing position with arms spread apart parallel to the supporting surface of the first training object 401.

[0113] In some non-limiting embodiments of the present technology, to generate the first training canonical pose 601 SMPL, the server 202 is capable of using a trained machine learning algorithm based on the SMPL model (not shown), which the server 202 can access, for example, via the communication network 208. In general, the trained machine learning algorithm based on the SMPL model may be trained by the server 202 or any other third-party server to generate estimates of the canonical SMPL pose based on linear combinations of SMPL pose vectors representing artist-generated SMPL poses for various objects in desired poses, such as the aforementioned predetermined pose of the first training object 401. According to some non-limiting embodiments of the present technology, the SMPL pose vectors may be generated similarly to the first and second training vectors 505, 507 of the SMPL pose and, thus, include shape and pose parameters of the artist-generated SMPL poses. In a specific non-limiting example, the trained machine learning algorithm based on the SMPL model may be implemented and trained as described in the article by Loper et al. "SMPL: A Skinned Multi-Person Linear Model", ACM Transactions on Graphics (TOG), Volume 34, October 2015. Obviously, other machine learning algorithms trained to generate canonical SMPL poses can also be used.

[0114] In some non-limiting embodiments of the present technology, the server 202 is configured to download a first training canonical pose 601 SMPL for the first training object 401 from a reference library of canonical poses SMPL, which may be located on a third-party server (not shown) connected to the communication network 208.

[0115] It should be understood that, although not shown, the second training canonical pose SMPL for the second training object 403 can be implemented and received by the server 202 in a similar manner to the first training canonical pose 601 SMPL. In addition, as described in detail below, the server 202 is capable of using the first training canonical pose 601 SMPL and the second training canonical pose SMPL for the purpose of forming a training data set for training a plurality 210 of machine learning algorithms for determining density.

[0116] As shown in Fig. 5, in order to generate the training three-dimensional scene 502, the server 202, according to some non-limiting embodiments of the present technology, is capable of constructing a closed three-dimensional space 520 around the first and second training estimates 501, 503 of the SMPL pose generated as described above. There are no restrictions on the shape and dimensions of the closed three-dimensional space 520 and, in some non-limiting embodiments of the present technology, this space corresponds to an imaginary sphere 420 constructed around the training scene 402. In other words, in such embodiments, the closed three-dimensional space 520 can also be a sphere constructed around the first and second training estimates 501, 503 of the SMPL pose so that they are located in one of the sections of this sphere, and the dimensions of this sphere are determined by the ratio between the dimensions of the imaginary sphere 420 and the actual dimensions of the first and second training objects 401, 403.

[0117] Furthermore, according to some non-limiting embodiments of the present technology, the server 202 is capable of generating in the closed three-dimensional space 520 a plurality of viewpoints for the first and second training estimates 501, 503 of the SMPL pose. In some non-limiting embodiments of the present technology, the server 202 is capable of generating a plurality of viewpoints corresponding to a plurality of cameras used to generate a plurality of adjusted training two-dimensional images. For example, the server 202 is capable of generating (a) a first viewpoint 508 corresponding to the position and orientation of the first camera 408, and (b) a second viewpoint 510 corresponding to the position and orientation of the second camera 410. In other words, the server 202 is capable of generating the first and second viewpoints 508, 510 located on the inner surface of the closed three-dimensional space 520 and oriented toward the center of the closed three-dimensional space 520.

[0118] In addition, according to some non-limiting embodiments of the present technology, the server 202 is capable of employing a volumetric raycasting algorithm. For this, the server 202 is capable of constructing a plurality of rays from each of the first and second viewpoints 508, 510, in particular, a ray 525 emanating from the first viewpoint 508.

[0119] In some non-limiting embodiments of the present technology, the server 202 is capable of constructing a plurality of rays from each of the first and second viewpoints 508, 510 such that the rays in the plurality of rays are spaced equally apart. However, in other non-limiting embodiments of the present technology, the rays in the plurality of rays may be spaced randomly. Furthermore, in some non-limiting embodiments of the present technology, the number of rays in the plurality of rays depends on the required coverage of the rays of the first and second training estimates 501, 503 of the SMPL pose and/or the required number of training digital objects in the training data set for the plurality 210, 215 of machine learning algorithms for determining density and color. In this regard, the number of rays for the plurality of rays may be selected based on a trade-off between the available computing resources of the server 202 and the required training accuracy of the plurality 210, 215 of machine learning algorithms for determining density and color. For example, in some non-limiting embodiments of the present technology, the server 202 is capable of constructing a plurality of rays containing 2048 rays from each of the first and second viewpoints 508, 510.

[0120] Furthermore, according to some non-limiting embodiments of the present technology, the server 202 is capable of defining on each ray constructed from each of the first and second viewpoints 508, 510, in particular on the ray 525, a plurality of points, such as the point 530. In some non-limiting embodiments of the present technology, the server 202 is capable of defining on each of the plurality of rays the same number of points. In some non-limiting embodiments of the present technology, the server 202 is capable of defining on the ray 525 a plurality of points with a predetermined number of points. The number of points, as well as the number of rays, can be selected based on a trade-off between available computing resources and the required accuracy of training the plurality 210, 215 machine learning algorithms for determining density and color. In a specific non-limiting example, server 202 is capable of specifying 64 points on beam 525. However, in other non-limiting embodiments of the present technology, server 202 is capable of specifying 128, 256, or 1024 points on each beam of the sets of beams emanating from the viewpoints.

[0121] In some non-limiting embodiments of the present technology, the server 202 is capable of defining the plurality of points on the beam 525 uniformly. In other words, the plurality of points defined by the server 202 may be uniformly distributed along the beam 525.

[0122] In addition, using a plurality of points defined on rays constructed from viewpoints, in particular from the first and second viewpoints 508, 510, the server 202 is able to generate data sets for training in determining the density for the purpose of training each of the plurality 210 machine learning algorithms for determining the density to find the boundary between the first and second objects 301, 303.

[0123] In this regard, according to some non-limiting embodiments of the present technology, the server 202 is capable of finding for the point 530 in each of the first training canonical pose 601 SMPL and the second training canonical pose SMPL associated with the first and second training objects 401, 403, a corresponding canonical point.

[0124] Furthermore, according to some non-limiting embodiments of the present technology, the server 202 is capable of generating a set of density training data for each machine learning algorithm of the plurality 210 density training algorithms. In particular, for the first density training algorithm associated with the first training object 401, the server 202 is capable of generating a first set of density training data comprising a first plurality of density training digital objects, each of which comprises (a) coordinates of a canonical point in a coordinate system (not shown) associated with the first training canonical pose 601 of the SMPL, and (b) a first training pose vector 505 of the SMPL.

[0125] Similarly, for the second machine learning algorithm of the plurality 210 machine learning algorithms for determining density, the server 202 is capable of generating a second set of data for training to determine density, comprising a second plurality of digital objects for training to determine density, each of which includes, for the point 530 (a), coordinates of a canonical point in a coordinate system (not shown) associated with a second training canonical pose SMPL associated with the second training object, and (b) a second training vector 507 of the pose SMPL.

[0126] The method for determining the canonical points in the SMPL canonical poses by the server 202 is not limited, and in some non-limiting embodiments of the present technology, the server 202 is capable of finding a canonical point, for example, in the first training canonical pose 601 of the SMPL, by applying a skinning algorithm based on linear blending (skeletal animation with weights) using the SMPL model. In particular, in such embodiments, the server 202 is capable of finding a canonical point using the following formula:

where x ^c is the canonical point associated with the training object;

x ^d - point 530 on ray 525;

B _i is a transformation matrix associated with the joint j _i of the first training object 401 and formed on the basis of the first SMPL pose estimate 501;

n _b - the number of joints in the first training object 401;

w _i ^d is a function characterizing the distribution of weights in the skinning process, determined based on the coordinates of the vertex of the first SMPL pose estimate 501 closest to the given point.

[0127] According to some non-limiting embodiments of the present technology, the server 202, at a given training iteration, is configured to transmit the digital density learning objects associated with the point 530 to each of the plurality 210 density machine learning algorithms, whereby each machine learning algorithm of the plurality 210 density machine learning algorithms generates a signed distance field prediction value for the respective canonical points associated with the first and second training objects 401, 403. In other words, at a given training iteration, the server 202 is configured to (a) transmit each digital density learning object of the first plurality of digital density learning objects to the first density machine learning algorithm, whereby the first density machine learning algorithm generates a first signed distance field prediction value for the canonical point in the first training canonical pose 601 SMPL, and (b) transmit each digital density learning object of the second plurality of digital density learning objects to the second machine learning algorithm density learning, whereby the second density learning algorithm produces a second predicted signed distance field value for the canonical point in the second training canonical pose SMPL.

[0128] As explained in the description below, in some non-limiting embodiments of the present technology, each of the plurality 210 machine learning algorithms for determining the density, after receiving the digital object for learning to determine the density, is further configured to generate a vector representation of a signed distance field for the corresponding canonical point. In other words, the first machine learning algorithm for determining the density is configured to generate, along with a first predicted value of the signed distance field, a first vector representation of the signed distance field for the canonical point corresponding to the point 530 in the first training canonical pose 601 of the SMPL. Similarly, the second machine learning algorithm for determining the density is configured to generate, along with a second predicted value of the signed distance field, a second vector representation of the signed distance field for the canonical point corresponding to the point 530 in the second training canonical pose of the SMPL.

[0129] In addition, to train the plurality 210 machine learning algorithms for determining the density, the server 202 is configured to apply a loss function for a signed distance field. According to some non-limiting embodiments of the present technology, the loss function for the signed distance field is configured to assign a penalty to the first and/or second predicted values of the signed distance field if the predicted value of the signed distance field generated by a given machine learning algorithm for determining the density is equal to the predicted value of the signed distance field generated by another machine learning algorithm for determining the density of the plurality of machine learning algorithms. In some non-limiting embodiments of the present technology, the loss function for the signed distance field is configured to assign a penalty to the first and/or second predicted values of the signed distance field if both the predicted values of the signed distance field generated by the given machine learning algorithm and another machine learning algorithm of the plurality 210 machine learning algorithms for determining the density are negative.

[0130] In some non-limiting embodiments of the present technology, the loss function for the signed distance field may be expressed by the following formula:

where P is the number of training objects, which in the presented example is equal to two;

- the number of unique pairs of training objects;

s _i ^k is the first predicted value of the signed distance field for the canonical point corresponding to the point 530 in the first training canonical pose 601 of the SMPL, generated by the first machine learning algorithm for determining the density;

s _i ^p is the second predicted signed distance field value for the canonical point corresponding to point 530 in the second training canonical pose of SMPL, generated by the second density machine learning algorithm.

[0131] By transmitting the first and second sets of digital objects for training to determine the density and then applying the above-mentioned loss function for the signed distance field at each iteration of training, the server 202 is configured to train each of the first and second machine learning algorithms of the plurality 210 machine learning algorithms for determining the density to determine the values of the signed distance field for the first and second objects 301, 303. In some non-limiting embodiments of the present technology, after training the first and second machine learning algorithms for determining the density of the plurality 210 machine learning algorithms for determining the density, the server 202 is configured to use the first and second machine learning algorithms for determining the density when determining the value of the signed distance field for the first and second objects 301, 303 to determine the boundary between these objects, as described below.

[0132] In other non-limiting embodiments of the present technology, the server 202 is configured to continue the training process using the current predictions of the plurality 210 density machine learning algorithms generated at a given training iteration to train the plurality 215 color machine learning algorithms.

[0133] As with training the plurality 210 machine learning algorithms for determining density, the server 202 is capable of generating a set of data for training the determination of color for each machine learning algorithm from the plurality 215 machine learning algorithms for determining color. In particular, for the first machine learning algorithm for determining a color associated with the first training object 401, the server 202 is capable of generating a first set of data for learning to determine a color, comprising a first plurality of digital objects for learning to determine a color, each of which includes, for the point 530, (a) coordinates of a canonical point for the first training canonical pose 601 SMPL, (b) a first vector representation of a signed distance field generated by the first machine learning algorithm for determining the density, and (c) a label comprising a color value for a pixel 540 on the adjusted training two-dimensional image 406 through which the ray 525 passes. In some non-limiting embodiments of the present technology, each digital object for learning to determine a color from the first plurality of digital objects for learning to determine a color may further include (d) a second training canonical pose SMPL for the first training estimate 501 of the pose SMPL and (e) normals to the surface of the first training canonical pose 601 SMPL. In some non-limiting embodiments of the present technology, the server 202 is capable of determining the surface normals of the first training canonical pose 601 SMPL as gradients of signed distance field values determined by a plurality 210 of machine learning algorithms for determining density.

[0134] Similarly, for training a second machine learning algorithm associated with the second training object 403, from the plurality 215 of machine learning algorithms for determining color, the server 202 is capable of generating a second set of data for learning color, comprising a second plurality of digital objects for learning color, each of which includes, for the point 530, (a) coordinates of a canonical point for a second training canonical pose SMPL, (b) a second vector representation of a signed distance field generated by the second machine learning algorithm for determining density, and (c) a label comprising a color value for a pixel 540 in the adjusted training two-dimensional image 406 through which the ray 525 passes. In some non-limiting embodiments of the present technology, each digital object for learning color from the second plurality of digital objects for learning color may further include (d) a second training canonical pose SMPL for a second training estimate 503 of the pose SMPL and (e) normals to the surface of the second training canonical pose SMPL. Similarly, in some non-limiting embodiments of the present technology, the server 202 is capable of determining the surface normals of the second training canonical pose SMPL as gradients of the signed distance field values determined by the plurality 210 of machine learning algorithms for determining density.

[0135] In some non-limiting embodiments of the present technology, the server 202 is configured to select a set of points from a plurality of points defined on the beam 525 for the purpose of forming data sets for training to determine a color for a plurality 215 of machine learning algorithms for determining a color. According to some non-limiting embodiments of the present technology, the server 202 is configured to find, in the plurality of points defined on the beam 525, a set of points located on the beam 525 in areas with a higher point density. In other words, when selecting a set of points on the beam 525, the server 202 is configured to find, in the plurality of points on the beam 525, points related to the first and second training objects 401, 403 (that is, points located on the first and second training estimates 501, 503 of the SMPL pose), and to filter out points from the plurality of points related to the background.

[0136] According to some non-limiting embodiments of the present technology, the server 202 is configured to find a set of points on the beam 525 for the purpose of generating data sets for training to determine a color based on the density values for the plurality of points. In some non-limiting embodiments of the present technology, the server 202 is configured to determine the density values based on the predicted signed distance field values generated by the plurality 210 of machine learning algorithms for determining the density for the canonical points of the first and second training objects 401, 403 that are associated with the plurality of points defined on the beam 525, such as the point 530. To this end, according to some non-limiting embodiments of the present technology, the server 202 is configured to apply a normalized cumulative Laplace distribution function to the predicted signed distance field values associated with the point 530. Then, the server 202, based on the density values determined in the specified manner for the point 530, can either include the point 530 in the set of points of the beam 525, or refuse to include this point in the set of points. For example, in some non-limiting embodiments of the present technology, the server 202 is capable of (a) determining an average density value for the point 503 based on the density values for the corresponding canonical points, (b) comparing the average density value with a given density threshold value, and (c) if the average density value is equal to or exceeds the given density threshold value, including the point 530 in the set of points for subsequent formation of data sets for training the color determination for the purpose of training the plurality 215 of machine learning algorithms for determining color.

[0137] In other non-limiting embodiments of the present technology, the server 202 is capable of applying a significance sampling algorithm to the density values associated with the point 530. In some non-limiting embodiments of the present technology, the significance sampling algorithm may be implemented as an opacity function, as described in detail, for example, in the aforementioned article "Volume Rendering of Neural Implicit Surfaces."

[0138] In general, the importance sampling algorithm is capable of generating, based on the density values determined based on the signed distance field values obtained using the plurality 210 of machine learning algorithms for determining the density, a larger number of points on the beam 525 representing each of the first and second training estimates 501, 503 of the SMPL pose. Furthermore, according to some non-limiting embodiments of the present technology, the server 202 is capable of combining the points defined on the beam 525 for each of the first and second training estimates 501, 503 of the SMPL pose in order to subsequently determine the corresponding integral values, as described in detail below.

[0139] In addition, in the next iteration of training, after causing the plurality 210 machine learning algorithms for determining the density of predicted values of the signed field of distances for all points on the ray 525 to form, the server 202, in accordance with some non-limiting embodiments of the present technology, is capable of transmitting to each of the plurality 215 machine learning algorithms for determining a color a digital object for learning to determine a color associated with the point 530, as a result of which each of the plurality of machine learning algorithms for determining a color forms a predicted color value for a canonical point associated with the corresponding training object. In particular, when receiving a digital object for learning to determine the color from a first plurality of digital objects for learning to determine the color, the first machine learning algorithm for determining the color is capable of generating a first predictive color value for a canonical point associated with the first learning object 401. Similarly, when receiving a digital object for learning to determine the color from a second plurality of digital objects for learning to determine the color, the second machine learning algorithm for determining the color is capable of generating a second predictive color value for a canonical point associated with the second learning object 403.

[0140] Thereafter, in accordance with some non-limiting embodiments of the present technology, the server 202 is capable of combining the predicted color values generated by the plurality 215 of machine learning algorithms for determining the color for each of the plurality of points (or for a set of points selected from this plurality) defined on the beam 525 in the next iteration of training, in order to generate an intermediate integral color value for the beam 525. For example, the server 202 is capable of determining the intermediate integral color value by summing the predicted color values for the points defined on the beam 525.

[0141] According to some non-limiting embodiments of the present technology, the server 202 is configured to determine the intermediate integral color value as a weighted sum, such as according to the following formula:

where c _i ^p is the predicted color value for the canonical point associated with object p , generated by a given machine learning algorithm from a set of 215 machine learning algorithms for color determination;

w _i ^p - the weight value determined for the predicted color value based on the predicted density values;

bg _r - background color value associated with ray 525.

[0142] There is no limitation on the method by which server 202 determines the background color value associated with ray 525, and in some non-limiting embodiments of the present technology, server 202 is capable of training and then using a special machine learning algorithm for the background. In a specific non-limiting example, a special machine learning algorithm for the background may be implemented and trained as described in detail in the paper by Guo et al., “Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition,” published on arxiv.org in February 2023, the contents of which are incorporated herein by reference in their entirety.

[0143] According to some non-limiting embodiments of the present technology, the server 202 is configured to determine a weight value for a predicted color value, in particular for a first or second predicted color value associated with the point 530, based on the predicted density values associated with the point 530. In some non-limiting embodiments of the present technology, the server 202 is configured to determine the weight value using the following formula:

where Δ x _i is the length of the segment between point 530 and the next point on the ray 525;

σ _i ^p is a predicted density value generated by a machine learning algorithm from a set 210 of machine learning algorithms for determining the density associated with a training object p , in particular with the first training object 401 or with the second training object 403, based on the corresponding value of the signed distance field.

[0144] After determining the intermediate integral color value for the beam 525, as described above, the server 202 is capable of jointly training the plurality 215 machine learning algorithms for determining the color, applying in the next iteration of training a color loss function capable of assigning a penalty to the predicted color values generated by the plurality 215 machine learning algorithms for determining the color for all points on the beam 525 if the intermediate integral color value differs from the color value of the pixel 540 from the label of the digital objects for color determination training associated with the beam 525. In a specific non-limiting example, the color loss function is an RGB loss function, as described in detail in the above-mentioned article by Guo et al.

[0145] It should be understood that there is no limitation on the number of training digital objects for training individual machine learning algorithms from the plurality 210 machine learning algorithms for determining density, as well as for training individual machine learning algorithms from the plurality 215 machine learning algorithms for determining color, and the server 202 is capable of generating thousands, tens of thousands, hundreds of thousands or even millions of training digital objects for training each machine learning algorithm based on other adjusted training two-dimensional images from the sequence of adjusted training two-dimensional images, as described above in relation to the adjusted training two-dimensional image 406.

[0146] Thus, by jointly applying the loss function for the signed distance field and the loss function for the color over a plurality of training iterations, the server 202, in accordance with some non-limiting embodiments of the present technology, is able to train a plurality 210 of machine learning algorithms for determining the density to determine the values of the signed distance field and a plurality 215 of machine learning algorithms for determining the color to determine the color values for working interpenetrating objects, such as the first and second objects 301, 303, schematically represented in Fig. 3.

[0147] Furthermore, in those non-limiting embodiments of the present technology in which the server 202 is capable of jointly training the plurality 210 machine learning algorithms for determining density and the plurality 215 machine learning algorithms for determining color using a loss function for color, the server 202 is capable of further training each of the plurality 210 machine learning algorithms for determining density to determine values of a signed distance field based on labels of digital objects for learning to determine color.

[0148] The server 202 is then capable of using a plurality 210 of machine learning algorithms for determining density and a plurality 215 of machine learning algorithms for determining color for the purpose of volumetric rendering of the scene 302 containing the first and second objects 301, 303, as described below.

Worker process

[0149] According to some non-limiting embodiments of the present technology, the server 202 is capable of generating working digital objects to perform volume rendering using the plurality 210 machine learning algorithms for determining density and the plurality 215 machine learning algorithms for determining color.

[0150] For this, the server 202, in accordance with some non-limiting embodiments of the present technology, is capable of previously obtaining at least one sequence of working two-dimensional images representing the scene 302. In accordance with some non-limiting embodiments of the present technology, the at least one sequence of working two-dimensional images representing the scene 302 can be formed similarly to the sequences of training two-dimensional images for the training scene 402.

[0151] Fig. 7 is a schematic representation of a scene 302 in the system shown in Fig. 4 that is used to generate working digital objects, in accordance with some non-limiting embodiments of the present technology.

[0152] As shown most clearly in Fig. 7, a plurality of cameras for generating sequences of working two-dimensional images representing the scene 302, in particular the first and second cameras 408, 410, can be located on the inner surface of an imaginary sphere 420 constructed around the scene 302, as in the case of generating sequences of training two-dimensional images. Each of the first and second cameras 408, 410 can face the center of the imaginary sphere, as in the camera system for the training scene 402 described above with reference to Fig. 4.

[0153] Thus, the server 202 is capable of receiving (a) a first sequence of working two-dimensional images from the first camera 408 and (b) a second sequence of working two-dimensional images from the second camera 410. Then, as in the case of generating a sequence of corrected training two-dimensional images, based on the first and second sequences of working two-dimensional images representing the scene 302, the server 202, in accordance with some non-limiting embodiments of the present technology, is capable of generating a sequence of corrected working two-dimensional images representing the three-dimensional scene 302.

[0154] Thereafter, the server 202, based on the working corrected two-dimensional image 806 from the sequence of corrected working two-dimensional images, is capable of generating (a) a three-dimensional scene (for example, a three-dimensional scene 902 shown in Fig. 8), (b) working digital objects based on the three-dimensional scene 902 for determining the boundary between the first and second objects 301, 302 in the three-dimensional scene 902, as well as color values for points on rays passing through this scene.

[0155] Fig. 8 is a schematic representation of a three-dimensional scene 902 generated from a corrected working two-dimensional image 806, in accordance with some non-limiting embodiments of the present technology.

[0156] According to some non-limiting embodiments of the present technology, in order to generate three-dimensional models of the first and second objects 301, 303, the server 202 is capable of generating corresponding working estimates of the pose SMPL. According to some non-limiting embodiments of the present technology, the server 202 is configured to generate, based on the adjusted working two-dimensional image 806, a first working estimate 901 of the SMPL pose, comprising a first working vector 905 of the SMPL pose, for the first object 301 and a second working estimate 903 of the SMPL pose, comprising a second working vector 907 of the SMPL pose, for the second object 303, obtaining the poses of the first and second objects 301, 303, as represented in the adjusted working two-dimensional image 806, similar to the first and second training estimates 501, 503 of the SMPL pose for the first and second training objects 401, 403.

[0157] According to some non-limiting embodiments of the present technology, the server 202 is capable of generating or otherwise obtaining for the first and second objects 301, 303, respectively, first and second working canonical poses SMPL similar to the first training canonical pose 601 SMPL described above with reference to Fig. 6.

[0158] In addition, as described above with reference to Fig. 5, the server 202 is capable of constructing around the first and second working estimates 901, 903 of the SMPL pose a closed three-dimensional space 520 containing a plurality of viewpoints, such as the first and second viewpoints 508, 510, located on the inner surface of the closed three-dimensional space 520.

[0159] As described above with reference to Fig. 5, the server 202 is configured to (a) conduct a plurality of operating beams from each of the first and second viewpoints 508, 510 and (b) define a plurality of operating points on the operating beam 925, such as the operating point 930.

[0160] In addition, as described above with respect to point 530, server 202 is capable of finding first and second canonical points in first and second working canonical poses of SMPL.

[0161] Also, the server 202 is capable of generating working digital objects for each machine learning algorithm from the plurality 210 machine learning algorithms for determining the density and each machine learning algorithm from the plurality 215 machine learning algorithms for determining the color, which are associated with a given object. In particular, the server 202 is capable of generating for the first machine learning algorithm from the plurality 210 machine learning algorithms for determining the density and the first machine learning algorithm from the plurality 215 machine learning algorithms for determining the color, associated with the first object 301, a first working digital object, which includes (a) coordinates of a canonical point in the first working canonical pose SMPL, associated with the first object 301, in the grid of the signed distance field and (b) a first working vector 905 of the pose SMPL. Similarly, the server 202 is capable of generating for the second machine learning algorithm from the plurality 210 of machine learning algorithms for determining the density and the second machine learning algorithm from the plurality 215 of machine learning algorithms for determining the color, associated with the second object 303, a second working digital object, including (a) the coordinates of the canonical point in the second working canonical pose SMPL, associated with the second object 303, in the grid of the signed distance field and (b) the second working vector 907 of the pose SMPL.

[0162] In addition, the server 202 is capable of transmitting the first working digital object (a) to a first machine learning algorithm for determining the density, whereby the first machine learning algorithm for determining the density generates a first working value of the signed distance field for the canonical point in the first working canonical pose SMPL associated with the first object 301, and (b) to a first machine learning algorithm for determining the color, whereby the first machine learning algorithm for determining the color generates a first working color value for the canonical point in the first working canonical pose SMPL associated with the first object 301. Similarly, the server 202 is capable of transmitting the second working digital object (a) to a second machine learning algorithm for determining the density, whereby the second machine learning algorithm for determining the density generates a second working value of the signed distance field for the canonical point in the second working canonical pose SMPL associated with the second object 303, and (b) to a second machine learning algorithm for determining the color, whereby the second machine learning algorithm for determining the color generates a second working color value for canonical point in the second working canonical pose SMPL associated with the second object 303.

[0163] Also, the server 202, based on the first and second working values of the signed field of distances associated with the point 930, is able to determine whether the point 930 is within one of the first and second working estimates 901, 903 of the pose SMPL. Based on this, the server 202 is further able to determine the boundary between the first and second objects 301, 303 within the overlap region 305.

[0164] In addition, in order to obtain a final color value for the first and second working estimates 901, 903 of the SMPL pose, the server 202 is configured to determine an integral color value for the beam 925. In this regard, the server 202 is configured to (a) determine first and second working color values generated by the plurality 215 of machine learning algorithms for determining color, for each operating point defined on the working beam 925, (b) determine first and second working values of a signed distance field generated by the plurality 210 of machine learning algorithms for determining density, for each operating point defined on the working beam 925, (c) determine, based on the first and second working values of the signed distance field, first and second working values of density for each operating point defined on the working beam 925, as described above, and (d) determine the integral color value for the beam 925 by applying formulas (3) and (4) to the first and second working color values and to the first and the second working density values.

[0165] Thus, using the plurality 210 of machine learning algorithms for determining the density and the plurality 215 of machine learning algorithms for determining the color, the server 202 is able to determine an operating value of the signed distance field for a given point on each ray of the three-dimensional scene 902 and an integral color value for each ray emanating from each of the first and second viewpoints 508, 510. As a result, the server 202 is able to construct images of the first three-dimensional model 1001 for the first object 301 and the second three-dimensional model 1003 for the second object 303, schematically shown in Fig. 9, in accordance with some non-limiting embodiments of the present technology.

[0166] Then, after constructing the images of the first and second three-dimensional models 1001, 1003 in the three-dimensional scene 902, the server 202, in accordance with some non-limiting embodiments of the present technology, is capable of transmitting the three-dimensional scene 902 as part of the three-dimensional animated video content 214 to the electronic device 204 for demonstrating the three-dimensional scene 902 to the user 206.

[0167] From the perspective of the first and second three-dimensional models 1001, 1003 shown in Fig. 9, it can be seen that as a result of applying the variants of the present technology for determining the boundary between the first and second objects 301, 303 and the color values for them, the overlapping region 305, which is present in the original scene 302, is accurately processed in the three-dimensional scene 902, which provides a more realistic representation of the first and second objects 301, 303. This makes it possible to improve the user's 206 experience of viewing the three-dimensional animated video content 214 transmitted by the server 202.

Way

[0168] Taking into account the above-described architecture and the given examples, it is possible to implement a method for volume rendering of three-dimensional scenes representing a plurality of interpenetrating objects, in particular, a three-dimensional scene 902 representing the first and second objects 301, 303, which form an overlapping region 305 between them. Fig. 10 shows a block diagram of a method 1100 in accordance with some non-limiting embodiments of the present technology. The method 1100 can be implemented, for example, on a server 202.

[0169] As described in detail above, the method 1100 includes training a plurality 210 of machine learning algorithms for determining density and a plurality 215 of machine learning algorithms for determining color in order to determine, respectively, values of the signed field of distances and color for the first and second objects 301, 303.

Step 1102: obtaining from a given camera of the plurality of cameras a sequence of training two-dimensional images representing a plurality of interpenetrating training objects.

[0170] The method 1100 begins with step 1102, in which the server 202 is configured to receive from a plurality of cameras, such as the first and second cameras 408, 410, sequences of training two-dimensional images representing the training scene 402, as described above with reference to Fig. 4.

[0171] As indicated above, a plurality of cameras may be located on the inner surface of the imaginary sphere 420 and facing its center.

[0172] The next step in method 1100 is step 1104.

Step 1104: forming a sequence of corrected training two-dimensional images based on the sequences of training two-dimensional images obtained from a plurality of cameras.

[0173] In step 1104, according to some non-limiting embodiments of the present technology, the server 202 is configured to generate, based on the training images from the simultaneously obtained sequences of training two-dimensional images, a sequence of corrected training two-dimensional images, as described above with reference to Fig. 4.

[0174] The next step in method 1100 is step 1106.

Step 1106: generating a pose estimate according to the SMPL model for a given training object from a plurality of interpenetrating training objects on a given adjusted training 2D image from a sequence of adjusted training 2D images.

[0175] Then, in step 1106, according to some non-limiting embodiments of the present technology, the server 202 is capable of initiating the construction of a training three-dimensional scene 502 based on the adjusted training two-dimensional image 406. For this, as described in detail above with reference to Fig. 5, the server 202 is capable of pre-generating for the first and second training objects 401, 403, respectively, first and second pose estimates 501, 503 SMPL, which characterize the poses of the first and second training objects 401, 403 in the adjusted training two-dimensional image 406.

[0176] The next step in method 1100 is step 1108.

Step 1108: obtaining, for a given training object from a plurality of interpenetrating training objects, a canonical pose SMPL, including canonical vertices forming the surface of the given training object in a predetermined pose.

[0177] In step 1108, according to some non-limiting embodiments of the present technology, the server 202 is configured to receive (or generate) for the first and second training objects 401, 403 training canonical poses SMPL representing the first and second training objects 401, 403 in predetermined poses, such as the first training canonical pose 601 SMPL for the first training object 401 described above with reference to Fig. 6.

[0178] The next step in method 1100 is step 1110.

Step 1110: constructing a closed three-dimensional space around the SMPL pose estimates associated with a plurality of interpenetrating training objects, and forming on the inner surface of the closed three-dimensional space a plurality of viewpoints, each of which is oriented toward the center of the closed three-dimensional space.

[0179] In step 1110, according to some non-limiting embodiments of the present technology, the server 202 is configured to construct a closed three-dimensional space 520 around the first and second training estimates 501, 503 of the SMPL pose, containing the first and second viewpoints 508, 510, as described in detail above with reference to Fig. 5.

[0180] The next step in method 1100 is step 1112.

Step 1112: Construct from each viewpoint of the set of viewpoints a set of rays through the SMPL pose estimates for the set of interpenetrating training objects.

[0181] In step 1112, according to some non-limiting embodiments of the present technology, the server 202 is configured to construct from each of the first and second viewpoints 508, 510 a plurality of rays through the first and second training estimates 501, 503 of the SMPL pose, in particular a ray 525 emanating from the first viewpoint 408. In some non-limiting embodiments of the present technology, the rays in the plurality of rays are spaced from each other by the same distance. However, this is not required for all embodiments of the present technology and the rays in the plurality of rays may be distributed differently. In some other non-limiting embodiments of the present technology, the plurality of rays emanating from each of the first and second viewpoints 508, 510 comprises 2048 rays.

[0182] Furthermore, according to some non-limiting embodiments of the present technology, the server 202 is capable of defining a plurality of points on the beam 525, such as the point 530. In some non-limiting embodiments of the present technology, the plurality of points are uniformly distributed on the beam 525. In some non-limiting embodiments of the present technology, the server 202 is capable of defining 64 points on the beam 525.

[0183] The next step in method 1100 is step 1114.

Step 1114: determining, for a given point on a given ray, a canonical point for the canonical pose of the SMPL associated with each of the plurality of interpenetrating training objects.

[0184] In step 1114, according to some non-limiting embodiments of the present technology, the server 202 is configured to find, using formula (1), a canonical point for the first training canonical pose 601 SMPL associated with the first training object 401 and for the second training canonical pose SMPL associated with the second training object 403.

[0185] The next step in method 1100 is step 1116.

Step 1116: Generating a training dataset for a given machine learning algorithm from a set of machine learning algorithms.

[0186] At step 1116, in accordance with some non-limiting embodiments of the present technology, the server 202 is configured to generate a density training data set for training each machine learning algorithm of the plurality 210 machine learning algorithms for determining density.

[0187] In particular, for the first machine learning algorithm for determining the density associated with the first training object 401, the server 202 is capable of generating a first set of data for training the density determination, comprising a first plurality of digital objects for training the density determination, each of which comprises (a) coordinates of a canonical point in a coordinate system associated with the first training canonical pose 601 of the SMPL, and (b) a first training vector 505 of the SMPL pose.

[0188] Similarly, for the second machine learning algorithm of the plurality 210 machine learning algorithms for determining density, the server 202 is capable of generating a second set of data for training to determine density, comprising a second plurality of digital objects for training to determine density, each of which for a point 530 comprises (a) coordinates of a canonical point in a coordinate system associated with a second training canonical pose SMPL associated with a second training object 403, and (b) a second training vector 507 of the pose SMPL.

[0189] The next step in method 1100 is step 1118.

Step 1118: jointly training multiple machine learning algorithms to determine the boundary of a given object from a set of interpenetrating objects using the training datasets.

[0190] In step 1118, according to some non-limiting embodiments of the present technology, the server 202 is configured to use the density training data sets generated in step 1116 to train each of the plurality 210 density training machine algorithms.

[0191] In particular, at a given iteration of training, the server 202 is configured to (a) transmit each digital object for learning to determine the density of the first plurality of digital objects for learning to determine the density to the first machine learning algorithm for determining the density, whereby the first machine learning algorithm for determining the density generates a first predicted value of the signed distance field for the canonical point in the first training canonical pose 601 of the SMPL, and (b) transmit each digital object for learning to determine the density of the second plurality of digital objects for learning to determine the density to the second machine learning algorithm for determining the density, whereby the second machine learning algorithm for determining the density generates a second predicted value of the signed distance field for the canonical point in the second training canonical pose SMPL.

[0192] In addition, the server 202 is configured to apply a loss function for the signed distance field. According to some non-limiting embodiments of the present technology, the loss function for the signed distance field is configured to assign a penalty to the first and/or second predicted values of the signed distance field if (a) the predicted value of the signed distance field differs from a reference value of the signed distance field associated with a given machine learning algorithm for determining density of the plurality 210 machine learning algorithms for determining density, and (b) the predicted value of the signed distance field generated by the given machine learning algorithm for determining density is equal to the predicted value of the signed distance field generated by another machine learning algorithm for determining density of the plurality of machine learning algorithms. In some non-limiting embodiments of the present technology, the loss function for the signed distance field may be expressed by formula (2).

[0193] In addition, in some non-limiting embodiments of the present technology, the server 202 is configured to train a plurality 215 of machine learning algorithms for determining color based on the predicted values of the signed distance field generated by the plurality 210 of machine learning algorithms for determining density, as described in detail above with reference to Fig. 5.

[0194] In particular, as in training the plurality 210 machine learning algorithms for determining the density, the server 202 is capable of generating a set of data for training the determination of color for each machine learning algorithm of the plurality 215 machine learning algorithms for determining color. For the first machine learning algorithm for determining color associated with the first training object 401, the server 202 is capable of generating a first set of data for training the determination of color, comprising a first plurality of digital objects for training the determination of color, each of which, for a point 530, comprises (a) the coordinates of a canonical point for a first training canonical pose 601 of the SMPL, (b) a first training vector 505 of the pose of the SMPL and (c) a label comprising a color value for a pixel 540 on the adjusted training two-dimensional image 406 through which the ray 525 passes.

[0195] Similarly, for training a second machine learning algorithm associated with a second training object 403, from a plurality 215 of machine learning algorithms for determining a color, the server 202 is capable of generating a second set of data for training to determine a color, comprising a second plurality of digital objects for training to determine a color, each of which for a point 530 comprises (a) coordinates of a canonical point for a second training canonical pose SMPL, (b) a second training vector 507 of a pose SMPL and (c) a label containing a color value for a pixel 540 on the adjusted training two-dimensional image 406 through which the ray 525 passes.

[0196] In addition, at a given iteration of training the plurality 215 machine learning algorithms for determining the color, the server 202 is capable of transmitting to each of the plurality 215 machine learning algorithms for determining the color a digital object for learning to determine the color associated with the point 530, as a result of which each of the plurality of machine learning algorithms for determining the color generates a predicted color value for the canonical point associated with the corresponding training object. Then, at the same iteration of training the plurality 215 machine learning algorithms for determining the color, the server 202 is capable of (a) receiving predicted values of the signed distance field generated by the plurality 210 machine learning algorithms for determining the density for each point specified on the beam 525, (b) determining, based on the predicted values of the signed distance field for the point 530, the density values, and (c) determining, based on the density values and the predicted color values for all points on the beam 525, an integral color value for the beam 525.

[0197] Also, the server 202 is capable of jointly training a plurality 215 machine learning algorithms for determining a color by applying, at a given iteration of training the plurality 215 machine learning algorithms for determining a color, a loss function for a color capable of assigning a penalty to the predicted color values generated by the plurality 215 machine learning algorithms for determining a color for all points on the beam 525 if the intermediate integral color value differs from the color value of the pixel 540 from the label of the digital objects for learning to determine a color associated with the beam 525. In addition, as noted above, in such embodiments, the server 202 is capable of additionally training each of the plurality of machine learning algorithms for determining a density to determine the values of the signed distance field for the points on the beams by applying the loss function for a color.

[0198] Finally, after training each machine learning algorithm of the plurality 210 machine learning algorithms for determining the density and each machine learning algorithm of the plurality 215 machine learning algorithms for determining the color, the server 202 is able to use them to construct (render) a three-dimensional scene 902 containing the first and second three-dimensional models 1001, 1003 of the first and second objects 301, 303, as described in detail above with reference to Figs. 7-9. Then, upon receiving a three-dimensional animation request 212, as described in detail above with reference to Fig. 2, the server 202 is able to transmit the three-dimensional scene 902 with the first and second three-dimensional models 1001, 1003 as part of the three-dimensional animated video content 214 to the electronic device 204 for demonstrating the three-dimensional scene 902 to the user 206.

[0199] This completes the implementation of method 1100.

[0200] Thus, some non-limiting embodiments of the present technology make it possible to more accurately determine the boundary between the first and second objects 301, 303 and, as a result, to correctly process the overlapping regions between them, such as the overlapping region 305, due to which the color values of the first and second three-dimensional models 1001, 1003 are more accurately determined and these three-dimensional models look more realistic. It is believed that this contributes to improving the user 206 experience of viewing the three-dimensional animated video content 214.

[0201] It is obvious that not all technical effects mentioned in this document are subject to implementation in each specific embodiment of the present technology.

[0202] Possible changes and improvements to the above-described embodiments of the present technology may be obvious to a person skilled in the art. The preceding description is provided for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of the present technology is determined solely by the scope of the appended claims.

Claims (91)

1. A computer method for volume rendering of three-dimensional scenes, including training a given machine learning algorithm from a plurality of machine learning algorithms to determine the boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene, wherein the training includes: - obtaining from a given camera from a plurality of cameras a sequence of training two-dimensional images representing a plurality of interpenetrating training objects; - formation of a sequence of corrected training two-dimensional images based on sequences of training two-dimensional images obtained from multiple cameras; - forming, for a given training object from a plurality of interpenetrating training objects on a given corrected training two-dimensional image from a sequence of corrected training two-dimensional images, an SMPL pose estimate containing pose vertices forming the surface of the training object in the corresponding pose on the corrected training two-dimensional image, and an SMPL pose vector containing the values of a plurality of SMPL pose parameters characterizing the pose of the training object on the corrected training two-dimensional image; - obtaining for a given training object from a set of interpenetrating training objects a canonical pose SMPL containing canonical vertices that form the surface of the training object in a predetermined pose; - construction of a closed three-dimensional space around SMPL pose estimates associated with a set of interpenetrating training objects; - formation on the inner surface of a closed three-dimensional space of a plurality of viewing points, each of which is oriented in the direction of the center of the closed three-dimensional space; - construction from each viewpoint of a set of viewpoints of a set of rays through SMPL pose estimates for a set of interpenetrating training objects; - determining for a given point on a given ray a canonical point in the canonical pose of the SMPL associated with each of the set of interpenetrating training objects; - forming, for a given machine learning algorithm, from a plurality of machine learning algorithms, a training data set containing a plurality of training digital objects, wherein the given training digital object for a given point on a given ray contains the coordinates of a canonical point associated with this training object, and a pose vector SMPL associated with this training object; - joint training of multiple machine learning algorithms to determine the boundary of an object from a set of interpenetrating objects using training datasets by: - transmitting training digital objects associated with a given point to each machine learning algorithm of the plurality of machine learning algorithms, as a result of which each of the plurality of machine learning algorithms generates a predicted value of the signed distance field for the canonical point; and - applying a loss function for the signed distance field that is capable of assigning a penalty to the predicted value of the signed distance field generated by a given machine learning algorithm for a given point if the predicted value of the signed distance field generated by the given machine learning algorithm is equal to the predicted value of the signed distance field generated by another machine learning algorithm from the set of machine learning algorithms. 2. The method according to claim 1, wherein the closed three-dimensional space is a sphere constructed around a given training three-dimensional scene, and each of the plurality of viewing points is located on the inner surface of this sphere. 3. The method according to claim 1, wherein the plurality of viewpoints comprises two opposite viewpoints oriented in the direction of the center of the closed three-dimensional space. 4. The method according to claim 1, wherein the plurality of rays emanating from a given viewpoint from a plurality of viewpoints are spaced from each other by the same distance. 5. The method according to paragraph 1, additionally providing for finding on a given ray a set of points containing a given number of points uniformly distributed along this ray. 6. The method according to claim 1, wherein determining the canonical point involves using a skinning algorithm based on linear mixing using the SMPL model. 7. The method according to item 6, wherein the following formula is used in the skinning algorithm: , where x ^c is the canonical point associated with the training object; x ^d - a given point on the ray; B _i is the transformation matrix associated with the joint j _i of the training object and formed based on the SMPL pose estimate associated with the training object; n _b - the number of joints in the training object; w _i ^d is a function characterizing the distribution of weights in the skinning process, determined based on the coordinates of the SMPL pose estimate vertex associated with the training object, closest to the given point. 8. The method according to claim 1, wherein the loss function for the signed distance field is expressed by the following formula: , where P is the number of training objects in the set of interpenetrating training objects; - the number of unique pairs of training objects in a set of interpenetrating training objects; s _i ^k - the predicted value of the signed distance field for a given point, calculated by the first algorithm from the set of machine learning algorithms associated with the first training object from the set of interpenetrating training objects; s _i ^p - the predicted value of the signed distance field for a given point, calculated by the second algorithm from the set of machine learning algorithms associated with the second training object from the set of interpenetrating training objects. 9. The method according to claim 1, in which the transmission of training digital objects associated with a given point to each machine learning algorithm of the plurality of machine learning algorithms additionally initiates the formation by each of the plurality of machine learning algorithms of a vector representation of a signed distance field for a canonical point, and the training of the plurality of machine learning algorithms provides for joint training of the plurality of machine learning algorithms and a second plurality of machine learning algorithms for the purpose of determining color values for a plurality of interpenetrating objects displayed in a three-dimensional scene, wherein the training of the second plurality of machine learning algorithms includes: - obtaining at the current iteration of training from each of the set of machine learning algorithms the predicted value of the signed distance field for the canonical point associated with the training object; - determination of current predicted density values for canonical points based on predicted values of the signed distance field; - forming for another machine learning algorithm from a second plurality of machine learning algorithms another training data set containing another plurality of training digital objects, wherein the specified training digital object for a specified point on a specified ray contains coordinates of a canonical point associated with a training object associated with another machine learning algorithm, a vector representation of a signed distance field for a canonical point associated with this training object, and a label representing the color value of a pixel on a corrected training two-dimensional image associated with the specified ray; - transmitting training digital objects associated with a given point from each other training data set to each machine learning algorithm of a second plurality of machine learning algorithms, as a result of which each of the second plurality of machine learning algorithms generates a predictive color value for a canonical point associated with the training object; - determining, based on the predicted color values generated by each of the second set of machine learning algorithms and the current predicted density values generated by the set of machine learning algorithms for each point of a given ray, an intermediate integral color value for a given ray at the current training iteration; - application of a loss function for color that is capable of assigning a penalty to an intermediate integral color value if this value differs from the color value from the corresponding label. 10. The method according to claim 9, wherein determining the intermediate integral color value involves determining the intermediate integral color value according to the following formula: , where c _i ^p is the predicted color value for the canonical point associated with object p , generated by a given machine learning algorithm from the second set of machine learning algorithms; w _i ^p - the weight value determined for the predicted color value based on the current predicted density values; bg _r - background color associated with the beam. 11. The method according to paragraph 10, additionally providing for determining the weight value using the following formula: , where Δ x _i is the length of the segment between a given point and the next point on a given ray; σ _i ^p is the current predicted density value generated by a machine learning algorithm from a set of machine learning algorithms associated with a given training object, based on the value of the signed distance field. 12. The method according to claim 9, wherein determining the current predicted density values for the canonical point involves applying a normalized integral Laplace distribution function to each of the predicted values of the signed distance field. 13. The method according to claim 9, which, before forming another training data set, additionally provides for selecting from the set of points of the given beam a set of points for training a second set of machine learning algorithms in the areas of maximum density of the training object. 14. The method according to claim 13, wherein the selection of the set of points involves the use of a significance sampling algorithm capable of forming points on the beam that represent each of a plurality of interpenetrating training objects through which the specified beam passes. 15. The method according to claim 14, further comprising combining ray points representing each of a plurality of interpenetrating training objects through which the specified ray passes. 16. The method of claim 14, wherein the importance sampling algorithm is an opacity function. 17. The method of claim 9, further comprising using a plurality of machine learning algorithms to determine a boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene, including: - obtaining from a given camera from a plurality of cameras a working sequence of two-dimensional images representing a plurality of interpenetrating objects; - formation, based on working sequences of two-dimensional images obtained from multiple cameras, of a sequence of working corrected two-dimensional images; - formation for a given object from a set of interpenetrating objects on each working corrected two-dimensional image from a sequence of working corrected two-dimensional images of a working estimate of the SMPL pose, containing: - working pose vertices that form the surface of the object in the corresponding pose on the working corrected two-dimensional image; and - a working SMPL pose vector containing the values of a set of SMPL pose parameters characterizing the pose of an object in a working corrected two-dimensional image; - obtaining for a given object from a set of interpenetrating objects a working canonical pose SMPL, containing canonical vertices that form the surface of a given object in a predetermined pose; - construction of a closed three-dimensional space containing a set of viewpoints formed on its internal surface around working SMPL pose estimates associated with a set of interpenetrating objects; - construction from each viewpoint of a set of viewpoints of a set of rays through working SMPL pose estimates; - determination for a given point on a given ray of a working canonical point for the working canonical pose SMPL associated with each of the set of interpenetrating objects; - formation for a given machine learning algorithm associated with an object from a set of interpenetrating objects, a given working digital object containing the coordinates of the working canonical point associated with the corresponding object, and the SMPL pose vector associated with the corresponding object; - passing the working digital object to a given machine learning algorithm from a set of machine learning algorithms, as a result of which this machine learning algorithm determines the working value of the signed distance field for the canonical point; - determining whether the surface of a given object passes through a given point or not, based on the working values of the signed distance field at the canonical point, generated by a set of machine learning algorithms. 18. The method according to claim 17, in which the transfer of a given working digital object to a given machine learning algorithm from a plurality of machine learning algorithms additionally initiates the formation by this machine learning algorithm of a working vector representation of a signed distance field for a canonical point associated with the corresponding object, and also additionally provides for the use of a second plurality of machine learning algorithms for the purpose of determining a color value for each of a plurality of interpenetrating objects by: - determination of the density values at this canonical point based on all working values of the signed field of distances at the canonical point, generated by a set of machine learning algorithms; - formation of a working digital object for determining the color, containing the coordinates of the working canonical point associated with the corresponding object, and a working vector representation of the signed field of distances for this canonical point; - transferring to another machine learning algorithm from a second set of machine learning algorithms associated with the corresponding object, a working digital object for determining the color, as a result of which this other machine learning algorithm generates a color value for the canonical point of this object; - determining, based on the density values generated by a plurality of machine learning algorithms and the color values generated by a second plurality of machine learning algorithms for the canonical point, an integral color value for each ray; - application of integral color value for volumetric rendering of interpenetrating objects of a three-dimensional scene. 19. A server for volume rendering of three-dimensional scenes, comprising at least one processor and at least one physical memory storing executable commands, the execution of which by at least one processor on the server ensures training of a given machine learning algorithm from a plurality of machine learning algorithms for determining the boundary between a plurality of interpenetrating objects displayed in a three-dimensional scene, by: - obtaining from a given camera from a plurality of cameras a sequence of training two-dimensional images representing a plurality of interpenetrating training objects; - formation of a sequence of corrected training two-dimensional images based on sequences of training two-dimensional images obtained from multiple cameras; - forming, for a given training object from a plurality of interpenetrating training objects on a given corrected training two-dimensional image from a sequence of corrected training two-dimensional images, an SMPL pose estimate containing pose vertices forming the surface of the training object in the corresponding pose on the corrected training two-dimensional image, and an SMPL pose vector containing the values of a plurality of SMPL pose parameters characterizing the pose of the training object on the corrected training two-dimensional image; - obtaining for a given training object from a set of interpenetrating training objects a canonical pose SMPL containing canonical vertices forming the surface of the training object in a predetermined pose; - constructing a closed three-dimensional space around SMPL pose estimates associated with a set of interpenetrating learning objects; - the formation on the inner surface of a closed three-dimensional space of a plurality of viewing points, each of which is oriented in the direction of the center of the closed three-dimensional space; - constructing a set of rays from each viewpoint of a set of viewpoints through SMPL pose estimates for a set of interpenetrating training objects; - determining for a given point on a given ray the canonical point in the canonical pose of the SMPL associated with each of the set of interpenetrating training objects; - forming, for a given machine learning algorithm, from a plurality of machine learning algorithms, a training data set containing a plurality of training digital objects, wherein the given training digital object for a given point on a given ray contains the coordinates of a canonical point associated with this training object, and a pose vector SMPL associated with this training object; - joint training of multiple machine learning algorithms using training data sets to determine the boundary of an object from a set of interpenetrating objects by: - transmitting training digital objects associated with a given point to each machine learning algorithm from a plurality of machine learning algorithms, as a result of which each of the plurality of machine learning algorithms generates a predicted value of the signed distance field for the canonical point; - applying a loss function for the signed distance field that is capable of assigning a penalty to the predicted value of the signed distance field generated by a given machine learning algorithm for a given point if the predicted value of the signed distance field generated by the given machine learning algorithm is equal to the predicted value of the signed distance field generated by another machine learning algorithm from the set of machine learning algorithms. 20. The server according to claim 19, wherein the transmission of training digital objects associated with a given point to each machine learning algorithm from a plurality of machine learning algorithms additionally initiates the formation by each of the plurality of machine learning algorithms of a vector representation of a signed distance field for a canonical point, and the provision by the training server of a plurality of machine learning algorithms during the execution of executable commands provides for the joint training of the plurality of machine learning algorithms and a second plurality of machine learning algorithms for the purpose of determining color values for an object from a plurality of interpenetrating objects displayed in a three-dimensional scene, by: - obtaining, at the current iteration of training, from each of the set of machine learning algorithms a predicted value of the signed distance field for the canonical point associated with the training object; - determination of current predicted density values for canonical points based on predicted values of the signed distance field; - forming for another machine learning algorithm from a second plurality of machine learning algorithms another training data set containing another plurality of training digital objects, wherein the given training digital object for a given point on a given ray contains coordinates of a canonical point associated with this training object associated with another machine learning algorithm, a vector representation of a signed distance field for a canonical point associated with this training object, and a label representing the color value of a pixel on a corrected training two-dimensional image associated with the given ray; - transmitting training digital objects associated with a given point from each other training data set to each machine learning algorithm of a second plurality of machine learning algorithms, as a result of which each of the second plurality of machine learning algorithms generates a predicted color value for a canonical point associated with the training object; - determining, based on the predicted color values generated by each of the second set of machine learning algorithms and the current predicted density values generated by the set of machine learning algorithms for each point of a given ray, an intermediate integral color value for a given ray at the current training iteration; - applying a loss function for color that can assign a penalty to an intermediate integral color value if this value differs from the color value from the corresponding label.