RU2823750C1 - System and method for obtaining processed output image having user-selectable quality factor - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to computer engineering, namely to image processing. Disclosed is a system and a method for obtaining a processed image having a user-selectable quality factor, designed to reduce computations by adding early exit branches to the original main network, and dynamic switching of the computation path depending on how difficult it will be to visualize the output. System comprises an electronic device having a display, memory storing initial images and a set of generative adversarial networks (GAN), and a predictor. Each GAN is pre-trained and consists of N computing modules forming the main network, and multiple branches of earlier output, each of which is connected after each computing module. Predictor is an artificial neural network and is configured to predict the quality index of the processed image for each output from each branch of the earlier output based on the initial image which the user intends to supply to the input of the selected GAN.

EFFECT: reduction of computational loads and reduction of processing time.

21 cl, 15 dwg

Description

Field of technology to which the invention relates

The invention relates to the field of obtaining a processed image with a user-selectable quality indicator. Description of the prior art

Generative adversarial networks (GANs) are a class of generative frameworks that are based on the competition between two neural networks, namely a generator and a discriminator [23–25]. While the latter performs the classification task (decides whether the generated image is real or not), the former synthesizes an image from a target distribution.

Conditional GANs are a variation of the original framework [52]. Their architecture allows for additional information to be introduced, which is used to restrict the target space according to it. Consequently, it is possible to fit the network to the required conditions using, for example, a mask [35], a label [55], or text [60].

In recent years, there has been a growing popularity of neural head avatars as a practical method for creating head models.

They allow to reconstruct a face with a given expression and pose. Such models can be divided into two groups - models with hidden geometry [5, 15, 17, 81, 82] and models with a 3D a priori, for example a head mesh [16, 20, 28, 43, 43, 48, 85]. In addition, a number of works are focused on the entire human body, including the head and face, which can be divided according to the requirements for input data.

Some of them take just a few images [1], while others require video [7, 22, 31, 33, 40, 41, 58, 80, 86].

Generative DNNs are powerful tools for image synthesis, but they are limited by their computational load. On the other hand, given a trained model and a task, such as generating faces within a range of features, the quality score of the output image will be unevenly distributed across images with different features. This means that on some instances, the complexity of the model can be limited while maintaining a high quality score.

Recently, much attention has been paid to image synthesis using generative adversarial networks (GANs) [64, 72], with applications ranging from image-to-image transformation [35] to text-to-image rendering [21], generating neural head avatars, [17] and more. However, this approach suffers from high computational load when generating photorealistic images. The present invention is based on the observation that deep neural networks (DNNs) output images with varying but constant quality scores when they are conditioned by certain parameters. Since their

The expressiveness is not uniform across the set of possible generated images, therefore for some examples a simpler DNN may be sufficient to generate output with the required quality score.

On the other hand, approaches aimed at reducing the heavy computational load of DNNs have shown excellent results, significantly reducing the computational redundancy [2,13]. While strategies such as pruning [46, 53, 65] or knowledge distillation [4, 9, 26] produce DNNs with fewer parameters, the Early Exit (EE) configuration [42, 7 9] allows for dynamically changing the computational load and is therefore an ideal candidate for an image generation strategy aimed at outputting images with a constant quality score while avoiding the excessive computation caused by uneven rendering complexity.

Despite this, the implementation of EE strategies remains outside the scope of research on generative models. This is probably due to the fact that EE processes logits of intermediate layers, thereby limiting its scope of application to tasks where these layers are meaningful (e.g., classification), and excluding pipelines in which meaningful outputs are produced only at the last layer (e.g., in generative convolutional networks).

Early outputs are a computationally efficient strategy used primarily in classification problems [45,62]. They are characterized by adding outputs to a DNN that can be used to approximate the final output at a lower computational cost. Over time, early outputs have been rediscovered as a stand-alone approach, although they were originally implemented in architectures such as Inception [68] as a countermeasure to overfitting. This approach has also sometimes been called cascade learning [47,51,74], adaptive neural network [3], or simply branching [61]. Proposed implementations differ in three design choices: the architecture of the outputs, i.e., what type of layers to use to process the main network logits; where to attach outputs to distribute the computation evenly among them; and how to choose the computation path. The latter problem is often solved by introducing a trust mechanism and selecting a single output [42, 69, 87] or by reusing predictions for further computations [77,7 9]. To a lesser extent, learnable output policies have also been proposed [8, 12, 61].

As a method for efficiently using a single output during logical inference, a change in the computation path for each input is proposed [4 9, 54]. The basis for the emergence of the proposed approach was a method first applied in the field of neural architecture search: using the so-called predictor to speed up the performance evaluation of a given architecture [6, 76], as well as in natural language processing [18, 78], which was used for logical inference via early outputs for resource-constrained edge AI. [14].

Early image synthesis methods were based on searching for examples in large image databases [30, 34, 39, 44]. This is in contrast to modern DNN technologies, which rely on a large number of parameters to output photorealistic images. On the other hand, semiparametric generation has been proposed to exploit the strengths of both approaches [59, 66, 70]. In particular, the use of fragments, reminiscent of older methods, allows achieving high accuracy [27, 50, 71].

Storing a large image database poses a problem when it comes to querying it to retrieve a desired sample. When searching for guide images, it is necessary to use an algorithm that will quickly find a similar image or fragment. Caches [2, 9, 56] and, in particular, nearest neighbor search [27, 36, 83] are used for this purpose, where pre-trained models are used as visual feature extractors and the weights of the image encoders are fixed.

The proposed invention is based on the observation that DNNs produce images with different but constant quality scores when they are conditioned on certain parameters.

Since their expressiveness is uneven across the set of possible generated images, therefore, for some examples a simpler DNN may be sufficient to generate output with the required quality score.

ESSENCE OF THE INVENTION

A system is proposed for obtaining a processed image having a quality indicator selected by the user, containing:

- an electronic device, a memory storing input images and a set of generative artificial neural networks (GAN), wherein the electronic device is configured to perform artificial neural network operations;

- each GAN is configured to allow the user to select from a set of GANs stored in memory to obtain an image with a predetermined quality score, and each GAN is pre-trained and consists of:

N computing modules that form the core network, and

a plurality of earlier output branches, each of which is attached after each computing module, except for the last computing module of the main network, wherein each earlier output branch contains as many computing modules as there are left in the main network after the attachment point of the earlier output branch before the output of the main network, wherein each computing module of each earlier output branch performs the same function as the corresponding remaining computing module in the main network, and the computing resource of each earlier output branch is less than the computing resource of the corresponding remaining computing modules of the main network before the output of the main network,

and the closer a specific branch of an earlier output is to the output of the main network, the higher the computational resource for obtaining the output image generated by a specific branch of an earlier output, and

where the quality indicators of the output image generated by the branches of the earlier output differ from each other;

- a set of predictors, each of which is an artificial neural network, wherein each predictor is created and pre-trained for a specific GAN stored in memory, the predictor is configured to predict a quality score of the processed image to be output from each branch of an earlier output of a specific GAN based on an input image that the user intends to feed into the specific GAN;

wherein the GAN is configured to output processed output images from the branch of the earlier output that generates a processed output image having a quality score that most closely matches the quality score selected by the user.

Moreover, the closer a specific branch of an earlier output is to the output of the main network, the higher the quality indicator of the processed image generated by this specific branch of an earlier output.

Each early exit branch is pre-trained on an adversarial loss function with copies of the original main network discriminator.

The system may further comprise a database storing guide data, wherein the main network is further configured to retrieve guide data from the database for source images input to the main network, wherein the guide data is connected to data from the computing module before the point of attachment of the earlier output branch, and the data obtained after the connection is fed to the earlier output branch for further processing. In other words, the system may further comprise a database storing guide data, wherein the GAN is further configured to retrieve guide data from the database and connect them with source image data input to the GAN, wherein the guide data is connected to data from the computing module before the earlier output branch, and the connected data is fed to the earlier output branch for further processing. The guide data may be guide features. The guide data may be images, image fragments, features, or feature fragments (intermediate tensors). The fragment size may vary and be as large as the source image or feature.

A method is proposed for obtaining a processed image having a user-selectable quality indicator using the proposed system, consisting in the fact that:

select, by means of the user, from memory the input images, the pre-trained GAN and the pre-trained predictor corresponding to this GAN;

select a quality indicator for the processed output images by the user;

feeding at least one of the input images to a corresponding pre-trained predictor;

predicting, based on at least one of the supplied input images, by means of a corresponding pre-trained predictor, a quality indicator for the processed output images that will be generated by each branch of the earlier output;

selecting one branch of the earlier output that generates a processed output image having a quality score that most closely matches the quality score selected by the user;

transform input images into input data that can be processed by GAN;

feed the input data into the GAN with the earlier output branch selected for processing;

process the input data through a GAN with the earlier output branch selected;

receive processed output images at the output of the selected branch of the earlier output.

Alternatively, according to the proposed method:

the user selects from memory at least one source image, a pre-trained GAN and a corresponding pre-trained predictor,

feed the source image into the predictor;

predict, via the predictor, a quality score for each image generated by the main network and each branch of the earlier GAN output;

select a quality indicator based on the forecast by the user;

transform input images into input data to be processed by the GAN;

feed input data into the GAN with an earlier output branch for processing;

process input data through GAN and

display on the display of the electronic device the finished processed image, output from the output of the main network or one of the branches of an earlier output, which provides the selected quality indicator.

According to the method, the finished processed output images, derived from the output of the main network or one branch of an earlier output, which provides the selected image quality indicator, are additionally stored in memory.

The finished processed output images can be

displayed on the display of an electronic device or transmitted to other processing facilities for further use.

The closer a particular branch of the earlier output is to the output of the main network, the higher the quality score of the processed image generated by a particular branch of the earlier output. The quality score is expressed in FID units.

A method for obtaining a processed image with a quality indicator selected by the user is also proposed, using the proposed system, which consists in the fact that:

select from memory by the user the input data, the pre-trained GAN and the pre-trained predictor corresponding to this GAN,

select a quality indicator for the processed output images by the user;

feeding at least one of the input images to a corresponding pre-trained predictor;

predict based on at least one of the input images supplied by means of the corresponding

pre-trained predictor quality score for the processed output images that will be generated by each branch of the earlier output;

selecting one branch of the earlier output that generates a processed output image having a quality score that best matches the user-selected quality score;

transform input images into input data to be processed by the GAN;

feed input data into the GAN with an earlier output branch for processing;

process the input data through a GAN with the earlier output branch selected,

while processing:

- selecting from the database guide data corresponding to guide data for at least one of the input images,

- connect the guide data to the data output from the computing module before the selected branch of the earlier output,

- feed the connected data into the selected branch of the earlier output or into the main network for further processing;

receive processed output images at the output of the selected branch of the earlier output. According to the method,

additionally, they store in memory the finished processed output images derived from the output of the main network or one branch of an earlier output that provides the selected image quality indicator.

The finished processed output images can be displayed on the display of an electronic device or transferred to other devices for further use.

The closer a particular branch of an earlier output is to the output of the main network, the higher the quality score of the processed image generated by that particular branch of an earlier output. The quality score can be expressed in FID units.

The guide data can be images, image fragments, features, or feature fragments.

Alternatively, according to the proposed method:

selecting from memory by the user at least one source image, a pre-trained GAN and a corresponding pre-trained predictor,

feed at least one source image to the predictor;

predict, via the predictor, a quality score for each image generated by the main network and each branch of the earlier GAN output;

select a quality indicator based on the forecast by the user;

transform input images into input data that can be processed by GAN;

feed input data into the GAN;

process input data using GAN, and during processing:

- select from the database the guide data corresponding to the guide image,

- connect the guide data with the data from the computing module before the branch of the earlier output,

- feed the connected data into the earlier output branch for further processing;

store in memory the finished processed image, derived from the output of the main network or one branch of an earlier output that provides the selected quality indicator.

The finished image can, for example, be displayed on the display of an electronic device. The user can save these images in advance based on the original images selected from memory.

A machine-readable medium is proposed that stores instructions for performing any of the proposed methods by an electronic device.

At least one of the plurality of computing modules may be implemented by an artificial intelligence (AI) model. The AI-related function may be performed by non-volatile memory, volatile memory, and a processor.

The processor may include one or more processors. Currently, the one or more processors may be general-purpose processors such as a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), a visual processing unit (VPU), and/or a special processor for AI such as a neural processing unit (NPU), etc.

These one or more processors control the processing of input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in non-volatile memory and volatile memory. The predefined operating rule or AI model is provided by learning or training.

In this context, provision by training means that by applying a training algorithm to a set of training data, a predetermined operating rule or AI model with the desired characteristic is created. The training may be performed in the device itself, in which the AI is executed according to the embodiment, and/or may be implemented on a separate server/system.

An AI model may consist of multiple neural network layers. Each layer has multiple weight values and performs the layer operation by computing the previous layer and operating on the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

In addition, the proposed method, performed by an electronic device, can be implemented using an artificial intelligence model.

An artificial intelligence model can be obtained by training. In this context, "obtained by training" means that a predetermined working rule or an artificial intelligence model that ensures the performance of a desired function (or goal) is obtained by training a base artificial intelligence model on a plurality of parts of training data using a training algorithm. An artificial intelligence model can include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of network parameter values and performs neural network computations by computing between the result of the computation of the previous layer and the plurality of parameter values.

Visual understanding is a method of recognizing and processing things similar to human vision, and it includes, for example, object recognition, object tracking, image retrieval, person recognition, scene recognition, 3D reconstruction/localization, or image enhancement.

Predictive reasoning in artificial intelligence is a method of logical reasoning and making predictions by identifying information, which includes, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendations.

Brief description of the drawings

The above and/or other aspects will become more apparent from the description of exemplary embodiments with reference to the accompanying drawings, in which:

Fig. 1 illustrates an example of the proposed system for obtaining a processed image having a user-selectable quality indicator.

Figure 2 illustrates the relationship between the quality metric (expressed in FID units) and the computations for all branches for different scaling factors of the OASIS implementation using the guideline database.

Fig. 3 illustrates examples of branch outputs for the OASIS pipeline.

Fig. 4 illustrates the distribution of computations among branches of the OASIS core network for a set of given LPIPS thresholds.

Fig. 5 illustrates example branch outputs for the MegaPortraits pipeline.

Fig. 6 illustrates the relationship between the quality score (expressed in LPIPS units) and the computations for all branches.

Fig. 7 illustrates the distribution of computations between branches of the MegaPortraits main network for a set of given LPIPS thresholds.

Fig. 8 illustrates the OASIS pipeline, distributing images into different branches depending on their head rotation angle (in the OASIS pipeline).

Fig. 9 illustrates a comparison between the distribution of quality scores of individual OASIS branches and the distribution of quality scores obtained using the predictor (P).

Fig. 10 illustrates a comparison between the quality metric and the OASIS pipeline calculations.

Fig. 11 illustrates a comparison of the performance of different scaling factors (in the OASIS pipeline).

Fig. 12 illustrates a comparison of the performance of different scaling factors (in the OASIS pipeline).

Fig. 13 illustrates a comparison of the performance of different scaling factors (in the MegaPortraits pipeline).

Fig. 14 illustrates a comparison of the impact of the database on the quality score distribution for different scaling factors (in the OASIS pipeline).

Fig.15 illustrates the distribution of images sent to different branches depending on their head rotation angle (in the MegaPortraits pipeline).

Detailed description

The proposed invention accelerates the operation of generative neural networks, allows neural networks to work faster or reduce energy consumption, while ensuring the required output frequency. This can reduce the power consumption of electricity, as well as the heating of devices used to operate a neural network, such as discrete graphics cards for PCs, servers or laptops, or SoC smartphones.

The present invention can be applied to any neural networks intended for generating images based on a latent vector, the architecture of which consists of a plurality of blocks. The blocks 11, 12, 13, 14 shown in Fig. 1 are modules of the neural network architecture. Each block performs the function of a small neural network, transforming inputs in accordance with multiplication by weights, adding biases and applying a nonlinear activation function and other operations. Specifically, the present invention can be applied to any multilayer neural networks, including any GAN.

The invention allows to accelerate the operation of neural networks or reduce energy consumption, while providing the required output frequency in real time. The invention also allows to reduce the load, energy consumption and heating of devices that are used to implement this neural network, and the invention is used in such devices as a discrete video card for a PC, server or laptop, as well as SoC for a smartphone, and in any electronic device that has a central processor. In addition, a machine-readable medium is proposed, on which a computer code is stored for performing the proposed method by a computer or a suitable electronic device. In this case, the electronic device can contain a display, memory for storing images and a set of artificial neural networks (ANN), and the electronic device is also made with the possibility of performing artificial neural network operations.

The proposed method allows to reduce the amount of computation by adding so-called early exit branches to the original architecture (the main network) and dynamically switching the computation path depending on the complexity of the output rendering.

The basic network is any generative model that uses a decoder, namely a neural network that takes a latent vector and produces an output, usually after processing the signal through several layers. The complexity is determined by the quality score of the original image (the input dataset). The worse the quality score of the original image, the higher the complexity. The quality score is measured by common image metrics (FID, LPIPS). Several paths are created with different numbers of parameters and a neural network called a predictor is trained to predict what image quality score will be obtained from each path. If the user specifies a lower quality score when generating images, the predictor suggests the simplest way to satisfy this requirement.

It should be noted that the neural network works with data tensors, so the images to be input into the neural network and processed in it should be converted into data tensors. The operation of converting images into tensors is well known and is not considered in this description.

Thus, in the further description, the term "input image" implies the use of the term "data tensors" obtained as a result of transforming the input images for further processing by the neural network.

The proposed method allows outputting images with a user-defined lower predefined output image quality indicator and can be applied wherever there is a model generating images using a decoder. As examples, the application of this method for generation based on a semantic map (known from the prior art) and cross-reproduction of facial expressions is considered. Image generation based on a semantic map is the task of creating an image with a list of all pixels belonging to a certain class as input. For example, for a street photo, the semantic map contains a list of all pixels that should contain a road, trees, buildings, etc. Cross-reproduction of facial expressions is a task in which two portraits are input, one defining a person, and the second one - the expression and head position that must be given to the first person. The present invention allows to reduce by half the computations of the neural network (compared to the original main network) for the quality indicator LPIPS (Learned Perceptual Image Patch Similarity) ≤ 0.1 of the image generated by the neural network.

The proposed method uses an "early exit" strategy for image synthesis, which dynamically directs the computation flow to the required early exit branches depending on the complexity of the images, which reduces the redundancy of the computations while maintaining the required quality indicator predetermined by the user.

The output branches of the early output are connected to the original basic independent artificial neural network (called the main network) which consists of an input, computational modules and an output), as shown in Fig. 1. The computational modules of the early output are denoted as corresponding numbers in ascending order of quality index (marked as 1, 2, 3 in circles, with 4 in the circle indicating the exit from the main network). These computing modules are built on a lightweight version (i.e. with fewer parameters) of the computing modules that make up the architecture of the main network, their complexity can be adjusted in accordance with the required quality indicator predefined by the user. The fewer parameters in the network, the less calculations will be performed, the output quality indicator will suffer from this, and therefore outputs are created taking into account the balance between the loss of the quality indicator and the acceleration of calculations.

Fig. 1 shows an example of the proposed system for obtaining a processed image having a quality indicator selected by the user (a predetermined quality indicator). The proposed system contains artificial neural networks (ANN), which also include GANs. The memory of the electronic device contains a plurality of GANs for different tasks. Each GAN stored in the memory contains N computing modules forming the main network, and a plurality of branches of an earlier output, each of which is attached after each computing module of the main network, with the exception of the last computing module of the main network. Each branch of an earlier output contains as many computing modules as there are left in the main network after the attachment point of the branch of an earlier output before the output of the main network. Each computing module of each branch of an earlier output performs the same function as the corresponding remaining computing module in the main network. The computing resource (number of computations) of each branch of an earlier output is less than the computing resource of the corresponding remaining computing modules of the main network before the output of the main network.

For example (see Fig. 1), the main network generator consists of computational modules l ₁ through l ₄ . Each branch of the earlier output is attached after each computational module of the main network (outputs 1, 2, 3 in circles in Fig. 1). In Fig. 1, three branches of the earlier output are shown, which adds the early outputs 1, 2, 3. Each branch of the earlier output contains so many computational modules , how many of them remain in the main network after the connection point of the branch of the earlier exit before the exit of the main network. Each computing module Each branch of the earlier output performs the same function as the corresponding remaining computing module in the main network. The closer a particular branch of the earlier output is to the output of the main network, the higher the computing resource for obtaining the processed image and the quality indicator of the processed image generated by a particular branch of the earlier output.

As shown in Fig. 1, each branch has a different depth (depth is the number of computing modules) and consists of lightweight computing modules , that is, computing modules that contain fewer parameters than the computing modules of the main network, although their structures are similar.

For example (this example does not limit the scope of application of the invention), the input data for the main network are 2 images selected by the user from the memory of the electronic device. One image, the identity of which must be saved, and the second image, the facial expression of which must be saved. At the output, it is necessary to obtain the first identity with the second facial expression. The memory contains a pre-prepared main network with connected pre-trained early output branches, at the output of which the quality indicator of the resulting image is lower than the quality indicator of the image obtained during the operation of the main network, and the computing resource of each branch is significantly less than the computing resource of the original main network. The user retrieves from the memory of the electronic device a GAN, i.e. a main network with connected early output branches, wherein this main network is suitable for processing images selected by the user from the memory. In this case, the selected main network can perform processing with the receipt of a finished image with the maximum quality indicator, and to obtain an image with the maximum quality indicator, the computing modules of the main network with a high computing resource are used.

The memory of the electronic device also contains a set of predictors, each of which is an artificial neural network. Each predictor is generated and pre-trained for a specific GAN stored in the memory. The predictor is configured to predict the quality score of the processed image for each output of each branch of the earlier output of the specific GAN based on the original image that the user intends to feed into the specific GAN. The predictors are grouped with the corresponding main network with the branch of the earlier output attached in the memory.

For example, two images are selected from the memory of an electronic device by the user. One image is the source, this is the image of the person whose appearance the user wants to save (the person on the left (for example, images (a) or (b)) in Fig. 1), the other image is the driver, this is any other photograph of the face whose expression the user wants to convey (for example, this is the image of the person on the right (images (a) or (b)) in Fig. 1)). That is, in the example shown in Fig. 1, the images of the person on the left (images (a) or (b)) are taken as the personality that is displayed with the facial expression of the person on the right (images (a) or (b)).

Figure 1 shows an example of the computation path for two separate inputs:

first input (image a): source - person on the left (image a), driver - person on the right (image a);

second input (image b): source is the person on the left (image b), driver is the person on the right (image b).

Each image (a) and (b) is processed separately, in Fig. 1 they are shown together for illustration purposes only.

The user selects an image(s) from memory. An already prepared and pre-trained GAN (the main network with early output branches attached) and the corresponding pre-trained predictor are already stored in memory to solve such an image processing task. Therefore, the user selects a suitable GAN along with the predictor from memory after selecting the images.

The user selects the desired quality score for the output image to be processed. The image of the person on the right from images (a) is fed to the predictor. The predictor predicts the quality score of the image generated by the main network and each earlier output branch of the selected GAN. When processing images (a) using the GAN, the earlier output branch that generates the processed output image having the quality score that most closely matches (or matches) the quality score selected by the user will be used.

If the user initially requires an image with the highest quality score, the main network without early exit branches will be used.

For example, in Fig. 1, for the maximum quality score requested by the user for images (a), the entire main network will be used without using any branches of the earlier output (output 4). For the quality score requested by the user for images (b), the main network with the earlier output branch 2 will be used, since according to the predictor's predictions, the quality score of the output image of the earlier output branch with output 2 in this case best matches the quality score selected by the user.

Examples of output images after l ₁ or l ₃ are not shown.

For the examples (a) and (b) given, the task of the neural network is to replace the image of the second person's head (the driver) in the second image or video with the head of the first person (the source). If a video is fed, the network replaces the driver's head with the source's head not only for one image, but throughout the entire video.

The finished processed image, output from the output of the main network or one of the branches of an earlier output that provides a given quality indicator, is displayed, for example, on the display of an electronic device. The more complex the image, the more calculations are required to obtain the required predetermined quality indicator.

The predictor is a DNN pre-trained on the outputs of the proposed branches and is able to specify the output required to output an image with a pre-defined quality score. The predictor is trained with a teacher setting the loss function to be the minimum mean squared error between its prediction and the actual quality score. The predictor is trained on examples and can predict the quality score of the output of all early outputs attached to the main network for a given input image. In doing so, output 4 is assigned to the more complex image (a) to maintain the desired quality score. And for the lower input image (b) (solid line), only output 2 is required to maintain the desired user-defined quality score.

During the operation, the driver image is passed through the predictor and the serial numbers of the early outputs are obtained, which, according to the predictor's forecast, can provide the required predetermined output image quality indicator specified in advance. The blocks (for example, 11,12,13,14 in Fig. 1) of the main neural network (the main network), to which the proposed method is applied, do not require modification. The early output branches provide an earlier output of the final image from the network than the usual output from the main network without early output branches.

The image quality score is calculated using conventional measures (LPIPS, FID), and the predictor is an auxiliary network trained on examples and a predefined quality score of the generated image.

images.

Thus, after the user provides input image data, the predictor calculates what generation quality score each output will have. This allows the fastest (computationally) output to be selected from those that satisfy a given quality score condition.

The predictor is a completely independent neural network. Its output is a list of image quality scores generated by all early outputs for a given input. The predictor is used in the proposed method to select the output with the quality score that best matches the user-selected quality score. The predictor defines the output quality score as the LPIPS metric (known in the art).

The number of computational modules (i.e. depth) of early exits varies depending on the number of main network modules remaining after the early exit joins it. This ensures sufficient processing of the intermediate logits of the main network, and the computations are faster, since the computational modules of the early exit have fewer parameters than the computational modules of the main network. It should be noted that the number of computational modules of the early exit is equal to the number of computational modules remaining in the main network, i.e. unclaimed computational modules of the main network. For example, if there are 4 computational modules left from the point of joining the early exit to the end of the main path, then the early exit will have 4 computational modules.

In this case, the following condition is supported: the computational resource of the output of the early output branch in sum with the computational resource of the previous outputs in the main network block diagram is always less than the computational resource of the early output with a higher ordinal number. This condition is necessary in order to have a set of outputs with an increasing number of calculations and, accordingly, with a better quality indicator.

The proposed system may additionally contain a database storing guide data (examples), from which guide examples are extracted, having, for example, a human image that matches the human image in the input source image (the human on the left in images a or b in Fig. 1), and are fed to each branch. In this case, the pose of the human in the guide image is closest to the pose of the human in the input driver image, but has the appearance of the source (the human on the left in images a or b in Fig. 1). For one task, one example is extracted, i.e. one example per pass. The example database is formed in advance for each task, for example, by the user.

For both examples (a) and (b) in Fig. 1, the guide image of the person on the left (images a or b) is retrieved from the database to improve the generation quality score. To improve the generation quality score, the guide data is concatenated with data from the computational module before the earlier output branch, and the concatenated data is passed to the earlier output branch for further processing.

For example, if a user wants to get his own image with the pose of another person depicted in another image, he first creates a database of guiding examples of his own photographs in different poses (photographs from different angles), then feeds any image of the user (source image) and an image with another person (driver image) to the input of the neural network with early outputs. During the operation of the neural network with early outputs, a guiding example is selected from the database of guiding examples created by the user from the user's images, in which the user is depicted in a pose closest to the pose of the driver image. If a user wants to receive a video with his own images (source images), but with poses of another person (driver images), then the neural network with early outputs processes each image from the video sequence separately, extracting from the database of guide examples for each frame the image of the user (source image) with a pose closest to the pose of the person in the corresponding frame (driver image) of the video sequence currently fed to the input of the neural network with early outputs.

The presence of a database of guiding examples provides an increase in the quality factor for the branches of an earlier output due to a small amount of memory and computations, and thus harmonizes the output quality factor at the outputs. This is very convenient for installations in which real-time rendering is required and guiding examples can be easily provided, for example, when creating a neural avatar. This refers to the task of creating an avatar, i.e. a virtual image of a person. For example, a user wants to superimpose another personality on his face during a digital conference in real time, while preserving all his reproducible facial expressions. In this case, the user can take a picture of his face from different angles in advance and upload it to the database. This will provide a significant help in generation, but is not an absolutely necessary step.

The proposed method can be applied to both untrained and already trained models, but requires additional training for newly introduced components.

The main network is always fixed, and the predictor specifies only the output that will produce the quality metric required by the user. It should be noted that when GAN (the main network and the earlier output branches) works, only the earlier output branch that produces the output image of the required quality (the highest

corresponding to the quality indicator selected by the user) at the output. To do this, any suitable known switching mechanism is implemented in the code, and all other early exit branches that are not required to perform the selected early exit are not used.

The proposed method can be applied to any generation tasks, it is only necessary to have a generator, i.e. a neural network that creates images from a hidden vector. The main result of the proposed method can be summarized as follows: this method can be easily applied to existing and trained generative models containing a generator (the main network), i.e. a neural network that creates images from a hidden vector.

The proposed method enables outputting images with a user-defined lower quality score threshold by routing lighter images along shorter computational paths, and the main gain in terms of computational savings per quality score loss is 1.2×10 ³ and 1.3×10 ³ GFLOPs/LPIPS for these two applications, respectively.

In other words, the proposed invention allows to reduce the number of calculations at the expense of the loss of the quality indicator. The more calculations are reduced to the "unit of the quality indicator" (LPIPS measure), the better.

GANs consist of two competing DNNs: a generator G and a discriminator D. The generator G is designed to synthesize arbitrary images when given a random low-dimensional feature vector: G : z → g, where z is the input and ag is the generated image. The discriminator D learns to distinguish the distribution of the generated images. from one of the original examples . Their goals can be generally represented as a minimax game (a minimax game is known from the state of the art and means a decision rule for minimizing possible losses from those that the decision maker cannot prevent in the worst-case scenario):

Where - the loss function that the weights of the generator G must minimize and the weights of the discriminator D must maximize;

- expected value expressions when a random variable x with probability distribution p is selected; D(x) and G(z) are the outputs of the discriminator D and the generator G.

By providing a condition c (for example, in the form of labels) to both the generator and the discriminator, the former can learn to synthesize images from the subspace p _g : , where G(x) is the generator output; p _z and p _g are the probability distributions of the input noise and output images, and c is the matching parameter.

Any GAN generator consists of a series of convolutional modules, denoted l _i . The output of each module, namely , is a candidate for early output, but it is not a rendered image. For this reason, it must be processed through a series of additional convolutions before an image can be extracted from it. These The new convolutional computing modules make up what is called a branch, and these are the early outputs.

For a main network built from N computing modules, after the computing module k a branch of length N - k is added. The computing modules of the branches are less complex than the modules of the main networks, their width, i.e. the number of channels, is reduced. Consequently, from the output of each branch an image is extracted that is rendered with fewer computations than the image extracted at the output of the main network. Each branch of the early output is pre-trained on an adversarial loss function with copies of the original discriminator of the main network.

In the inference stage, given a set of trained branches, each image can be synthesized through a different output. Given a quality score, it is necessary to be able to select only the branch that achieves it by performing the least possible computation. There is a single quality score threshold (i.e., a predefined quality score specified by the user) for all branches, and it is necessary to select the branch that will output images with an equal or higher quality score (i.e., a quality score that most closely matches the quality score selected by the user) while performing the least possible amount of computation. For this purpose, a predictor P consisting of convolutional and fully connected layers is used. The proposed predictor is supervised by using the input terms c as training examples and the LPIPS score vectors S for the images generated by the branches as labels. It should be noted that learning to create a semantic map and learning to create a cross-reproduction of facial expressions are no different.

where c are the matching parameters, S is the LPIPS score for the image generated with the above parameters, and P(z, c) is the predictor output.

Thus, by feeding input to a trained predictor, one can obtain an estimate of the quality score of the output of each branch and hence use this information to direct the computational flow to the output that performs the least amount of computation while maintaining the quality score threshold.

To further improve the synthesis quality index, it is necessary to move from a purely parametric method to a semi-parametric one, in which the generation process is guided by data fragments, i.e. data extracted from a database. The data processed by the main network computing modules located before the early output branch computing modules are underprocessed tensors. To further improve the synthesis quality index,

Underprocessed tensors are concatenated with data chunks before being fed into the early exit branch.

This provides a more noticeable increase in quality score on the earlier outputs, which are the fastest but suffer the most from quality score degradation due to their smaller number of parameters. By adding a moderate amount of memory and computation, better results are achieved, which harmonizes the output quality score of the different branches.

For example, a set of tensor pairs called key-value pairs are stored in the database as guide data. In this case, the guide data is generated by the main network when forming the database. When the image generation process occurs, the guide data is already in the database and is retrieved from it during the GAN operation when creating an image. The keys are obtained by applying all the trained layers of the main network to the original data up to the first early output branch and cutting the resulting tensors, called guide features, into non-overlapping fragments. In other words, the keys are obtained as follows: the original image data is fed to the main network, the output before the first early output branch is split into fragments, and these fragments are the keys. The values are obtained by applying the trained layers of the main network to the original images and cutting the obtained features into data fragments and dividing them into N fragments. It was observed that the best result is achieved by applying the trained layers of the main network up to its middle to the original images. Based on the input data, the database is searched for the most similar images from the images stored there. The full image is searched for in the data chunks. To do this, the inputs are processed by several layers of the original main network to reduce the dimensionality of the input images and speed up the search for similar images. When similar chunks are found, the found chunk is sent to the early output branch.

During inference, the main network processes each input up to the level preceding the first branch. The obtained features are taken, cut into chunks of data, and for each chunk of data, the closest key is searched for in the database. After extracting the values corresponding to all the chunks of data, they are glued together, and the obtained features are connected to the input of each branch. Given all the keys of the auxiliary data (in Fig. 1, the image of the person on the left (images a or b) for the most similar image, the corresponding processed data are fed to the input of the early output. This is done to improve the quality indicator of the generation, which especially helps the early outputs located closer to the beginning of the main network.

To quantify the success of the proposed method, we introduce a simple measure of saved computations. We use GFLOPs (floating point operations) and LPIPS [84] as units of measurement, respectively. For example, when cross-reproducing facial expressions, an average gain in quality score of 1.3×10 ³ GFLOPs/LPIPS is achieved, which means that reducing the quality score threshold by +0.01 LPIPS will lead to a decrease of 13 GFLOPs.

The proposed method can be applied to a variety of DNNs for various synthesis tasks. To demonstrate its versatility, we apply it, for example, to two different image synthesis tasks:

1) outdoor photo synthesis starting from a semantic label map, using the OASIS architecture as the backbone [63];

2) cross-reproduction of facial expressions, synthesis of neural head avatars starting from an image that is the target expression and avatar position, based on the MegaPortraits architecture as the main network [17].

In the first example, the pipeline is implemented using the OASIS model as the backbone [63]. The OASIS generator consists of 6 SPADE ResNet modules [57], which make up the computational units l _i , i ∈ of the backbone [1, 6]]. Four branches are added, one after each backbone module l ₁ through l ₄ . The computational units branches are also modules of SPADE ResNet, and their the length varies to save, where len is the number of computational units of the earlier output, k is the number of outputs. That is, if the number of outputs k is 2, then the number of computational units of the earlier output branch is 4.

The early exit branch computation modules are a lightweight version of the main network computation modules, since their width, i.e. the number of channels, is reduced by setting the scaling factor (SF) s=1/2, 1/3, 1/4 to reduce computations.

Thus, a total of 5 compute routes are created for each scaling factor for the main network of the OASIS model, their GFLOPs (floating point operations) are listed in Table 1. GFLOPs determines the number of operations performed in one run. The more parameters, the more operations, since these parameters are multiplied together, and so on. Different SFs reduce the number of parameters in different ways. BB is the main network listed in the table for comparison, it is not affected by the quality metric. SF is the compression factor; if the original compute module has N channels, the compression in noms will be N*SF. 64 is an arbitrary number. Channels are part of the convolutional network architecture.

Table 1 shows the comparison of GFLOPs of all 5 compute paths through branches 1-4 and the OASIS main network (BB, far right column). Different rows correspond to different scaling factors (SF). As can be seen from the table, SF does not affect all compute modules equally, since the minimum number of channels is set to 64, after which no further scaling is set. It should be noted that the number 64 is set arbitrarily and changes in the following tests. Channel is a standard term, RGB images have 3 channels, convolutional networks create other channels depending on their architecture, in general, it is a network parameter that needs to be reduced.

To implement the proposed method, all branches and the predictor are trained. Each branch is trained by specifying an adversarial loss function, as in equation (1), obtained by competing with copies of the OASIS discriminator. Along with this, VGG [37] and LPIPS [84] losses are specified using the image synthesized by the main network as a reference.

Where - total loss consisting of the original loss , as well as VGG and LPIPS losses . α and β are hyperparameters chosen to equalize the contribution of the loss; the overall learning rate is set to 4×10 ^-4 , and the coefficients are set to α = 10 and β = 5 to equalize the contribution of the loss. The discriminators retain their original losses. The generator and discriminators are trained using Adam optimization [89] with β1 = 0, β2 = 0, 999. These computations were performed using distributed data from the PyTorch library [10] in parallel on two NVIDIA P40 GPUs with batch = 2 and lasted approximately 6 days. The results are presented in Tables 2 and 3.

Table 2 presents the quantitative results for the OASIS pipeline. The minimum number of channels is 64. This minimum number of channels sets a lower limit on the quality score. For three different SF scaling factors (first column), the pipeline was tested with and without a bank, indicated in the Bank column by a check mark (with bank) or a cross (without bank). Four columns, one for each branch, contain the FID (initial Frechet distance, a measure of quality score) and mIOU (average intersection over connection, a measure of quality score); at the bottom, these two values are presented for the main network. From Table 2, we can conclude that adding a bank provides a higher improvement in quality score for the early outputs than for the late outputs. It also clearly shows how the compression ratio affects the quality score. Any compression will do, and the choice depends on the quality score required. The lower the FID, the higher the quality score. The higher the mIOU, the higher the quality score.

From Table 2, we can conclude that adding the database provides a higher improvement in the quality score of the early outputs than that of the late outputs. It also clearly shows how the compression ratio affects the quality score. From Table 2, we can see that the largest scaling factor that keeps the number of parameters large enough gives better results with the database (FID=52.8; mIOU=67.5) than without the database (FID=64.2; mIOU=59.8). In addition, the fewer parameters (S=1/4) in the same early output branch (branch 1), the worse the output image quality score. However, with the database (FID=54.9; mIOU=65.5), the output image quality score is better than without the database (FID=69.6; mIOU=5 7.5). In addition, the closer the early output branch s is to the main network output, the better the image quality score, but also the output image quality score is better with the database than without the database. It is obvious that the output image quality score for the main network without early output modules is the best.

Table 3 shows the quantitative results for the OASIS pipeline at different scaling factors. The minimum number of channels is 32. For 4 different SF scaling factors, the pipeline was tested with and without a directional database, indicated by a check mark or a cross in the "Bank" column. Four columns, one for each branch, contain the FID (initial Fréchet distance) and mIOU (average intersection over connection) estimates; at the bottom, these two values are given for the main network. It can be concluded that adding a bank provides a higher improvement in the quality score of the early outputs than the late outputs. It also clearly shows how the compression ratio affects the quality score. Any compression will do, the choice depends on the quality score required.

From Table 3, as well as from Table 2, it can be seen that adding a database improves the output image quality index, and increasing the SF scaling factors worsens the output image quality index, in addition, the higher the output number of the early output branch, the higher the output image quality index.

The OASIS predictor was trained to output an image quality score for each branch. The minimum squared error loss between its predictions and the actual qualities was specified as described above:

where c are the matching parameters; S is the LPIPS score for the image generated with the above parameters; and P(c) is the predictor output.

The learning rate was set to 0.01, and the loss was optimized using stochastic gradient descent with cosine decay [90]. The learning rate is a common parameter in the field that is necessary for training; if the user wants to reproduce experiments with high accuracy, he needs to know this coefficient. However, this is not part of the invention.

The choice of the training dataset for the predictor was non-trivial, since the input to the pipeline consists of a semantic map concatenated with a 3D noise tensor. Due to the high dimensionality of the noise space, uniform sampling from it does not guarantee any convergence for the training process. Instead, 100 3D noise tensors are randomly extracted and concatenated with 500 semantic maps, yielding 50,000 examples. The method is then tested using 100, 300, and 500 noise tensors. After training, the predictor error is measured using 500 semantic maps concatenated with the same noises used for training and with new noises. The results are presented in Tables 4 and 5, respectively.

Table 4 shows the validation error for the OASIS predictor. The validation dataset was created by concatenating the noise (random signal) used for training with 500 semantic maps. The first column shows the amount of noise used to train the predictor, and columns B1-B4 show the error obtained by individual branches 1 through 4 during validation. The last column shows the average of all errors. The table shows how the error decreases as the noise used for training increases.

Table S4 shows the testing error for the OASIS predictor. The test dataset was created by concatenating random noise with 500 semantic maps. The first column shows the amount of noise used to train the predictor, and columns B1–B4 show the error obtained by individual branches 1–4 during testing. The last column shows the average of all errors. The table shows how the error decreases as the noise used for training increases.

To implement the database for image generation direction, it is populated with 500 randomly extracted images from the training dataset.

For each randomly extracted image, 100 different inputs are created using a fixed set of 3D noise (noise that has a 3D tensor structure rather than a 2D matrix). These inputs are fed into the first 2D convolutional layer and the subsequent ResNet module of the main network. The resulting features are then divided into 8×16=128 non-overlapping chunks of data according to their resolution, which provide clues for finding the most similar image. The values are extracted by processing the inputs up to the third ResNet module of the main network (the computational module of the OASIS architecture) and cutting the resulting features into identical chunks of data. The database is populated once at the beginning of the training phase. To reduce redundancy in the keys, FPS sampling [19] is applied to them (since many images can be extremely similar and there is no point in storing all of them, an FPS sampling algorithm is used for this, which selects only one image from each similarity cluster) in the forward inference phase, after the input is processed by the first 2D convolutional layer and the next ResNet layer, it is divided into 128 identical data chunks (128 is an arbitrary number that preserves proportions. This is done in order to select more than one image that is somewhat similar to the original one, but to highlight more similar elements - a street, a corner of a house, etc. Then the key most similar to each data chunk is searched in the database using the FAISS library [38]. All 128 extracted values are then merged accordingly (the extracted values are

rectangular sections of the plane, and this plane is always cut into 128 data fragments, they are glued together by writing into a common matrix of the required size).

This composite feature is used to guide the synthesis process by connecting it to the input of each branch after appropriate resizing performed by convolution. Figure 2 shows the resulting quality score distribution among all branches, estimated by the initial Frechet distance (FID) [32]. Figure 2 shows the relationship between the quality score (expressed in FID units) and the computations for all branches at different scaling factors of the OASIS implementation using the guide database. The three curves in the graph connect the FID values obtained by outputs 1-4 (branches 1-4) for different scaling factors. The squares represent a scaling factor of 1/2, the dots represent a scaling factor of 1/3, and the triangles represent a scaling factor of

scaling by 1/2. A higher scale saves computation at the expense of a lower quality score. The asterisk indicates the raw quality score and computation for the OASIS pipeline. As noted above, the larger the FID, the lower the output image quality score and the lower the computational cost. Finally, a pipeline comprising all the generative branches and the main network, together with a guide dataset, is used to create the predictor training dataset. The OASIS input consists of a semantic map and a high-dimensional random noise space (a set of multidimensional vectors consisting of random numbers). This training is constrained to 100 fixed noise vectors in combination with the Cityscape training set. Figure 3 shows example outputs from the OASIS pipeline branches. The top left image represents the semantic map used as input to the pipeline, the top middle image is the output of the original OASIS model (the main network), the top right image is the output obtained by the first branch with the corresponding quality score of 0.13 LPIPS, the bottom left image is the output of the second branch with the corresponding quality score of 0.11 LPIPS; the bottom middle image is the output of the third branch with the corresponding quality score of 0.10 LPIPS; the bottom right image is the output of the fourth branch (early output) with the corresponding quality score of 0.10 LPIPS. Figure 3 shows the finished image obtained by each branch. It can be seen how the quality score deteriorates as the output order decreases, i.e. the first output has the worst quality score.

Figure 4 shows the overall result for the entire pipeline with SF=1/4. Figure 4 illustrates the distribution of computations among the branches of the OASIS main network for a range of given LPIPS thresholds (the lower bound on the quality score, measured in LPIPS metrics, is common in the state of the art). Branch 1 is "a"; branch 2 is "b"; branch 3 is "c"; branch 4 is "d"; the main network is "e". For each quality score threshold, the predictor directs computations to one of five possible outputs depending on the complexity of the input it has learned. As the quality score requirement decreases, the use of the early branches becomes more pronounced. All distributions are obtained by sampling the same 500 test images using a scaling factor of 1/4. The total GFLOPs for each distribution are shown as a solid line, and their absolute values are shown on the right. The last value shows the distribution of branches chosen by the predictor at different quality score thresholds. It can be seen how different quality score thresholds affect the output selection: setting the quality score very high narrows the range of possible outputs, while lower (but still high) requirements use all additional branches. Most importantly, the GFLOPs count shows a sharp decrease in computation when using earlier branches. Fitting the GFLOPs curve to a constant slope, we can estimate an average gain of 1.2x103 GFLOPs/LPIPS. Ratio is the proportion of computation relative to the number of computations in the main network. That is, for example, 1 corresponds to the number of computations equal to the number of computations in the main network, and 0.5 is half the computations in the main network.

The implementation of the neural head avatar is based on the MegaPortraits generation method [17] for images of 512 × 512 pixels. This pipeline consists of several stages that ensure the transfer of characteristic features from the source face to the driver face, i.e. the one whose orientation and expression need to be obtained. The main network computational modules are l _i , i ∈ [1, 9], its final set of computational modules covers 9 residual blocks, which in total is 213 GFLOPs. Three branches are attached, one after blocks number 2, 4 and 6 of the main network. Their computational modules are the same residual blocks, and their corresponding depth, i.e. the number of computing modules, mirrors the remaining path: 8, 6 and 4, thereby preserving k+len=9 {2, 4, 6}. To facilitate branches, three different scaling factors are applied to the width of the compute modules, i.e. the number of channels. Their total GFLOPs are listed in Table b.

Table 6 illustrates the GFLOP comparison of all 4 computational routes through the branches and the main network of MegaPortraits (BB, far right column). Different rows correspond to different scaling factors (SF). Table 6 shows the computations of all outputs. It can be seen that the earlier the output, the less computations.

The branches are trained by specifying an adversarial loss function as in equation (1), obtained by competing with copies of the Mega-Portraits discriminator. Along with this, the VGG [37], MS-SSIM [75] and between the synthetic branch images and the main network. In addition, intermediate logits of the main network are used to specify the feature matching loss (FM) [73] and preserve the original gaze loss (GL) [17].

Where - standard adversarial loss function for GAN networks; - loss of VGG [37]; - loss of MS-SSIM [75]; - loss of L1, - loss of feature matching [73]; and - loss of gaze [17]. The coefficients C ₁ are chosen to harmonize the effects of losses.

The database is populated with images of a source face with many different orientations and expressions. At each iteration, the database is searched for the face that is most similar to the driver face, i.e., the one whose orientation and expression are to be obtained. To perform this search, the driver is fed to the first computational module of MegaPortraits, which extrapolates the angles describing the direction of the face and a multidimensional vector encoding the facial expression. The authors limited themselves to 3 angles for encoding the face directions, and the expression space was 512-dimensional. The result is a vector characterizing the driver that is closest to the images in the database. The resulting image is then concatenated with the input of each computational module. branches after appropriate resizing.

Figure 5 shows example branch outputs for the MegaPortraits pipeline. The top row uses the first and second images as source and driver, respectively. The appearance of the source is superimposed on the driver expression. The third image is the output of the original MegaPortraits pipeline, and the fourth image is retrieved from the database. Next come the outputs of branch 1 with LPIPS 0.1, branch 2 with LPIPS 0.08, and branch 3 with LPIPS 0.05. The bottom row is a mirror image of the top row, except that the source (the image of the person on the left in Figure 1(b)) and driver (the image of the person on the right in Figure 1(b)) images are changed.

Figure 6 shows the obtained quality score distribution over the branches. Figure 6 illustrates the relationship between the quality score (expressed in LPIPS units) and the computation for all branches at different scaling factors of the MegaPortraits implementation using the guided database. In Figure 6, three curves in the graph connect the FID values obtained by outputs 1-3 (branches 1-3) for different scaling factors. The dots represent the scaling factor of 1/3, the squares represent 1/6, and the squares represent 1/15. It can be seen that the higher scaling factor saves computation while reducing the quality score. The asterisk shows the original quality score and the computation for the OASIS pipeline.

Finally, the predictor is trained. After this, any quality score threshold can be specified, and the predictor will be able to choose the path that satisfies it with the least computation. Figure 7 shows the overall results for the entire pipeline. Branch 1 is "a"; branch 2 is "b"; branch 3 is "c"; the main network is "d". Figure 7 illustrates the distribution of computation between the branches of the MegaPortraits main network for a set of given LPIPS thresholds. For each quality score threshold, the predictor routes computation to one of four possible outputs depending on the complexity of the input it has learned. This is shown by four different columns for each LPIPS value. Columns "a", "b", "c" and "d" correspond to the ratio of outputs routed through outputs 1,2,3, respectively, and the main network. All distributions were obtained by sampling the same 7 02 test images using SF=1/15. The total GFLOPs for each distribution are shown as a solid line, with their absolute values shown on the right. It can be seen how lower quality thresholds can be maintained while significantly reducing GFLOPs by using lighter branches. Fitting the GFLOP curve to a constant slope yields an average gain of 1.3 × ¹⁰³ GFLOPs/LPIPS.

Although image generation is possible without the image guide database, it is necessary to provide the quality score of the earlier branches. In fact, it can be argued that its implementation harmonizes the quality score of the outputs by influencing the earliest branches, as shown in Figure 8. The numbers of images on the y-axis indicate the numbers of images with the LPIPS quality score on the x-axis. Figure 8 shows a comparison of the quality score distributions for the OASIS pipeline with SF=1/4 with and without the guide database. The LPIPS were obtained by comparing with the main network images. The quality score distributions of each branch are numbered differently: 1, 1', 2, 2', 3, 3', 4, 4'. The distributions without the database are shown as a dashed curve (1', 2', 3', 4'), and the distributions obtained with the database are shown as a solid curve (1, 2, 3, 4). The closer to the left edge of the curve, the better the quality score of the output image. It can be seen that the quality score distributions for the first branches are more biased towards better quality after the database is implemented. For example, the dashed line 1' shifts sharply to the left, becoming solid curve 1 when the database is used, while the dashed curve 4' shifts less, becoming solid line 4. The curves were plotted for a sample of 500 images using a kernel density estimate with a bandwidth of 0.3.

Additionally, this database can be used to compensate for the lack of a training set. The rendering difficulty is partly due to the lack of DNN training, which may well be inherent to a specific task, such as generating neural avatars of heads.

As discussed above, the proposed implementation of dynamic routing is based on the creation of a suitable early output and the use of a predictor. The latter is necessary to enforce user-defined quality score thresholds, since using single outputs only produces images with a fixed quality score distribution.

Furthermore, although all branches have a certain average quality score fixed by their FID (see Fig. 2 as an example), one cannot rely on only one branch to produce images with a suitable quality score. The spread of the quality scores of each output is quite large, and it becomes wider at the earliest outputs, as can be seen in Fig. 8. The predictor prevents this by choosing a heavier branch (having more parameters) when a lighter branch fails to provide a quality score.

Figure 9 shows a comparison of the image quality score distributions obtained from individual branches and the distributions obtained using a predictor configured to output a threshold equal to the average branch quality score (solid curve with predictor, dashed curve without predictor). The numbers of images on the Y-axis indicate the numbers of images with the LPIPS quality score on the X-axis. It is clearly seen how the predictor ensures this quality score threshold by directing difficult images to the next branches and thereby shifting the distribution. Figure 9 illustrates a comparison between the OASIS individual branch quality score distributions and the quality score distributions obtained with

using a predictor (P). The predictor was configured to enforce thresholds equal to the average branch quality score. LPIPS were obtained by comparing branch images for SF=1/4 with the main network images. Each distribution was sampled by inputting 500 semantic maps with random noise. The curves are the result of kernel density estimation with a bandwidth of 0.3. All four plots contain two distributions: the dashed curve 1' represents the higher quality score distribution for the proposed method when output images with a lower quality score threshold equal to the average distributions of branches 1-4 are specified. The solid curve 1 in each plot represents the branch distribution. It can be seen how the predictor implementation, which directs computations to enforce a lower quality score threshold, shifts the distributions towards the higher quality score, thus confirming its effectiveness.

Not all images are equally difficult to generate. This unevenness is the basis of the proposed method. This uneven distribution of complexity is due to many reasons. Some head turns or facial expressions may be less pronounced at the training stage, and therefore a heavier model is required to output images with a high quality score. To solve this problem, an analysis is applied by comparing images with different head turns and facial expressions, as well as their quality score. In particular, 7 02 head avatars were generated using the proposed pipeline and it was checked in which branch they were routed by the predictor.

Fig. 10 shows a comparison of the number of images sent to different branches depending on the head rotation. As can be seen in Fig. 10, the larger the angle between two images, the higher the complexity. The X-axis shows the distance in degrees, which means the angle between the head from the database and the driver's head.

Branches for SF=1/15 were used. Distributions were obtained for a sample of 7,02 images in total. The curves are the result of kernel density estimation with a bandwidth of 0.5. The quality indicator threshold was set to 0.09 LPIPS. It can be seen that branch 1 is used to output most images in which the driver's head is only slightly rotated relative to the original image (up to 6 degrees), and branches 2 and 3 are responsible for larger angles, which confirms the correlation between the angle and the complexity of installing MegaPortraits. The Z axis represents the number of images normalized to 1, i.e. 1=100 images.

Although the proposed method saves a large amount of computation, it has some limitations. The entire pipeline is applicable only to architectures containing a decoder, and cannot be applied without modification to transformers and other synthesis algorithms that do not have a decoder. There is no single recipe for filling the database. The authors decided to fill it randomly, but in fact this may not be the best choice. Since it is necessary to create a training dataset for the predictor, additional training inputs are needed, so the size of the effective databases will be larger. All branches require additional training, and the memory used to store the entire pipeline is larger than that used for the original DNN.

Saving computation is important for running complex algorithms that produce the best inferences, but these can be implemented mainly on "powerful" hardware.

It was shown how to create early outputs in decoders where the internal logits must be carefully processed before they can produce an image. A semi-parametric algorithm was created to control the synthesis, which favors earlier outputs. Then it was shown how to dynamically select the computation route using a neural network trained on examples of inputs and output quality metrics. The proposed invention takes a step in this direction, namely reducing the redundancy of the computation. In addition, it is necessary to emphasize how important saving computations and therefore energy is nowadays [67].

The original OASIS DNN generative network [63] consists of an initial 2D convolutional layer followed by 6 SPADEResBlock [57] modules and final Conv2D, LeakyRelu and TanH. Its total number of parameters is 74M. Four branches are attached to the network after ResBlock 1, 2, 3 and 4, respectively. Each branch consists of the same number of ResBlock modules as the rest of the main network. To create lighter computational paths, the number of channels of the branch computation units is reduced. To ensure consistency, all channels must be uniformly downscaled by multiplying them by the scaling factor (SF). Since such scaling with arbitrary factors may result in too few channels to be useful, its effect is limited by specifying a minimum number of channels at which scaling is not applied. In other words, if the minimum number is 64 and a factor of 1/3 is applied starting from 128, then the new number of channels will be 64 instead of 43.

Table 7 presents the module sizes for all branches as follows: (input channels, output channels, image height, image width). The table is divided into three subtables according to the applied scaling factor. The first column shows the module type, i.e., the transformation applied to the input data; columns 2-5 show the sizes of the computational modules of branches 1-4. At the bottom of each subtable, the total number of parameters with and without the addition of the auxiliary database is shown.

Table 7 shows that the higher the SF, i.e. the greater the number of computational parameters for each early output branch, the more input and output channels this early output branch has, while the height and width of the output image do not change. The higher the number of the early output branch, the fewer input channels and output channels, while the height and width of the output image increase.

In all branches after each SPADE-ResBlock except the last one, 2D nearest neighbor upsampling is also applied, doubling the height and width. When using the database, the input channels to the first ResBlock in each branch are multiplied by 1.5.

Figure 11 shows the comparison of the efficiency of different scaling factors. The minimum number of channels is 64. From top to bottom: SF=1/2, 1/3, 1/4. Branch 1 is "a"; branch 2 is "b"; branch 3 is "c"; branch 4 is "d"; the main network is "e". The X-axis shows the quality score thresholds expressed in LPIPS, and the Y-axis shows the computations required to output images expressed in GFLOPs. The bars in the graph show how much each branch is used to output images with a given quality score threshold. The sum of all bars for each threshold is 1. The curve on the top represents the total number of GFLOPs required to enforce the quality score threshold. This curve is used to evaluate the efficiency of the proposed model in terms of its slope, that is, the saving of computational resources at the expense of losing the quality score. The higher compression (SF=1/4) results in a bigger gain in terms of computational savings, since its slope is higher (almost reaching 127GB at the rightmost point, while the other scaling factors saturate earlier).

The database used was created from 500 semantic maps randomly selected from the training dataset, each of which was paired with 100 different 3D noise tensors to produce a set of different inputs, which were processed and divided into 128 non-overlapping data chunks, yielding a total of 500×100×128=6.4 million key-value pairs. Since redundancy in the key space is very likely, only up to 5K pairs are extracted from this set for each semantic class used.

The FPS sample [19] consisted of only 122,100 pairs. Each key is a 102-dimensional vector, and each value consists of a float32 tensor of dimensions (512, 4, 4). Thus, the total size of the stored parameters is 1.1 GB.

During the search, the guide features are taken after the first Conv2D and ResNet blocks of the main network. Then, for each of the N ∈ [1, 35] semantic classes present in the input, these features are cut into 128 data chunks and their 1024-dimensional space is scanned to find the closest database key with the corresponding semantic class. This search is performed quite quickly thanks to the FAISS library [38] and is thus computationally inexpensive.

After all 12 8 data chunks have been extracted, a guiding feature is created by gluing these chunks together. This feature is connected to the input of each branch, and therefore the number of channels in them must be increased. When using the database, the input channels for the first ResBlocks in each branch, listed in Table 8, are multiplied by 1.5.

The last key component of the proposed pipeline is the predictor. Its architecture is presented in Table 8.

Table 8 describes the architecture of the MegaPortraits and OASIS predictors. The dimensions are presented as (input channels, output channels). In both subtables, the left column shows the layers of the neural network, and the right column shows the sizes of its inputs and outputs. The bottom rows show the total number of parameters that make up these networks and the number of operations required to execute them once.

The original MegaPortraits DNN generative network [17] for 512×512 pixel images consists of a set of computational modules predicting the volume representation and another set called G2D that renders the output image from the processed volume. Its total number of parameters is 32M. The proposed branches are added after the 2, 4, 6 ResBlock2D modules. Their corresponding length is 7, 5, 3. As before, lighter computational paths are created by uniformly downscaling all channels. The new channel numbers are obtained by multiplying the original channels by a scaling factor. As before, the effect of this scaling is limited by setting the minimum number of channels to 24, a parameter chosen without strict justification. This is not a mandatory part of the method and can be replaced by another one, as shown by further analysis, in which further scaling is not required. Many different scaling factors are specified, which are presented in Table 9.

Table 9 describes the MegaPortraits pipeline. The compute unit sizes for all branches are presented as (input channels, output channels). The table is divided into four subtables according to the applied scaling factor. The first column shows the compute unit type, i.e., the transformation applied to the input data; columns 2, 3, and 4 show the compute unit sizes for branches 1–3. The bottom of each subtable lists the total number of parameters, including the addition of the auxiliary database. Res-Block2D consists of BatchNorm2D, h-swish, Conv2D, BatchNorm2D, h-swish, Conv2D, Conv2D layers with skip connections. All branches apply 2D bilinear upsampling before each ResBlock2D. When using this database, all input channel counts should be increased by 3.

For this task, a database containing 960 key-value pairs was used. The values consisted of RGB images of the source subject, uniformly covering the space

head turns and expressions. The keys were derived from MegaPortraits' initial computational modules, called encoders, which output the Euler angles through which the head rotates, as well as a set of parameters encoding facial expressions. Each key encoded 3 angles and a 512-dimensional vector for facial expressions.

Thus, the total size of the stored parameters was 10 ⁹ . The closest key search in the database was performed in the inference stage using the FAISS library [38]. Each extracted image was then appended to the input of all ResBlock2D modules in each branch, so when using the database, it is necessary to add 3 channels to all input channels in Table S9. The overall architecture of the MegaPortraits predictor is presented in Table 10.

Table 10 shows the architecture of the MegaPortraits predictor and the OASIS predictors. The sizes are listed as (input channels, output channels). In both subtables, the left column shows the layers of the neural network, and the right column shows the sizes of its inputs and outputs. The bottom rows show the total number of parameters that make up these networks and the number of operations required to execute them once.

Training details

For the MegaPortraits pipeline, branches are trained using piecewise linear adversarial losses, with each branch competing against a copy of the multi-scale chunk discriminator [11]. Additionally, the feature matching loss [73], the VGG19 perceptual loss [37], and the L1 and MS-SSIM losses [75] are specified. A specialized gaze loss computed with the VGG16 network, which combines gaze detection (RT-GENE, [89]) and blink detection (RT-BENE, [2]) into a single model, is also used. More details on these losses can be found in MegaPortraits [17]. All losses are computed with respect to the main network images and using only the foreground regions. Overall, the overall loss is

Where - standard adversarial loss function for GAN networks; - loss of VGG [37]; - loss of MS-SSIM [75]; - loss of L1, - loss of feature matching [73]; and - gaze loss [17]; with the following weights: c1=18, c2=0.84, c3=0.16, c4=40, c5=5. The branches and discriminators were trained using AdamW [91] optimizers with _β1 =0.05, _β2 =0.999, ∈= ^10-8 , weight decay= ^10-2 and initial learning rate=2× ^10-4 . Cosine learning rate schedulers with a minimum learning rate of ¹⁰⁻⁶ were used during training. The computations were performed in parallel using PyTorch distributed data. The model was trained with mixed precision on two NVIDIA P40 GPUs with an effective batch size of 6 for about 3 days. The obtained qualities can be found in Table 11.

Table 11 presents the quantitative results of the MegaPortraits pipeline for cross-replay. For four different SF scaling factors (first column), the pipeline was tested with and without a bank of guide data, which are indicated by a check mark or a cross in the Bank column. Three columns, one for each branch, contain the FID (initial Fréchet distance) and mIOU (average intersection over connection) scores; these two values are shown at the bottom for the main network. Thus, different compressions affect the quality score - the stronger the compression, the lower the quality score. Also, adding a bank improves the quality score, especially in the first output.

For each input, the predictor estimates the LPIPS for all branches. To train it, the MAE loss between the predicted state and the ground truth similarity state is introduced as described above:

where c are the matching parameters; S is the LPIPS score for the image generated with the mentioned parameters; and P(c) is the predictor output.

The AdamW optimizer with β ₁ = 0.05, β ₂ = 0/ 999 and an initial learning rate of 2×10 ^-4 is used together with a cosine learning rate scheduler.

All architectures listed in Table 7 and Table 9 were implemented. The overall results for the OASIS pipeline can be compared in Figure 11 and Figure 12, while Figure 13 shows them for the Mega-Portraits pipeline. Figure 12 illustrates the OASIS pipeline. The minimum number of channels is 32. Branch 1 is "a"; branch 2 is "b"; branch 3 is "c"; branch 4 is "d"; the main network is "e". A comparison of the performance of different scaling factors is shown. From top to bottom, left to right: SF=1/2, 1/3, 1/4, 1/6. Figure 13 illustrates the MegaPortraits pipeline, a comparison of the performance of different scaling factors. From left to right: SF=1/3, 1/6, 1/8, 1/15. It can be seen how different scaling factors produce different branch distributions. For all these graphs, the x-axis shows the thresholds expressed in LPIPS, and the y-axis shows the computation required to output the images expressed in GFLOPs. The bars in the graph show how much each branch is used to output images with a given quality score threshold. The sum of all the bars for each threshold is 1. The curve at the top represents the total number of GFLOPs required to meet the specified quality score threshold. This curve is used to evaluate the effectiveness of the proposed model in terms of its slope, i.e. the computational savings at the expense of quality score. The higher the compression, the higher the gain in terms of computational savings.

Figure 14 shows the impact of the database on the branches of all scaling factors. Figure 14 illustrates the OASIS pipeline, comparing the impact of the database on the distribution of quality scores for different factors.

scaling. The quality score distribution of each branch is numbered differently: 1, 1', 2, 2', 3, 3', 4, 4'. The distributions without database implementation are shown as a dotted curve (1', 2', 3', 4'), and the distributions obtained with database implementation are shown as a solid curve (1, 2, 3, 4). The minimum number of channels is 64. From left to right: SF=1/2, 1/3, 1/4. The X-axis shows the quality score in LPIPS units, and the Y-axis shows the number of images output with the specified quality score. Different branches output quality score distributions colored differently. The quality score distributions obtained without database implementation are shown as a dotted curve, and the quality scores with database implementation are shown as a solid curve. It can be clearly seen that the database implementation has had the greatest impact on the first branches, since the quality score distribution for these first branches is most skewed towards the best values. Therefore, the database is a powerful tool for harmonizing the output quality score.

For the MegaPortraits conveyor, the quality indicator

of the synthesized images correlates with the head rotation angle. This is also reflected in the proposed method. Indeed, heads rotated at larger angles are more likely to be sent to the later branch, as can be seen in Fig. 15. Fig. 15 illustrates the MegaPortraits pipeline, the distribution of images sent to different branches depending on the head rotation angle. The first row SF=1/8, the second row SF=1/15. The X-axis shows the angle between the reference head and the head in the output image, and the Y-axis shows the number of images output with the specified angle. It can be clearly seen that for small angles, most of the images are output by the first branch, and wider angles are processed by the second and third branches to obtain the output. It can be concluded that one of the sources of rendering complexity is the head rotation, which means that not all positions require as much computation as small angles, which confirms the need to implement the proposed branches and predictors.

The illustrative embodiments described above are examples and should not be considered as limiting. Furthermore, the description of these embodiments is intended to illustrate and not to limit the scope of the claims, and many alternatives, modifications and variations will be obvious to those skilled in the art.

Literature:

[1] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1506–1515. 2022.

[2] Ali Alqahtani, Xianghua Xie, and Mark W. Jones. Literature review of deep network compression. Informatics, 8(4). 2021.

[3] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning, pages 527-536. PMLR. 2017.

[4] Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pages 535-541, New York, NY, USA. Association for Calculating Machinery. 2006.

[5] Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.

[6] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations. 2020.

[7] Jianchuan Chen, Ying Zhang, Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, and Huchuan Lu. Animatable neural radiance fields from monocular rgb videos. arXiv preprint arXiv:2106.13629. 2021.

[8] Xinshi Chen, Hanjun Dai, Yu Li, Xin Gao, and Le Song. Learning to stop while learning to predict. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1520-1530. PMLR. 2020.

[9] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2019.

[10] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani , Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc. 2019.

[11] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017

[12] Xin Dai, Xiangnan Kong, and Tian Guo. Epnet: Learning to exit with flexible multi-branch network. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM '20, pages 235–244, New York, NY, USA. Association for Calculating Machinery. 2020.

[13] Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485-532. 2020.

[14] Rongkang Dong, Yuyi Mao, and Jun Zhang. Resourceconstrained edge ai with early exit prediction. Journal of Communications and Information Networks, 7(2):122-134. 2022.

[15] Michail Christos Doukas, Evangelos Ververas, Viktoriia Sharmanska, and Stefanos Zafeiriou. Free-headgan: Neural talking head synthesis with explicit gaze control. arXiv preprint arXiv:2208.02210. 2022.

[16] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14398–14407. 2021.

[17] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia. Association for Calculating Machinery. 2022.

[18] Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In ICLR 2020-Eighth International Conference on Learning Representations, pages 1-14. 2020.

[19] Y. Eldar, M. Lindenbaum, M. Porat, and Y.Y. Zeevi. The

farthest point strategy for progressive image sampling. IEEE Transactions on Image Processing, b(9):1305–1315. 1997.

[20] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images. ACM Trans. Graph., 40(4). 2021.

[21] Stanislav Frolov, Tobias Hinz, Federico Raue, J"orn Hees, and Andreas Dengel. Adversarial text-to-image synthesis: A review. Neural Networks, 144:187-209. 2021.

[22] Anna Fr"uhst"uck, Krishna Kumar Singh, Eli Shechtman, Niloy J. Mitra, Peter Wonka, and Jingwan Lu. Insetgan for full-body image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7723–7732. 2022.

[23] Liang Gonog and Yimin Zhou. A review: Generative adversarial networks. In 2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA), pages 505-510. 2019.

[24] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc. 2014.

[25] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM, 63(11):139–144. 2020.

[26] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819. 2021.

[27] Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani. Drop the gan: In defense of patches nearest neighbors as single image generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13460–13469. 2022.

[28] Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Niesner, and Justus Thies. Neural head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18653–18664. 2022.

[29] Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In International Conference on Learning Representations. 2017.

[30] James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3). 2007.

[31] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. Arch++: Animation-ready clothed human reconstruction revisited. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11046–11056. 2021.

[32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local our equilibrium. In I. Guyon, U. Von Luxburq, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. 2017.

[33] Tao Hu, Tao Yu, Zeronq Zhenq, He Zhang, Yebin Liu, and Matthias Zwicker. Hvtr: Hybrid volumetric-textural rendering for human avatars. arXiv preprint arXiv:2112.10203. 2021.

[34] Phillip Isola and Ce Liu. Scene collaqinq: Analysis and synthesis of natural images with semantic layers. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2013.

[35] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.

[36] Menglin Jia, Bor-Chun Chen, Zuxuan Wu, Claire Cardie, Serge Belongie, and Ser-Nam Lim. Rethinking nearest neighbors for visual classification. arXiv preprint arXiv:2112.08459. 2021.

[37] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision - ECCV 2016, pages 694-711, Cham. Springer International Publishing. 2016.

[38] Jeff Johnson, Matthijs Douze, and Herv'e J'egou. Billionscale similarity search with GPUs. IEEE Transactions on Biq Data, 7(3):535–547. 2019.

[39] Mica K. Johnson, Kevin Dale, Shai Avidan, Hanspeter Pfister, William T. Freeman, and Wojciech Matusik. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Transactions on Visualization and Computer Graphics, 17(9):1273–1285. 2009.

[40] Moritz Kappel, Vladislav Golyanik, Mohamed Elgharib, Jann-Ole Henningson, Hans-Peter Seidel, Susana Castillo, Christian Theobalt, and Marcus Magnor. Hiqh-fidelity neural human motion transfer from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), paqes 1541-1550. 2021.

[41] Moritz Kappel, Vladislav Golyanik, Mohamed Elgharib, Jann-Ole Henningson, Hans-Peter Seidel, Susana Castillo, Christian Theobalt, and Marcus Magnor. High-fidelity neural human motion transfer from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1541–1550. 2021.

[42] Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3301-3310. PMLR. 2019.

[43] Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. In European Conference of Computer Vision (ECCV). 2022.

[44] Jean-Franc' ois Lalonde, Derek Hoiem, Alexei A. Efros, Carsten Rother, John Winn, and Antonio Criminisi. Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3):3. 2007.

[45] Stefanos Laskaridis, Alexandros Kouris, and Nicholas D. Lane. Adaptive inference through early-exit networks: Design, challenges and directions. In Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning, EMDL' 21, pages 1-6, New York, NY, USA. Association for Calculating Machinery. 2021.

[46] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann. 1989.

[47] Sam Leroux, Steven Bohez, Elias De Coninck, Tim Verbelen, Bert Vankeirsbi1ck, Pieter Simoens, and Bart Dhoedt. The cascading neural network: building the internet of smart things. Knowledge and Information Systems, 52(3):791-814. 2017.

[48] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6): 194:1-194:17. 2017.

[49] Xiangjie Li, Chenfei Lou, Zhengping Zhu, Yuchi Chen, Yingtao Shen, Yehan Ma, and An Zou. Predictive exit: Prediction of fine-grained early exits for computation-and energy-efficient inference. arXiv preprint arXiv:2206.04685. 2022.

[50] Yikang LI, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. Pastegan: A semi-parametric method to generate image from scene graph. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. 2019.

[51] Enrique S. Marquez, Jonathon S. Hare, and Mahesan Niranjan. Deep cascade learning. IEEE Transactions on Neural Networks and Learning Systems, 29(11):5475-5485. 2018.

[52] Mehdi Mirza and Simon Osindero. Conditional qenerative adversarial nets. arXiv preprint arXiv:1411.1784. 2014.

[53] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), paqes 11256-11264. 2019.

[54] Augustus Odena, Dieterich Lawson, and Christopher Olah. Changing model behavior at test-time using reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net. 2017.

[55] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2642-2651. PMLR. 2017.

[56] Emin Orhan. A simple cache model for image recognition. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. 2018.

[57] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019.

[58] Sida Penq, Zhen Xu, Juntinq Donq, Qianqian Wanq, Shanqzhan Zhanq, Qinq Shuai, Hu j un Bao, and Xiaowei Zhou. Animatable implicit neural representations for creative realistic avatars from videos. arXiv preprint arXiv:2203.08133. 2022.

[59] Xiaojuan Qi, Qifenq Chen, Jiaya Jia, and Vladlen Koltun. Semi-parametric imaginary synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.

[60] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanuqen Loqeswaran, Bernt Schiele, and Honqlak Lee. Generative adversarial text to image synthesis. In Proceedings of The 33 International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1060-1069, New York, New York, USA. PMLR. 2016.

[61] Simone Scarpiniti, Danilo Cominini el lo, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Differentiable branching in deep networks for fast inference. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4167-4171. 2020.

[62] Simone Scarpiniti, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should we add early exits to neural networks? Cognitive Computation, 12(5):954-966. 2020.

[63] Edgar Schonfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In International Conference on Learning Representations. 2021.

[64] Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Huiyu Zhou, RuiliWang, M. Emre Celebi, and Jie Yang. Image synthesis with adversarial networks: A comprehensive survey and case studies. Information Fusion, 72:126-146. 2021.

[65] Maying Shen, Pavlo Molchanov, Hongxu Yin, and Jose M. Alvarez. When to prune? a policy towards early structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12247–12256. 2022.

[66] Yupeng Shi, Xiao Liu, Yuxiang Wei, Zhongqin Wu, and Wangmeng Zuo. Retrieval-based spatially adaptive normalization for semantic image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11224–11233. 2022.

[67] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics. 2019.

[68] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015.

[69] Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464-2469. 2016.

[70] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang. Deep image harmonization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.

[71] Hung-Yu Tseng, Hsin-Ying Lee, Lu Jiang, Ming-Hsuan Yang, and Weilong Yang. Retrievegan: Image synthesis via differentiable patch retrieval. In Computer Vision - ECCV 2020, pages 242-257. Springer International Publishing. 2020.

[72] Lei Wang, Wei Chen, Wenjia Yang, Fangming Bi, and Fei Richard Yu. A state-of-the-art review on image synthesis with generative adversarial networks. IEEE Access, 8:63514–63537. 2020.

[73] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.

[74] Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E Gonzalez. Idk cascades: Fast deep learning by learning not to overthink. arXiv preprint arXiv:1706.00885. 2017.

[75] Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality index assessment. In The Thrity-

Seventh Asilomar Conference on Signals, Systems & Computers, volume 2, pages 1398-1402. 2003.

[76] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender,

and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Computer Vision - ECCV 2020, pages 660-676. Springer International Publishing. 2020.

[77] Maciej Wol czyk, Bartosz W'ojcik, Klaudia Bal azy, Igor T Podolak, Jacek Tabor, Marek ' Smieja, and Tomasz Trzcinski. Zero time waste: Recycling predictions in early exit neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 2516-2528. Curran Associates, Inc. 2021.

[78] Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. Berxit: Early exiting for bert with better fine-tuning and extension to regression. In Proceedings of the 16th conference of the European chapter of the association for computational linguistics: Main Volume, pages 91-104. 2021.

[79] Qunliang Xing, Mai Xu, Tianyi Li, and Zhenyu Guan. Early exit or not: Resource-efficient blind quality index enhancement for compressed imaqes. In Computer Vision - ECCV 2020, paqes 275-292. Springer International Publishing. 2020.

[80] Jae Shin Yoon, Duyqu Ceylan, Tuanfenq Y. Wanq, Jinqwan Lu, Jimei Yanq, Zhixin Shu, and Hyun Soo Park. Learninq motion-dependent appearance for hiqh-fidelity renderinq of dynamic humans from a single camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3407–3417. 2022.

[81] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of oneshot realistic head avatars. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020, pages 524-540. Springer International Publishing. 2020.

[82] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2019.

[83] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better visionlanguage modeling. arXiv preprint arXiv:2111.03930. 2021.

[84] Richard Zhanq, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wanq. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.

[85] Yufenq Zhenq, Victoria Fern'andez Abrevaya, Marcel C. B"uhler, Xu Chen, Michael J. Black, and Otmar Hilliges. I m avatar: Implicit morphable head avatars from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), paqes 13545-13555.

[86] Zeronq Zhenq, Han Huanq, Tao Yu, Honqwen Zhanq, Yandong Guo, and Yebin Liu. Structured local radiance fields for human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15893–15903. 2022.

[87] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330-18341. Curran Associates, Inc. 2020.

[88] Kevin Cortacero, Tobias Fischer, and Yiannis Demiris. Rtbene: A dataset and baselines for real-time blink estimation in natural environments. In Proceedings of the IEEE International Conference on Computer Vision Workshops.

[88] Tobias Fischer, Hyung Jin Chang, and Yiannis Demiris. RTGENE: Real-Time Eye Gaze Estimation in Natural Environments. In European Conference on Computer Vision, pages 339-357.

[89] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.2014.

[90] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

[91] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations. 2019.

Claims (51)

1. A system for producing output images having a user-selectable quality indicator, comprising: - an electronic device, a display, a memory storing input images and a set of generative artificial neural networks (GAN), wherein the electronic device is configured to perform artificial neural network operations; - each GAN is configured to allow the user to select from a set of GANs stored in memory to obtain an image with a predetermined quality score, and each GAN is pre-trained and consists of: N computing modules forming a main network and a plurality of branches of an earlier output, each of which is connected after each computing module, except for the last computing module of the main network, wherein each branch of an earlier output contains as many computing modules as there are left in the main network after the point of connection of the branch of an earlier output before the output of the main network, wherein each computing module of each branch of an earlier output performs the same function as the corresponding remaining computing module in the main network, and the computing resource of each branch of an earlier output is less than the computing resource of the corresponding remaining computing modules of the main network before the output of the main network, and the closer a specific branch of an earlier output is to the output of the main network, the higher the computational resource for obtaining the output image generated by a specific branch of an earlier output, and the closer a particular branch of an earlier output is to the output of the main network, the higher the quality indicator of the processed image generated by this particular branch of an earlier output, and where the quality indicators of the output image generated by the branches of the earlier output differ from each other; - a set of predictors, each of which is an artificial neural network, wherein each predictor is created and pre-trained for a specific GAN stored in memory, wherein the predictor is configured to predict a quality score of the processed image to be output from each branch of an earlier output of the specific GAN based on an input image that the user intends to feed into the specific GAN; wherein the GAN is configured to output to the display of the electronic device the processed output images from that branch of the earlier output that generates a processed output image having a quality indicator that most closely matches the quality indicator selected by the user. 2. The system of claim 1, further comprising a database storing guide data, wherein the GAN is further configured to select from the database guide data corresponding to a given input image and connect them with the input image data input to the GAN, wherein the guide data is connected with data from the computing module before the earlier output branch, and the data obtained after the connection is fed to the earlier output branch for further processing. 3. The system of claim 2, wherein said guide data are image fragments. 4. The system according to claim 2, wherein said guide data are features. 5. The system of claim 2, wherein said guide data are fragments of features. 6. The system according to any one of paragraphs 1-5, wherein the quality indicator is expressed in FID units. 7. A method for obtaining output images with a user-selected quality indicator, using the system of claim 1, comprising the steps of: select, from memory by the user, input images, a pre-trained GAN and a pre-trained predictor corresponding to this GAN; select, by means of the user, a quality indicator for the processed output images; feeding at least one of the input images, transformed into input data to be processed by the predictor, to a corresponding pre-trained predictor; predicting, based on at least one of the supplied input data, by means of a corresponding pre-trained predictor, a quality indicator for the processed output images that will be generated by each branch of the earlier output; selecting one branch of the earlier output that generates a processed output image having a quality score that most closely matches the quality score selected by the user; transform input images into input data for GAN processing; feed input data into the GAN with the earlier output branch selected for processing; process the input data through a GAN with the earlier output branch selected; receive finished processed output images at the output of the selected branch of the earlier output. 8. The method according to claim 7, in which the finished processed output images derived from the output of the main network or one branch of an earlier output that provides the selected image quality indicator are additionally stored in memory. 9. The method according to any of paragraphs 7, 8, in which the finished processed image, output from the output of the main network or one of the branches of an earlier output that provides the selected quality indicator, is additionally displayed on the display of the electronic device. 10. The method according to any one of paragraphs. 7-9, in which the quality indicator is expressed in FID units. 11. A method for obtaining output images with a quality indicator selected by the user, using the system of paragraph 2, comprising the steps of: select, from memory by the user, input images, a pre-trained GAN and a pre-trained predictor corresponding to this GAN, select, by means of the user, a quality indicator for the processed output images; feeding at least one of the input images, transformed into input data to be processed by the predictor, to a corresponding pre-trained predictor; predicting, based on at least one of the input images supplied, by means of a corresponding pre-trained predictor, a quality indicator for the processed output images that will be generated by each branch of the earlier output; selecting one branch of the earlier output that generates a processed output image having a quality score that best matches the user-selected quality score; transform input images into input data to be processed by the GAN; feed input data into the GAN with an earlier output branch for processing; process the input data through a GAN with the earlier output branch selected, while processing: - selecting from the database guide data corresponding to guide data for at least one of the input images, - connect the guide data to the data output from the computing module before the selected branch of the earlier output, - feed the connected data into the selected branch of the earlier output or into the main network for further processing; receive processed output images at the output from the selected branch of the earlier output. 12. The method according to claim 11, in which the finished processed output images derived from the output of the main network or one branch of an earlier output that provides the selected image quality indicator are additionally stored in memory. 13. The method according to any of paragraphs 11, 12, in which the finished processed image, output from the output of the main network or one of the branches of an earlier output that provides the selected quality indicator, is additionally displayed on the display of the electronic device. 14. The method according to any one of paragraphs 11-13, wherein said guide data are images. 15. The method according to any one of paragraphs 11-13, wherein said guide data are fragments of an image. 16. The method according to any one of paragraphs 11-13, wherein said guide data are features. 17. The method according to any one of paragraphs 11-13, wherein said guide data are fragments of features. 18. The method according to any one of paragraphs. 11-17, wherein the quality indicator is expressed in FID units. 19. The method according to any one of paragraphs 11-17, in which the guide data is stored by the user in the database in advance based on the input data selected from the memory. 20. A machine-readable medium storing instructions for performing the method according to claim 7 by means of an electronic device. 21. A machine-readable medium storing instructions for performing the method according to claim 11 by means of an electronic device.