RU2844852C1 - Method for face image editing - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to image processing. At the steps of the method, 3D-GAN inversion of the distorted input image of the face is carried out by optimizing the parameters of the camera and the hidden code of the face, which leads to obtaining a generated image. Obtained latent code and the new camera parameters are used to estimate the depth and then create 3D mesh, as well as to render this mesh to obtain a portrait. Blending is used, in which visible areas are reprojected, and occluded parts are restored using a generative neural network.

EFFECT: possibility of correcting human head rotation, eliminating perspective distortion.

5 cl, 7 dwg, 2 tbl

Description

The present invention relates to image correction, namely to correction of facial distortions and head pose in an image, such as a selfie photograph, obtained from a camera.

Description of the related art

Selfies are perhaps the most common type of photograph taken with a smartphone. Modern smartphone cameras produce high-quality images, but they suffer from unwanted facial geometry issues.

The distance from the camera plays a vital role in the perception of a portrait. Selfies taken from close distances often suffer from perspective distortion, which manifests itself in deformed and asymmetrical features, a huge nose and tiny or even invisible ears, creating unnatural and unpleasant images.

Another problem associated with 3D geometry is incorrect head pose. Posing for selfies requires practice, and choosing an attractive viewpoint is a non-trivial task: either several attempts are required to correct it, or in some scenarios, such a choice may not be possible. Therefore, the ability to correct the head pose during post-processing is a sought-after feature in selfie editing.

To date, various approaches have been proposed to manipulate facial geometry based on either 2D and 3D warping, neural emission fields (NeRF), or generative models (neural networks) to synthesize a portrait with updated observation conditions.

Known 2D warping-based dewarping methods rely on 2D flow map estimation to perform image warping. Such methods suffer from severe distortions due to inaccurate 3D geometry fitting. 3D warping-based approaches are excellent at preserving the original image details, but they fail to fill in the inevitable occluded regions.

NeRF-based methods provide full control over camera parameters to synthesize a new view, but do not use a priori data about the face. Accordingly, optimization is performed from scratch, and thus is very slow: despite some progress, these approaches are far from real-time performance.

Another branch of facial image manipulation techniques is based on the use of generative neural networks. In the generative pipeline, the original face image is encoded into a latent representation and then reconstructed with new viewing conditions and updated camera parameters using a pre-trained 3D-GAN. The latent face code, camera position, and focal length are estimated throughout the joint optimization procedure. However, fitting these parameters based on a single distorted image is a controversial and ill-posed problem, so existing methods face difficulties in reconstructing accurate 3D geometry. Although this can be improved to some extent by introducing geometry constraints, there is a more significant drawback: the generative approach cannot guarantee that distinctive features of an individual will be preserved. In addition, GANs tend to miss small details, which seriously affects the image quality.

Known 2D-GAN (generative adversarial network) inversion methods do not preserve facial structure during transformation, so uniformity across multiple views is not guaranteed. Recently introduced 3D-GANs demonstrate their ability to generate uniform results based on implicit 3D representations.

To reconstruct a face image under new viewing conditions, the original face must first be transformed into the latent space of a pre-trained GAN. This technique is called GAN inversion. 3D-GAN inversion methods rely on 2D-GAN inversion methods with pre-defined camera parameters that can be estimated using another method and then corrected or further tuned along with optimization of the latent face code.

Recently, DisCO proposed a sophisticated 3D inversion scheme for optimizing latent face code and camera parameters, which includes initialization with a short camera-to-face distance and camera reparameterization. The results achieved by a multi-stage optimization plan with geometry and landmark regularization look promising. However, the GAN-based inversion optimization is slow and, being a purely generative approach, DisCO does not preserve personality features and produces images without fine details.

Instead of optimization, GAN inversion can be performed using encoder-based techniques that transform the input image into latent space in a single forward pass and are orders of magnitude faster. Until recently, optimization-based methods were the best in terms of reconstruction quality, but newer encoder-based methods can produce more accurate and more view-consistent geometry [1].

In HFGI3D [2], GAN training is supervised using warped images, making

synthesized images are more realistic. Therefore, the output is generated entirely by the GAN, and warping is used only as a guide. On the other hand, in the proposed GAN pipeline, warping directly affects the final image.

When changing the head pose, deocclusions are inevitable and more noticeable, as previously invisible parts of the person's head and face become exposed. Thus, warping-based methods are insufficient for head pose correction if they are used alone, and 3D-trained methods dominate the field.

One line of work involves using NeRF-based methods to synthesize a new view from multiple images [17] or a single image [16]. However, such approaches may not guarantee the preservation of personality traits, since no a priori data about the face is used.

Known GAN-based approaches aim to preserve human-specific details by conditioning on the input image or video [18], latent code [19], or encoded facial attributes [20]. However, the preservation of personality features cannot be guaranteed, and the generated images exhibit slightly different features.

Thus, it is necessary to limit the various paradigms of face manipulation, approaches based on

GAN/NeRF/warping, by performing a warping merge for the visible parts and generation for the occluded parts. Drawings

The above and/or other aspects will be more apparent from the following description of exemplary embodiments with reference to the accompanying drawings, in which:

Fig. 1 illustrates examples of selfie editing results according to the proposed invention.

Fig. 2 illustrates an overview of the proposed conveyor.

Fig. 3 illustrates, on the left: the setup for filming HeRo using smartphones placed on a tripod; then: a series of pictures of the same people taken simultaneously by cameras from the front, left, right and top.

Fig. 4 illustrates a qualitative comparison of the CMDP dataset.

Fig. 5 illustrates images obtained in natural conditions with corrected distortion.

Fig. 6 illustrates the LPIPS, SSIM, ID and PSNR performance for different numbers of iterations.

Fig. 7 illustrates examples of pose correction taken from the HeRo dataset.

ESSENCE OF THE INVENTION

Photographs (images), such as self-portraits, taken from a short distance can look unnatural or even unattractive due to severe distortions that make facial features deformed and head poses strange. The proposed method performs 3D-GAN inversion of the distorted face image by optimizing the camera parameters and the hidden face code, which makes it possible to obtain

generated image. Visibility-based blending is used, where visible areas are reprojected and occluded parts are reconstructed using a generative neural network. Experiments on face distortion tests and a self-collected head rotation dataset (HeRo) demonstrate that the claimed method outperforms previous approaches both qualitatively and quantitatively, and thus opens up new possibilities for photorealistic selfie editing. Normalization of portraits (face images) allows for the desired head pose and/or the elimination of perspective distortions.

A method for editing images of faces in photographs is proposed, which contains:

a) the user selecting an image for editing containing an image of at least one person;

b) detecting images of faces in the selected image,

with segmentation of one face image from among the detected face images;

c) feeding the segmented face image to the input of a neural network that predicts the parameters and camera position for the segmented face image,

with the output receiving the initial parameters and position of the camera;

d) feeding the segmented face image to the input of a neural network that predicts the latent code of the face, with the latent code of the face being obtained at the output of the neural network that predicts the latent code of the face;

steps (c) and (d) are performed in parallel;

e) performing an iterative optimization process of 3D-GAN, for this purpose:

- feed the predicted parameters and position of the camera and the predicted hidden code of the face to the input of a 3D-GAN (generative adversarial network),

- receive a 3D-GAN generated image at the output,

- calculate the loss function between the face image obtained by segmentation and the generated image,

- iteratively change the predicted parameters and camera position and the predicted hidden face code, and

- feed the variable parameters and camera position and the variable hidden face code to the 3D-GAN input,

wherein the iterative optimization process is carried out until the said loss function reaches a minimum of the function in which the parameters and position of the camera and the latent code of the face satisfying this condition are the optimal parameters and position of the camera and the optimal latent code of the face;

f) feeding the optimal hidden face code to the input of said 3D-GAN,

with the supply of arbitrary new parameters and camera positions to the input of the mentioned 3D-GAN;

g) generating, using said 3D-GAN, a new generated image with a face image corresponding to the segmented face image, with a new face view corresponding to said new parameters and camera position,

with prediction of the depth map of the newly generated image;

h) processing the predicted depth map to construct a 3D mesh of the new generated image;

i) projection of the constructed 3D mesh onto the image plane with optimal parameters and camera position,

with the generation of a rendered image based on the given projection;

j) determining which vertices of a 3D mesh are visible and which are occluded after projection onto the image plane with optimal camera parameters and position,

obtaining a visibility mask based on said definition;

steps (i)-(j) are performed in parallel;

k) blending the new generated image and the rendered image using a visibility mask,

producing an edited segmented face image showing the face from a new angle;

l) transferring the obtained edited segmented face image to the selected image; and

steps (b)-(k) are performed for at least one face in the selected image;

m) displaying the selected image with at least one edited segmented face image on the screen to the user.

In this case, arbitrary new parameters and camera position can be selected by the user. The proposed method may additionally contain a step of selecting by the user one face image for segmentation from among the detected face images in the selected image.

At least one of the plurality of modules may be implemented using an AI model. The AI-related function may be performed via non-volatile memory, volatile memory, and a processor. The above-described method performed by an electronic device may be performed using an artificial intelligence model.

The processor may include one or more processors. At this time, the one or more processors may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP) or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a vision processing unit (VPU), and/or a special processor for AI such as a neural processing unit (NPU).

One or more processors control the processing of input data according to a predetermined operating rule or artificial intelligence (AI) model stored in non-volatile memory and volatile memory. This predetermined operating rule or AI model is provided by training or learning.

Here, provision by training means that a training algorithm is applied to a set of training data, providing a predetermined operating rule or AI model of a desired characteristic. The training may be performed on the device itself, on which, according to an embodiment, the AI is executed and/or may be implemented by a separate server/system.

An artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes a plurality of weight coefficients and performs a computation in the neural network by computing based on the result of the computation of the previous layer and the plurality of weight coefficients. Examples of neural networks include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GANs), and deep Q-networks.

A learning algorithm is a method for training a predetermined target device (e.g., a robot) using a set of training data to induce, enable, or control the target device to make a decision or make a prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

An AI model can be obtained by training. Here, "obtained by training" means that a predetermined operating rule or an AI model configured to perform a desired function (or purpose) is obtained by training the base AI model on many pieces of training data according to a learning algorithm.

Detailed description

A new approach to head pose and face image (portrait) distortion correction is provided, using the inversion of a perspective-taking 3D-GAN. The proposed method performs the functions of face distortion and head pose correction, for example, in selfie photographs or any photographs (images) containing a face obtained from cameras. Using the proposed method, the original image is processed and a corrected image is obtained. Using the proposed invention, the texture, geometry and details of the face are determined at different camera positions, resulting in selfie photographs with corrected face distortion and corrected head pose. The proposed invention allows for correcting the rotation of a person's head or eliminating perspective distortions of the original image. The proposed method can effectively perform correction of portraits taken by a camera (for example, selfies taken in real time or portraits selected by the user from the memory of a device, such as a smartphone) for aesthetic improvement of images.

The proposed method can be used in image applications for portrait normalization, for further processing in image/video analysis applications; to improve the quality of video streamed from the front camera, for example, in video conferencing or video blogging applications.

The proposed invention can be used for a smartphone, laptop, smart device or any electronic device capable of performing computer analysis of images connected to an RGB camera or receiving RGB images from another source. The proposed method can be implemented for a computer with a GPU. The minimum required set of components includes: a data storage device, a processor for processing information, a graphics processor.

The proposed method can be stored in the form of computer code on a machine-readable medium and can be implemented on a computer.

The proposed invention combines geometric and generative techniques, offering a robust solution for generating high-quality, feature-preserving images from new angles, for example in the context of selfies. Due to the natural limitations of 3D data, known approaches based on 3D warping (3D warping pipeline) tend to miss occluded areas. Meanwhile, known generative approaches (generative pipeline) cannot guarantee the preservation of human features, since the generated face may differ from the original one.

The proposed method combines the strengths of the generative and warping paradigms, and relies on the power of the generative neural network, while using most of the 3D-based warping approach (i.e., warping the original frame using some 3D information). The term "warping" refers to the warping of the original frame, and the term "generative" refers to the artificial generation of the image.

As mentioned earlier, the proposed invention allows to correct the rotation of a person's head or to eliminate perspective distortion of the original image. For example, in the original image the head is turned to the left, and the user wants to obtain from this original image an image in which the head is turned to the right. With any such manipulation, new areas of the face that were not visible in the original image will be visible in the final image, i.e. previously occluded areas of the face will be visible in the image of the face from a new angle. The occluded areas are taken from the generated image, which allows to restore invisible parts of the face and head. Meanwhile, the visible part of the face is reprojected, not generated, thus the personality traits are preserved. Generation is the receipt of a synthetic, new image. The reprojected image is the original image that has undergone "3D warping".

The excellent performance of the proposed method is proven on face distortion tests and a self-collected head rotation (HeRo) dataset, called HeRo and containing a number of individuals with different head positions and camera-to-face distances. The HeRo dataset was collected by the present authors and can be used to evaluate methods aimed at changing the human head pose.

Fig. 1 illustrates examples of selfie editing results according to the proposed invention. The top row (original) shows selfie photos taken by users, which are the original photos fed to the GAN. The bottom row (corrected) shows images obtained after processing according to the present invention. As can be seen from Fig. 1, the present invention can seamlessly change the pose of the head and eliminate perspective distortion, obtaining photorealistic and detailed corrected portraits.

As will be described in detail below, the novelty of the proposed invention consists in using 3D-GAN to generate a new generated image and a depth map, and then constructing a 3D mesh based on the depth map. Based on this 3D mesh, the original image is modified by warping to provide the rendered image, and by analyzing the visibility, it is determined which elements should be taken from the new generated image and which from the rendered image. Finally, blending of the generated image and the rendered image is performed to obtain the final result.

Prior art pure warping methods based on 3D mesh cannot extract invisible parts of the face. Typically, these areas include cheeks and ears in the case of distortion correction, and the invisible side of the face in the case of head rotation. Depending on the degree of portrait correction required, the areas to be drawn can vary from a few pixels to a noticeable part of the image.

The proposed invention uses GAN both to generate a new generated (synthesized) image (to restore invisible facial details) and to restore depth information for warping based on a 3D mesh. Thus, both the new generated image and the rendered image are consistent with each other, so they can be easily blended using a visibility mask, which will be described below.

The proposed method is shown schematically in Fig. 2. The proposed method contains the following steps:

1) The user selects an image containing at least one face image for editing. The selection is made from the device memory or the user takes a photo with the camera and selects this photo for editing. The photo may be a selfie portrait or any other image containing at least one face. The user may select at least one face image in the photo selected for editing, and may also select arbitrary new parameters and camera position, i.e. a new head rotation angle in the selected image (which will be described in step 5).

2) Detection and segmentation of one face image in the selected image are performed to obtain an input face image (“Input” in Fig. 2), which is a segmented face image. The detection process includes detection of face images in the selected image. The segmentation process includes cutting out an image of one face from the selected image. The proposed method implements editing of one face image. To edit all face images, it is necessary to apply the proposed method to each face image separately. The face image to be edited can be selected by the user in step 1, or all face images in the selected image are edited one by one. Segmentation is performed using a suitable preprocessing pipeline, which is taken from the corresponding GAN, such pipelines are known in the prior art (see, for example, source [6]).

3) Initialization: To render images of the same face but rotated at a different angle (new face angle) using 3D-GAN, we first need to obtain the original latent face code of the segmented face image, as well as the original parameters and camera position of the segmented face image.

The camera position is represented by a matrix that defines the position of the camera in space. It can be represented as a rotation and translation matrix that defines a projective transformation from a point X in 3D space to a point x (2D coordinates in image space).

The camera parameters are the matrix,

characterizing the internal parameters of the camera, for example, in a simple case, a perspective camera has 4 parameters that determine the projective transformation on the image plane:

- Сх and Су, which determine the projection of the optical center of the camera onto the image plane,

- Fx and Fy - the focal length of the camera in pixels (note that Fx and Fy may differ, since the pixel aspect ratio is not necessarily the same. In the case of square pixels, Fx=Fy).

More complex camera models are known, which are the basis for a larger number of parameters known in the art and not related to the development of the invention. The initial parameters and the position of the camera are initially unknown. The initial parameters and the position of the camera are determined using a neural network (as described below). Obtaining the initial hidden face code for the segmented face image and the initial parameters and the position of the camera for the segmented face image are performed as follows:

a) The initial parameters and the camera position are determined according to the segmented face image. In particular, the segmented face image is fed to the input of a neural network that predicts the parameters and the camera position (Deep3D FaceRecon in Fig. 2). After processing, said neural network outputs the initial parameters and the camera position, with ₀ . with ₀ is a tensor describing the parameters and the camera position. Processes and suitable neural networks for determining the parameters and the camera position are known in the art (see, for example, [7]). Figure 2 illustrates the known neural network Deep3D FaceRecon [7] as an example.

b) The original latent code of the face (it is denoted as w ₀ in Fig. 2) is determined according to the segmented face image. The segmented face image is fed to the input of a neural network, such as TriPlaneNet in Fig. 2, which predicts the latent code of the face. After processing, said neural network outputs the original latent code of the face w ₀ (the neural network TriPlanetNet is illustrated as an example).

The latent code is a tensor describing the image. The use of latent codes is known from the prior art (see, for example, [21]). The processes for determining the latent code of a face are known from the prior art (see, for example, [1]).

TriPlanetNet is trained to predict the hidden code for 3D-GAN so that using this hidden code and certain camera parameters, an image close to the original can be obtained. Accordingly, it is trained on photographs (real or artificial) of different people to predict the hidden code from the image.

Steps (a) and (b) are performed in parallel and are called the initialization process (initialization in Fig. 2).

4) Optimization. The 3D-GAN (generative artificial neural network (generative pipeline)) part does not produce an output image that is sufficiently similar to the segmented face image using the original parameters w ₀ and c ₀ . Therefore, several additional optimization steps need to be performed in the 3D-GAN to obtain a generated output image (Generated Image in Fig. 2) that is very similar to the input segmented face image. The following optimization process is used (see optimization in Fig. 2), in which:

feed the predicted camera parameters and position and the predicted hidden face code to the input of the 3D-GAN,

The output of 3D-GAN is a generated image,

calculate the loss function (loss function in Fig. 2) between the segmented face image and the generated image,

change the predicted parameters and position of the camera and the predicted hidden code of the face,

feed the variable parameters and camera position and the variable hidden face code to the input of the 3D-GAN.

The mentioned iterative optimization process is carried out until the loss function reaches the minimum of the function. The parameters and position of the camera, as well as the hidden code of the face, that satisfy this condition are the optimal parameters and position of the camera. and the optimal hidden face code . When using the obtained parameters, the personality traits in the original image will be preserved in the image obtained using 3D-GAN, i.e. the optimal parameters and camera position , as well as the optimal hidden face code correspond to the original segmented face image.

It should be noted that during the iterative optimization of 3D-GAN, the weights are not changed, only w and c are changed, for example, by gradient descent using a loss function. The gradients are calculated by backpropagating the error through the 3D-GAN network. In this way, it is calculated which values of the vectors w and c contribute the most to this error, and they are changed to reduce this error.

The loss function shows how similar one image is to another. The larger this function, the less similar the images are to each other.

In the proposed invention, the loss function consists of two components:

- LPIPS loss, which is a measure of the similarity (known from the state of the art https://richzhang.github.io/PerceptualSimilarity/) of one image to another image that is close to human perception of images, and

- landmark loss, which is defined as the sum of the distances between key points of the face.

The loss function can be calculated by any method known from the prior art, the main thing is that it satisfies the main condition - it determines the degree of similarity of two images.

Example of optimizing camera position and parameters (from ₀ ):

According to DisCo [3], the initial camera displacement i _Z0 (the initial distance from the camera to the center of the face (head)) is set small enough. i _Z0 is the distance from the camera to the center of the face, with which the focal length f of the camera is optimized, i _Z0 is changed during optimization. The distance i _Z0 is chosen experimentally, and if you take too small a distance (i.e., the camera position is too close to the face), the face will go beyond the image; if you take too large a distance (i.e., the camera position is too far from the face), this will also worsen the convergence, since the convergence can remain in a local minimum, which does not correspond to the real distance. Ideally, you need to choose an initial distance that will ensure stable convergence of the algorithm. The initial distance is experimentally selected on a large number of photographs and the optimal initial approximation is selected.

According to the change of t _z (the distance from the camera to the center of the face (head)), the focal length f of the camera changes from the original f ₀ (f in this example represents the position and parameters of the camera (c ₀ )). Each time t _z changes, a new value of f is obtained. In this case, the estimated depth d ₀ to the eyes in the original face image is used and remains unchanged. The estimated depth d ₀ to the eyes is estimated for the segmented face image and is the distance from the camera to the eyes in the segmented face image. When processing data containing the new value of f, a new image is obtained using 3D-GAN. Based on the new image, a loss function is calculated, which will be described below.

Focal length f of the camera:

Where:

d ₀ - the depth to the eyes obtained by the assessment,

t _z0 - the initial distance from the camera to the center of the face (head).

In detail, after the initialization process, the face latent code and the camera position and parameters are adjusted jointly (using an iterative process known from the prior art, such as the gradient descent method): Expression (1) below shows that with the optimal face latent code and the optimal camera parameters and position ( ) the minimum condition is achieved . Thus, the optimal are those optimal parameters and the position of the camera and the hidden code of the face, at which the loss function L reaches the minimum point of the function:

Where:

is the image produced by a pre-trained 3D-GAN parameterized by weights θ over the latent code w and camera parameters c;

x is the input segmented face image.

More precisely, the target is defined as a combination of an LPIPS loss that compares the perceived similarity of the input image x and the generated image , and losses on facial landmarks, which are specific points in the image (e.g. face contour, eyes), that is, the sum of the distances between the key points of the face in the input segmented face image and in the generated image is compared:

L - losses, the sum of two losses (LPIPS losses and landmark losses) with coefficients,

x - input segmented face image,

G - 3D-GAN, takes as input w - hidden face code, c - camera position and parameters;

G depends on the θ GAN parameters;

- weighting factors with which losses are summed up;

LPIPS is a loss that compares the perceived similarity of the input image x and the generated image ,

f(x) calculates landmarks in the image,

Calculates landmarks in the generated image.

Landmarks are detected using a neural network that performs optimization (e.g., the MediaPipe FaceMesh-V2 landmark estimation model from [8]) on the segmented face image, and a new image is generated using 3D-GAN such that the landmarks in the generated image have the same locations as those in the segmented face image. The landmark loss is the sum of the distances between the facial keypoints in the original image and the compared generated image. In the formula below, the landmark loss is the sum of the squared distances:

- a set of (x, y) coordinates of points on the generated and reference (original) images, respectively, we have ||M|| the number of sets of (x, y) coordinates of points.

Losses take into account the total deviation of all sets (x, y) of coordinates of points from the reference ones (that is, from the coordinates of the points of the original image (the greater the distance between key points, the higher the loss function will be)), where m and These are normalized 3D key points, a. is the number of landmark points, no additional constraints are imposed based on landmark uncertainty. The hidden code and camera parameters are optimized with a learning rate of 0.001 over 200 iterations.

Instead of the exhaustive optimization-based GAN inversion used in DisCo [3], the inversion task is solved using an E-network encoder that maps the real image to a latent code. The camera-conditioned encoder TriPlaneNet [1] is chosen because it is able to separate geometry from camera effects, which is important in this case.

During the optimization, gradient calculations with respect to the latent code of the face and the camera alternate, and they are optimized using the cyclic enumeration method, i.e. the parameters are optimized one after the other. In the first step, the camera parameters are fixed and the latent code is optimized; in the second step, on the contrary, the latent code is fixed and the parameters and position of the camera are optimized, and this is done in a similar way until the optimal latent code of the face and the optimal position and parameters of the camera ( ) will not be achieved. In general, this is just one optimization strategy that is not mandatory to follow. Any other known method can be used. Experiments prove that the alternating update strategy provides superior results compared to the joint optimization used in DisCO.

The following steps relate to the synthesis of a new perspective (see Fig. New Generated Image and Depth Map. Optimal Hidden Code faces obtained in the previous stage, as well as arbitrary new parameters and camera position ( and C _novel in Fig. 2) are fed to the input of the 3D-GAN. Arbitrary new parameters and the position of the c _novel camera correspond to a new view of the face (a new view of the same face) and can be selected either arbitrarily or by the user when selecting an image for editing (step 1). For example, to select new parameters and the position of the c _novel camera, the user can use sliders on the screen of the electronic device, with which the user can set the rotation of the head to the sides, up and down, as well as the distance from the center of the head to the center of the camera.

The 3D-GAN generates a new generated image (New Generated Image in Fig. 2) with a face image corresponding to the segmented face image, with a new face view corresponding to the new parameters and camera position, i.e. from a different angle.

The newly generated image can contain new, previously invisible parts of the face (such as the right ear when turning left) while maintaining the quality (Original Resolution) of the original segmented face image.

In addition, 3D-GAN predicts the depth map (Depth Map in Fig. 2) of the new generated image.

b) 3D Mesh. The depth map is processed using a deterministic 3D mesh based warping approach, such as described in [9]. The 3D Mesh of the new generated image (a 3D representation of the face as a 3D polygonal mesh) is constructed using the predicted depth map (3D Mesh in Fig. 2).

7) Warping based on a 3D mesh, the resulting image is rendered. This 3D mesh, which is a parametric 3D approximation of the shape and surface of a 3D object, is then projected onto the image plane with optimal parameters and camera position.

This projection gives an idea of how the 3D polygonal mesh (the geometric representation of the face in 3D space) would appear if this face position were captured with optimal camera settings and position. , predicted for the original segmented face image (arrows from the 3D mesh and to "3D Grid Based Warping" in Fig. 2).

Based on this projection, the rendered image is generated (the rendered image in Fig. 2). The rendered image is generated using only the pixels of the segmented face image. For the rendered image, areas that were not visible in the segmented face image are not generated, but only the pixels from the segmented face image are used.

The resulting rendered image shows how the face would be viewed from a new angle, that is, with some specific new parameters and camera position (c _novel ).

In other words, the rendered image is an image obtained by transforming the original image using mesh-based warping (in a manner known from the prior art). Figuratively speaking, the original segmented face image is stretched onto a 3D mesh, and the 3D mesh is colored with the colors of the original segmented face image. And then this colored 3D mesh is displayed from a different angle (according to the new camera parameters c _novtl ).

This allows parts that were not visible in the original segmented image to be revealed; in this case, “holes” are formed in the image, which are determined by analyzing the visible parts. These holes are filled with parts of the generated image during the blending step (see step 8).

8) Vertex Visibility Analysis, Visibility Mask. The visibility (whether visible or not) of each vertex of a 3D polygonal mesh is determined when projecting the 3D polygonal mesh onto the image plane with optimal parameters and camera position. (arrows to "Vertex Visibility Analysis" in Fig. 2). Specifically, first the visibility for each vertex on the mesh is calculated using the z-buffer in the rasterization method. Then, mesh edges that are nearly orthogonal (have an angle greater than 80°) are filtered out in the viewing direction.

Since the face is represented as a 3D mesh (polygon mesh), some vertices of this mesh are not visible at different angles. Based on this definition, the visible vertices (not occluded, for example, when the person is facing the camera with his left side, the right side is not visible) are selected so that the corresponding normal of the triangle of the polygon mesh faces the camera (the normal to the triangle is the direction perpendicular to the plane of the triangle). The visibility mask is generated according to the above analysis.

The use of visibility masks in computer graphics and rendering is known in the art (see, for example, VM Visibility Mask (telecomtrainer.com)). A visibility mask (VM) is a technique used to optimize the rendering process by selectively rendering only the visible parts of a scene. The visibility mask is represented by a buffer or data structure that keeps track of the visibility of individual pixels or fragments of a scene. A coarse mesh is obtained by connecting adjacent vertices on a grid. It is further refined by bilateral blurring antialiasing to avoid any unnatural sharp edges in the rendered image. A kernel of size 5 is used. Finally, the mesh is projected using the estimated original camera position to obtain texture coordinates, and the texture is resampled to obtain the new view.

9) Blending, Result. The newly generated image and the rendered image are blended using a visibility mask (Blend in Fig. 2). Accordingly, these two images are well-matched in structure, which allows them to be composed with minimal effort. Image blending is a technique known from the prior art that allows part of one image to be inserted into another in such a way that the composition of the images looks natural, without seams at the boundaries of the insert. Image blending is performed by adding two images (the newly generated image and the image obtained by blending) with different weight coefficients, using the weight coefficients from the visibility masks, and also using a method for smoothing the edges of these two images being added. Such methods are known from the state of the art, for example, Blending with the Laplace Pyramid: http://graphics.cs.cmu.edu/cours es/15-463/2005_fall/www/Lectures/Pyramids.pdf.

By means of this operation, areas are synthesized that were not visible at the original angle (observation) of the face, but became visible at the changed angle of the face (Result in Fig. 2).

The described method focuses exclusively on cropped areas of the face, but it can be easily extended to work with full-frame images, for example by following the steps described in [3]. Faces are detected using the solution [4] and a mask is applied using MODNet [5].

EXPERIMENTS

Datasets

Facial distortion removal was evaluated on two different publicly available datasets. The Caltech Multi-Distance Portraits (CMDP) [9] contained frontal portraits of 53 individuals with different facial attributes, each photographed from seven distances. The In-the-Wild Images [3] represented in-the-wild heavily distorted portraits taken from the web. Since no reference or ground truth (GT) images were available, only the reference images were used for qualitative comparison.

Head rotation datasets

A Head Rotation (HeRo) dataset was collected, containing portraits of 19 people with different attributes (glasses, facial hair, facial expressions). There were 68 series of four photographs each.

The portraits were taken using the front cameras of four Samsung Galaxy S23FE smartphones mounted on a tripod (Fig. 3). Figure 3 illustrates the setup used to collect the data set and samples from it. The invention was subsequently tested on this data set. From the collected photographs, we have, for example, what a person should look like from the left angle, given the front angle. Accordingly, the proposed method was tested on a frontal photograph and head pose correction was performed at left/top/right angles, then these photographs were compared with real photographs. All devices were synchronized with 1, so that a series of 4 images were taken simultaneously.

Metrics

Four standard evaluation metrics were used to evaluate portrait perspective correction. Specifically, the photometric errors between the matched output images and the corresponding references were calculated, including PSNR, SSIM, and LPIPS [10]. In addition, the preservation of facial features was assessed using the ID metric, which is the cosine distance between the predicted and reference facial features from ArcFace [11].

RESULTS

Correcting facial distortion

Table 1 shows the evaluation of the claimed approach in comparison with competing methods of facial distortion correction. It is obvious that the claimed method significantly outperforms the others in most indicators, especially in preserving facial features. By sampling pixels from the original image, the claimed solution preserves important personality details, such as eye color, wrinkles, earrings, etc.

Table 1: Quantitative comparison of facial distortion correction methods on CMDP [9]. The best results are highlighted in bold.

In this table 1, the method disclosed here is compared with its competitors. Four metrics are used that compare frames pairwise. PSNR, SSIM compare the quality of photos, LPIPS compares the perceived similarity of images, ID compares the similarity of individuals. The up arrow means that the higher the better, and the down arrow means that the lower the better. The proposed method outperforms its competitors in all metrics.

Portraits corrected by different approaches, including the claimed solution, are shown in Fig. 4. A comparison of the distortion correction method disclosed here with other methods is illustrated. The claimed method copes excellently with heavily distorted faces. The neural network not only successfully restores occluded areas, but also preserves important details of the personality, which is highlighted in the fragments provided. Two samples are used: the first column contains the input image, the last column contains what should be obtained. The second and fourth rows show the details from rows 1 and 3, demonstrating the effectiveness of the method disclosed here. The proposed method preserves the details, for example, we preserve the earrings and eyes, while 3DP, HFGI3D, TriplaneNet, DisCO do not. At the same time, it is seen in rows 1 and 3 that the perspective distortion has also been corrected, compared to the Fried and Shin methods. Warping-based solutions [9] and [12] do not seem to have a significant impact on the input data. In contrast, 3DP introduces noticeable changes, but at the same time increases distortions, so that the middle part of the face shows less distortions, and the head and chin are deformed. Generative TriplaneNet and DisCo create a face that looks noticeably different. HFGI3D [2] is able to preserve personality features by combining warping and generation, but at the same time creates visual artifacts and tends to over-smooth the images. The claimed method generates faces with less perspective distortions, while preserving personality features, as shown in Fig. 5. The proposed method (Proposed method in Fig. 5) shows a realistic result, generating new, previously invisible areas and preserving the original quality. The claimed method successfully provides a balance between generating new parts of the face and preserving personality features.

To identify the necessary and sufficient number of optimization stages, the number of iterations is changed and the achieved quality is assessed. As can be seen in the graphs in Fig. 6, the number of iterations of optimization of the hidden face code and camera parameters affects the quality of the output result. The x-axis shows the number of iterations, and the y-axis shows the quality metric of the algorithm. For all quality metrics, about 200 optimization stages are required, during which the method reaches a plateau in quality. Optimal quality is achieved during the first 100 iterations, while further optimization gives a slight increase in ID, but does not improve LPIPS, SSIM and PSNR.

PSNR peaks after 100 iterations and then decreases slightly as the number of iterations increases. Accordingly, 200 iterations are used by default. Note that the basic DisCo variant requires approximately 1200 iterations of comparable complexity.

Head posture correction

Table 2 presents the evaluation of the approach compared to competing head pose correction methods. The qualitative results are shown in Fig. 7. The proposed method is able to change the pose of a person while preserving the original image quality and the person’s similarity to himself, better than any other competing method. The main advantage of the proposed approach is the preservation of personality traits, as evidenced by the exceptional ID score.

Table 2: Quantitative comparison of head pose corrections from the HeRo dataset.

In this table 2, the method disclosed here (ours) is compared to its competitors. Four metrics are used that compare frames pairwise. PSNR, SSIM compare the quality of photos, LPIPS compares the perceived similarity of images, ID compares the similarity of individuals. An up arrow means that the higher the better, and a down arrow means that the lower the better.

CONCLUSION

This application discloses a method for selfie editing that removes facial perspective distortion when cropping a close-up face. This approach complements robust, physically correct 3D warping with the flexibility and expressiveness of a 3D generative neural network. The resulting image is a blend of a warped image obtained using grid-based rendering and another image obtained using a 3D-GAN. Visibility-based blending is used, in which visible regions are reprojected and occluded parts are reconstructed using a generative model. Evaluation on face dewarping tests and a new head rotation dataset confirm that the claimed method provides more realistic results with finer details and better preservation of personality features compared to existing selfie editing methods, and sets a new state-of-the-art in facial dewarping and head pose correction tasks.

LIST OF INFORMATION SOURCES

[1] Ananta R. Bhattarai, Matthias NieBner, and Artem Sevastopolsky, "Triplanenet: An encoder for eg3d inversion," 2024.

[2] Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen, "High-fidelity 3D-GAN inversion by pseudo-multi-view optimization," 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 321-331, 2022.

[3] Zhixiang Wang, Yu-Lun Liu, Jia-Bin Huang, Shin'ichi Satoh, Sizhuo Ma, Gurunandan Krishnan, and Jian Wang, "DisCo: Portrait distortion correction with perspective-aware 3D-GANs," 2023.

[4] Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann, “BlazeFace: Sub-millisecond neural face detection on mobile gpus,” 2019.

[5] Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson W.H. Lau, "Modnet: Real-time trimap-free portrait matting via objective decomposition," in AAAI, 2022.

[6] Eric Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, S. Khamis, Tero Karras, and Gordon Wetzstein, “Efficient geometry-aware 3d generative adversarial networks,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16102-16112, 2021.

[7] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong, “Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 285-295, 2019.

[8] Camillo Lugaresi, Jiugiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann, “Mediapipe: A framework for perceiving and processing reality,” in Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019, 2019.

[9] Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein, “Perspective-aware manipulation of portrait photos,” ACM Transactions on Graphics (TOG), vol. 35, no. 4, pp. 1-10, 2016.

[10] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.

[11] Jiankang Deng, J. Guo, J. Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou, “Arcface: Additive an-gular margin loss for deep face recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 5962-5979, 2018.

[12] YiChang Shih, Wei-Sheng Lai, and Chia-Kai Liang, “Distortion-free wide-angle portraits on camera phones,” ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1-12, 2019.

[13] Meng-Li Shih, Shin-Yang Su, Johannes Kopf, and Jia-Bin Huang, “3d photography using context-aware layered depth inpainting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8028-8038.

[14] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or, “Pivotal tuning for latent-based editing of real images,” ACM Transactions on Graphics (TOG), vol. 42, no. 1, pp. 1-13, 2022.

[15] Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwang seok Ryoo, and Seung Wook Kim, “3D-GAN inversion with pose optimization,” 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2966-2975, 2022.

[16] Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo and Xuelong Li, One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field, arXiv preprint, arXiv:2304.05097, 2023.

[17] Jiawei Yang, Marco Pavone and Yue Wang, “FreeNeRF: Improving Few-shot Neural Rendering with Free Freguency Regularization,” Conference on Computer Vision and Pattern Recognition (CVPR) 2023.

[18] Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqbal, and Jan Kautz, “Generalizable One-shot Neural Head Avatar,” arXiv preprint.

[19] Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Ogras and Linjie Luo, “PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360 degree,” arXiv preprint, arXiv:2303.13071, 2023.

[20] Yue Wu, Yu Deng, Jiaolong Yang, and Fangyun Wei, Qifeng Chen, and Xin Tong, “AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars,” Advances in Neural Information Processing Systems (NeurlPS), 2022.

[21] Karras, T., Laine, S., & Aila, T. (2018). A Style-Based Generator Architecture for Generative Adversarial Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 4396-4405. https://arxiv.org/abs/1812.04948

Claims (33)

1. A method for editing a face image in an image, the method comprising: a) the user selecting an image for editing containing at least one image of a face; b) detecting images of faces in the selected image, with segmentation of one face image from among the detected face images; c) feeding the segmented face image to the input of a neural network that predicts the parameters and camera position for the segmented face image, with the output receiving the initial parameters and position of the camera; d) feeding the segmented face image to the input of a neural network that predicts the latent code of the face, with the latent code of the face being obtained at the output of the neural network that predicts the latent code of the face; e) performing an iterative optimization process using a generative adversarial network (3D-GAN), for this purpose: - feed the predicted parameters and camera position and the predicted hidden face code to the input of the 3D-GAN, - receive a 3D-GAN generated image at the output, - calculate the loss function between the face image obtained by segmentation and the generated image, iteratively change the predicted parameters and camera position and the predicted hidden face code, and - feed the modified parameters and camera position and the modified hidden face code to the 3D-GAN input, wherein the iterative optimization process is carried out until the said loss function reaches a minimum of the function in which the parameters and position of the camera and the latent code of the face satisfying this condition are the optimal parameters and position of the camera and the optimal latent code of the face; f) feeding the optimal hidden face code to the input of said 3D-GAN, with the supply of arbitrary new parameters and camera positions to the input of the mentioned 3D-GAN; g) generating, using said 3D-GAN, a new generated image with a face image corresponding to the segmented face image, with a new face view corresponding to said new parameters and camera position, with prediction of the depth map of the newly generated image; h) processing the predicted depth map to construct a 3D mesh of the new generated image; i) projection of the constructed 3D mesh onto the image plane with optimal parameters and camera position, with the generation of a rendered image based on the given projection; j) determining which vertices of a 3D mesh are visible and which are occluded after projection onto the image plane with optimal camera parameters and position, obtaining a visibility mask based on said definition; steps (i)-(j) are performed in parallel; k) blending the new generated image and the rendered image using a visibility mask, producing an edited segmented face image showing the face from a new angle; l) transferring the obtained edited segmented face image to the selected image; steps (b)-(k) are performed for at least one face in the selected image; and m) displaying the selected image with at least one edited segmented face image on the screen to the user. 2. The method according to claim 1, wherein the arbitrary new parameters and position of the camera are selected by the user. 3. The method according to any one of paragraphs 1, 2, additionally comprising the step the user selecting one face image for segmentation from among the detected face images in the selected image. 4. The method according to any one of claims 1-3, wherein steps (c) and (d) are performed in parallel.