WO2025097814A1 - New viewpoint image synthesis method and system, electronic device, and storage medium - Google Patents

Abstract

Disclosed are a new viewpoint image synthesis method and system, an electronic device, and a storage medium. The method comprises: acquiring a sample image and first camera pose information corresponding to the sample image; by means of a spatial matching method, optimizing the first camera pose information of the sample image to obtain second camera pose information corresponding to the sample image; on the basis of the sample image and the second camera pose information, training an initial neural radiance field to obtain a trained neural radiance field; and by means of the trained neural radiance field, performing new viewpoint rendering to obtain a new viewpoint image. According to the solution of the embodiments of the present application, a neural radiance field is trained on the basis of optimized camera pose information, thereby significantly improving the training effect; and by means of the trained neural radiance field, a more realistic new viewpoint synthesized image can be obtained.

Description

New viewpoint image synthesis method, system, electronic device and storage medium

This application claims priority to the Chinese patent application filed on November 9, 2023, with application number 202311492488X, and invention name “A new viewpoint image synthesis method, system, device and storage medium”, the content of which should be understood as incorporated into this application by reference.

Technical Field

This article relates to but is not limited to the field of image processing technology.

Background Art

The continuous development of artificial intelligence technology and hardware computing power has brought new development opportunities to video/image processing technology. Breaking through the viewpoint constraints of actual video/image shooting and generating new images from various viewpoints on demand has gradually become a necessary function for many application systems. This technology of generating new videos/images from viewpoints that do not exist in the input video/image is called new viewpoint synthesis technology. How to obtain more realistic new images is an important aspect that new viewpoint image synthesis technology strives to pursue.

Summary of the invention

The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

The embodiments of the present application provide a new viewpoint image synthesis method, system, electronic device and storage medium, which perform neural radiation field training based on optimized camera posture information, significantly improve the training effect, and can obtain a more realistic new viewpoint synthesized image.

The present application provides a new viewpoint image synthesis method, including:

Acquire a sample image and first camera posture information corresponding to the sample image;

By using a spatial matching method, the first camera posture information of the sample image is optimized to obtain the second camera posture information corresponding to the sample image;

Training the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field;

The trained neural radiation field is used to perform new viewpoint rendering to obtain a new viewpoint image.

The present application also provides a new viewpoint image synthesis system, including:

Client and server;

The client is configured to obtain a sample image and first camera posture information corresponding to the sample image;

The server is configured to optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image;

The server is also configured to train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field.

An embodiment of the present application further provides an electronic device, including one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement Now, a new viewpoint image synthesis method as described in any embodiment of the present application.

An embodiment of the present application further provides a computer storage medium, in which a computer program is stored, wherein the computer program is configured to execute the new viewpoint image synthesis method as described in any embodiment of the present application when running.

The new viewpoint image synthesis system framework provided in the embodiment of the present application adopts a system architecture that combines the client and the server, and deploys computing power in a distributed manner. It can support flexible rendering execution deployment and meet the functional requirements of more application scenarios.

Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

FIG1 is a flow chart of a new viewpoint image synthesis method provided in an embodiment of the present application;

FIG2 is a flow chart of another new viewpoint image synthesis method provided in an embodiment of the present application;

FIG3 is a flow chart of another new viewpoint image synthesis method provided in an embodiment of the present application;

FIG4 is a flow chart of another new viewpoint image synthesis method provided in an embodiment of the present application;

FIG5 is a flow chart of another new viewpoint image synthesis method provided in an embodiment of the present application;

FIG6 is a structural diagram of a new viewpoint image synthesis system provided in an embodiment of the present application;

FIG. 7 is a structural diagram of another new viewpoint image synthesis system provided in an embodiment of the present application.

Details

In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

It is understood that the terms "first" and "second" used in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

As used herein, the singular forms "a", "an" and "the" may include the plural forms as well, unless the context so dictates. It should also be understood that the terms "include/comprise" or "have" etc. specify the presence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and/or" used in this specification includes any and all combinations of the relevant listed items.

The present application embodiment provides a new viewpoint image synthesis method, as shown in FIG1 , comprising:

Step 110, obtaining a sample image and first camera posture information corresponding to the sample image;

Step 120, optimizing the first camera pose information of the sample image by a spatial matching method to obtain second camera pose information corresponding to the sample image;

Step 130, training the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field;

Step 140, performing new viewpoint rendering using the trained neural radiation field to obtain a new viewpoint image.

In some exemplary embodiments, step 110 includes: obtaining an image to be processed, and obtaining camera posture information corresponding to the image to be processed by a real-time positioning and mapping method;

According to the viewpoint diversity principle, multiple frames of images in the image to be processed are selected as the sample images, and the camera posture information corresponding to the sample images is the first camera posture information.

In some exemplary embodiments, the simultaneous positioning and mapping method includes: a SLAM (Simultaneous Localization and Mapping) algorithm.

It can be understood that in some exemplary embodiments, the image to be processed includes multiple frame images in the captured video, also referred to as multiple image frames, or multiple frames for short. That is, the original source of the sample image is obtained by capturing the video. It can be a video facing the target object, or a video circling the target object, or other videos captured on a free path to the target object, and is not limited to a specific shooting method.

In some exemplary embodiments, selecting multiple frames of images from the to-be-processed images as the sample images according to the viewpoint diversity principle includes:

According to the camera posture information corresponding to the image to be processed and in accordance with the viewpoint diversity principle, sparse multiple frame images are selected from the image to be processed as the sample images.

Since the image to be processed generally includes multiple frames of images under multiple viewpoints, the corresponding viewpoint can be known according to the camera posture information corresponding to each frame of image. According to the principle of viewpoint diversity, key frames are selected, and multiple frame images are dispersedly selected as key frames within the viewpoint range corresponding to all frame images, so that the coverage rate of the viewpoint corresponding to the selected frame image to all viewpoints exceeds the first set ratio. Among them, frame images of the second set ratio are selected from all frame images as key frames. The first set ratio and the second set ratio are flexibly set as needed. For example, 40% (the second set ratio) of images are selected from all frame images as key frames, and the viewpoints corresponding to these key frames cover 80% (the first set ratio) of the viewpoint range of all frame images, and the viewpoints corresponding to any two key frames can be the same, or different.

It can be understood that selecting frame images as key frames from the images to be processed according to the viewpoint diversity principle can reduce the number of samples and maintain the richness of the samples, which can ensure the effectiveness of training and the training effect while reducing the amount of training data.

In some exemplary embodiments, the optimizing the first camera pose information of the sample image by a spatial matching method to obtain the second camera pose information corresponding to the sample image includes:

Extracting feature points of the sample image;

Establishing an initial map according to the first camera posture information;

According to the initial map, obtaining the spatial distance between the sample images;

Matching the feature points of the sample image whose spatial distance is within the spatial distance threshold to obtain a matching result;

According to the matching result, the first camera posture information is optimized to obtain the second camera posture information.

In some exemplary embodiments, optimizing the first camera posture information according to the matching result to obtain the second camera posture information includes:

The first camera posture information is optimized through bundle adjustment to obtain the second camera posture information.

The sample image includes a plurality of selected frame images, each frame image corresponds to a first camera posture information, and an initial map is established according to the first camera posture information; and feature points are extracted for each frame image.

In some exemplary embodiments, the optimizing the first camera pose information of the sample image by a spatial matching method to obtain the second camera pose information corresponding to the sample image includes:

Extracting feature points of the sample image;

Establishing an initial map according to the feature points of the sample image and the first camera posture information corresponding to the sample image;

According to the initial map, obtaining the spatial distance between the sample images;

Matching the feature points of the sample image whose spatial distance is within the spatial distance threshold to obtain a matching result;

The initial posture transformation is optimized through bundle adjustment to obtain the camera posture transformation.

The sample image includes a plurality of selected frame images, feature points are extracted for each frame image, each frame image corresponds to a first camera posture information, and an initial map is established according to the feature points and the first camera posture information.

It can be understood that matching the feature points of the sample images whose spatial distances are within the spatial distance threshold to obtain matching results means matching the feature points between key frames with similar spatial distances, and then optimizing the first camera posture information using bundle adjustment, which can reduce reprojection errors. Compared to some SFM (Structure from Motion) algorithms that do not consider spatial distance constraints and match feature points before placing key frames into a map, the embodiment of the present application matches the feature points of sample images (key frames) whose spatial distances are within the spatial distance threshold and then optimizes them, which not only improves the calculation effect, but also avoids erroneous matching of frame images at a distance due to repeated textures.

In some exemplary embodiments, step 130 includes:

Acquire a plurality of training samples according to the sample image and the second camera posture information, wherein each training sample is composed of light emitted by a pixel of the sample image and a color corresponding to the pixel;

The initial neural radiation field is trained according to the multiple training samples.

In some exemplary embodiments, the step of acquiring a plurality of training samples according to the sample image and the second camera posture information includes:

The light emitted by the pixel point is determined according to the second camera posture information corresponding to the sample image where the pixel point is located and the position of the pixel point in the sample image.

In some exemplary embodiments, obtaining a plurality of training samples according to the sample image and the second camera posture information includes:

Resolve the sample image and the second camera posture information into the light emitted by each pixel;

The light emitted by each pixel and the color of the pixel form a sample.

In some exemplary embodiments, the color of each pixel p of a sample image is c. Combining the second camera posture information and the pixel position corresponding to the sample image, the light emitted by the pixel can be obtained and recorded as l(p, d). Where p = (x, y, z) is the position coordinate of the pixel point in the three-dimensional Cartesian space coordinate system. is the solid angle parameter of the light direction in the spherical coordinate system, where θ represents the polar angle or latitude, which is the angle between the vector from the reference axis (usually the positive z-axis) to the point and the reference axis. The value range of θ is usually 0 to π. Represents azimuth or longitude, which is the angle between the projection of a reference direction (usually the positive x-axis) to a point on a reference plane and the reference direction. The value range of is usually 0 to 2π.

It can be understood that a sample image includes multiple pixels, and correspondingly obtains multiple training samples consisting of light and color emitted by the pixels, that is, one pixel corresponds to one training sample, and the multiple sample images obtain more training samples.

In some exemplary embodiments, training the initial neural radiation field according to the plurality of training samples comprises:

After randomly shuffling all training samples, the initial neural radiation field is trained.

For each input sample (l, c), l is light and c is color. In the divided space, the coordinate p is spatially deformed and its position is queried through hash coding. The multi-layer perceptron of the node is used to encode it. The encoded features f and d are encoded together using a global multi-layer perceptron to output the color c _pred and the difference disp.

In some exemplary embodiments, the loss function corresponding to the training is

Where n _r is the sample size.

In some exemplary embodiments, the spatial division in the neural radiation field may be a uniform grid-based spatial division or an octree-based spatial division.

In some exemplary embodiments, the training of the initial neural radiation field comprises the following steps: space division, space deformation, hash coding and training;

In some exemplary embodiments, the spatial partitioning includes: performing spatial partitioning on the region of interest by using an octree, if there is a visible camera with a distance to a node that is less than λ times its side length (for example, λ is 3), then the node is evenly divided into eight child nodes. This process is repeated until each node cannot be further subdivided. Each node contains a multi-layer perceptron.

In some exemplary embodiments, the spatial deformation includes: performing spatial deformation on each child node of the octree; that is, in order to better express the space, it is necessary to perform spatial deformation on each child node of the octree, including:

All cameras in the space are denoted as {C _i |i＝1…n _c }, where n _c is the number of cameras. The function that deforms a point in the three-dimensional space to the camera projection space is {y＝G(x)＝{C ₁ (x)…C _nc (x)}. Uniformly sample n _p points in the space {x _j |j＝1…n _p }, and then obtain the deformed point {y _j |j＝1…n _p } through y＝G(x). The matrix composed of the first three eigenvectors of the covariance matrix of {y _j } is denoted as M, and the final space deformation function F(x)＝M·G(x) is obtained.

Through spatial division and spatial deformation, we can effectively process a variety of sample images obtained under flexible shooting methods, which improves the problem of some solutions requiring more constraints on the shooting angles of sample image acquisition. This makes it possible for images obtained under free shooting methods to participate in training as effective samples, thereby improving the training effect.

In some exemplary embodiments, hash coding includes: using multiple hash functions to encode the deformed space, thereby accelerating spatial query.

In some exemplary embodiments, the training includes:

For each input sample (l, c), l is light and c is color. In the divided space, the coordinate p is spatially deformed and its position is queried through hash coding. The multi-layer perceptron of the node is used to encode it. The encoded features f and d are encoded together with a global multi-layer perceptron to output the color c _pred and the difference disp.

In some exemplary embodiments, the loss function for training is

Wherein, n _r is the number of samples, f ₀ , f ₁ are features extracted from two adjacent octree nodes randomly sampled in space, and n _b is the number of samples; in some exemplary embodiments, n _b is set to 10,000.

In some exemplary embodiments, the spatial deformation in the neural radiation field may be a spatial deformation based on normalized device coordinates, or a spatial deformation based on a perspective projection coordinate system.

In some exemplary embodiments, step 140 includes: obtaining the new viewpoint image through the trained neural radiance field according to the viewpoint information to be rendered;

The viewpoint information to be rendered includes the camera posture of the viewpoint to be rendered and the preview image resolution, and the resolution of the new viewpoint image is not less than the preview image resolution.

In some exemplary embodiments, obtaining the new viewpoint image through the trained neural radiance field according to the viewpoint information to be rendered includes:

Calculate the ray of each pixel on the entire image based on the viewpoint information to be rendered and the image height and width;

For each pixel, sampling is performed along the ray through the ray advancing method, and the color value of the sampling point given by the trained neural radiation field is integrated to obtain the color value of the pixel.

It can be known that after the color values of all pixels are obtained according to the viewpoint information to be rendered, the entire image is finally obtained, that is, a new viewpoint image is rendered according to the viewpoint information to be rendered.

In some exemplary embodiments, step 140 includes: generating a three-dimensional model according to the trained neural radiation field, wherein the three-dimensional model includes a three-dimensional mesh and a texture map;

The three-dimensional model is rendered by a three-dimensional rendering method to obtain the new viewpoint image.

It can be understood that the three-dimensional mesh and texture map included in the three-dimensional model can be used to obtain a new viewpoint image through three-dimensional pipeline rendering. In some exemplary embodiments, the new viewpoint image obtained by the trained neural radiation field is recorded as the first new viewpoint image; the new viewpoint image obtained by rendering the three-dimensional model through a three-dimensional rendering method is recorded as the second new viewpoint image.

In some exemplary embodiments, generating a three-dimensional model according to the trained neural radiation field includes:

According to the training results of the neural radiation field, the triangular mesh G = {V, E} that approximates the neural radiation field is calculated by extracting the isosurface algorithm; where vertices V = { _vi |i = 1… _nv }, edges E = { _ei |i = 1… _ne }, _nv is the number of vertices, and _ne is the number of edges.

Assign an offset _Δvi and a weight vector _wi to each vertex _vi , a camera pose _pi of each input image _Ij , and a color value cij at the corresponding vertex _vi . Then, the optimal _Δvi and _cij are solved by minimizing the following _energy function:

Where v _l is the mean of adjacent vertices in the neighborhood of _vi , n _v is the number of vertices, and m is the number of images.

According to the optimization result, the vertices of the triangular mesh G are updated, and the texture map included in the three-dimensional model is generated according to the optimized weight vector _wi ; the triangular mesh G = {V, E} is the three-dimensional mesh included in the three-dimensional model.

The present application also provides a new viewpoint image synthesis method, as shown in FIG2 , including:

Step 210: The first client obtains a sample image and first camera posture information corresponding to the sample image, and sends Send to the server;

Step 220: The server optimizes the first camera pose information of the sample image by a spatial matching method to obtain second camera pose information corresponding to the sample image;

Step 230: The server trains the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field.

In some exemplary embodiments, as shown in FIG3 , the method further includes:

Step 2401: The second client obtains viewpoint information to be rendered and sends it to the server;

Step 2501: The server renders according to the viewpoint information to be rendered using the trained neural radiation field to obtain a new viewpoint image, recorded as the first new viewpoint image, and sends the new viewpoint image to the second client.

In some exemplary embodiments, step 2401 includes: the second client obtains viewpoint information to be rendered and target image resolution, and sends them to the server;

Correspondingly, step 2501 includes: the server renders according to the viewpoint information to be rendered and the target image resolution through the trained neural radiation field to obtain a new viewpoint image, recorded as the first new viewpoint image, and sends the new viewpoint image to the second client.

In some exemplary embodiments, as shown in FIG4 , the method further includes:

Step 260: the server generates a three-dimensional model according to the trained neural radiation field, and sends the three-dimensional model to the second client; wherein the three-dimensional model includes a three-dimensional mesh and a texture map;

Step 270: The second client renders the 3D model using a 3D rendering method according to the viewpoint information to be rendered, to obtain a new viewpoint image, which is recorded as a second new viewpoint image.

In some exemplary embodiments, the first client and the second client are the same client, or different clients, without limitation to a specific aspect.

In some exemplary embodiments, step 260 further includes: the server sending the three-dimensional model to the second client according to the three-dimensional model request of the second client.

In some exemplary embodiments, as shown in FIG5 , the method includes:

Step 2401: The second client obtains information of multiple candidate viewpoints to be rendered;

Step 2402: The second client sends the plurality of candidate viewpoint information to be rendered and the preview image resolution to the server;

Step 2501: the server performs rendering through the trained neural radiance field according to multiple candidate viewpoint information to be rendered and the preview image resolution, obtains multiple candidate new viewpoint images, and sends the multiple candidate new viewpoint images to the second client;

Step 2502: The second client determines a viewpoint information to be rendered from the plurality of candidate new viewpoint images in response to the user's selection instruction;

Step 2503: The second client sends the viewpoint information to be rendered and the target image resolution to the server;

Step 2504: the server performs rendering through the trained neural radiance field according to the viewpoint information to be rendered and the target image resolution to obtain a new viewpoint image (referred to as the first new viewpoint image), and sends the new viewpoint image to the second client;

The target image resolution is greater than or equal to the preview image resolution.

It can be understood that according to this embodiment, the client can first submit preview requirements of multiple new viewpoints to the server, and determine whether it is the new viewpoint effect required by the client according to the corresponding rendering results. After the selection, the server will A higher resolution rendering generates the final new viewpoint image. In the preview stage, in order to improve the server response speed and reduce the waste of server computing power and network resources, a lower resolution preview image is rendered.

The embodiment of the present application further provides a new viewpoint image synthesis system, as shown in FIG6 , comprising:

Client 610 and server 620;

The client 610 is configured to obtain a sample image and first camera posture information corresponding to the sample image;

The server 620 is configured to optimize the first camera pose information of the sample image through a spatial matching method to obtain the second camera pose information corresponding to the sample image;

The server 620 is further configured to train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field.

In some exemplary embodiments, the client 610 is further configured to obtain viewpoint information to be rendered and send it to the server 220;

The server 620 is further configured to perform rendering according to the viewpoint information to be rendered using the trained neural radiation field to obtain a new viewpoint image, recorded as a first new viewpoint image, and send the new viewpoint image to the client 610 .

In some exemplary embodiments, the client 610 is further configured to display the new viewpoint image.

It can be understood that in some exemplary embodiments, neural radiation field training and rendering generation of the first new viewpoint image are both performed on the server side, which can fully utilize the powerful computing power of the server, ensure the rendering effect, and significantly reduce the computing pressure of the client.

In some exemplary embodiments, the server 620 is further configured to generate a three-dimensional model according to the trained neural radiation field, and send the three-dimensional model to the client 610, wherein the three-dimensional model includes a three-dimensional mesh and a texture map;

The client 610 is further configured to render the three-dimensional model by a three-dimensional rendering method to obtain a new viewpoint image, which is recorded as a second new viewpoint image.

In some exemplary embodiments, the client 610 is further configured to obtain viewpoint information to be rendered; render the three-dimensional model by a three-dimensional rendering method according to the viewpoint information to be rendered, and obtain a new viewpoint image, which is recorded as a second new viewpoint image.

It can be understood that in some exemplary embodiments, neural radiation field training is performed on the server side, and the rendering and generation of the second new viewpoint image is performed on the client side. This can overcome the problem that in some application scenarios, the real-time interaction between the client and the server is poor and cannot meet the application needs in a timely manner. In this case, the client can use local rendering to generate a new viewpoint image.

It should be noted that the client includes one or more clients, and the client that provides the sample image and the client that obtains the viewpoint information to be rendered can be the same client, or different clients, and is not limited to a specific aspect.

In some exemplary embodiments, the client 610 is further configured to obtain information of a plurality of candidate viewpoints to be rendered; send the information of the plurality of candidate viewpoints to be rendered and the preview image resolution to the server 620;

The server 620 is also configured to render multiple candidate new viewpoint images based on multiple candidate viewpoint information to be rendered and preview image resolutions through the trained neural radiation field, and send the multiple candidate new viewpoint images to the client 610.

In some exemplary embodiments, the client 610 is further configured to, in response to a user's selection instruction, determine a viewpoint information to be rendered from the multiple candidate new viewpoint images; and send the viewpoint information to be rendered and the target image resolution to the server 620;

The server 620 is further configured to perform rendering through the trained neural radiance field according to the viewpoint information to be rendered and the target image resolution, obtain a new viewpoint image, recorded as a first new viewpoint image, and send the new viewpoint image to the client;

The target image resolution is greater than or equal to the preview image resolution.

The embodiment of the present application also provides a new viewpoint image synthesis system, as shown in FIG7 , including: a client 610 and a server 620;

The client 610 includes: an image acquisition module 6110, a SLAM module 6120 and a key frame selection module 6130; the server 620 includes: a posture optimization module 6210, a neural radiation field training module 6220, a neural radiation field rendering module 6230 and a three-dimensional model generation module 6240;

Wherein, the image acquisition module 6110 is configured to acquire an image to be processed;

The SLAM module 6120 is configured to use a SLAM algorithm to obtain camera posture information corresponding to the image to be processed;

The key frame selection module 6130 is configured to select, according to the viewpoint diversity principle, multiple frames of images in the image to be processed as the sample images, and the camera posture information corresponding to the sample images is the first camera posture information; and send the sample images and the first camera posture information corresponding to the sample images to the server 620;

The posture optimization module 6210 is configured to optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image.

The neural radiation field training module 6220 is configured to train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field.

In some exemplary embodiments, the image to be processed includes multiple frame images in the captured video, also referred to as multiple image frames, or multiple frames for short. That is, the original source of the sample image is obtained by capturing a video. It can be a video facing the target object, or a video circling the target object, or other videos captured on a free path to the target object, and is not limited to a specific shooting method.

In some exemplary embodiments, the client further includes: an interaction module 6140, and the server further includes: a neural radiation field rendering module 6230;

The interaction module 6140 is configured to obtain viewpoint information to be rendered and send it to the server;

The neural radiation field rendering module 6230 is configured to perform rendering according to the viewpoint information to be rendered by using the trained neural radiation field to obtain a new viewpoint image, and send the new viewpoint image to the client;

The interaction module 6140 is also configured to display the new viewpoint image.

In some exemplary embodiments, the client further includes: an interaction module 6140 and a real-time rendering module 6150, and the server further includes: a three-dimensional model generation module 6240;

The interaction module 6140 is configured to obtain viewpoint information to be rendered and send it to the real-time rendering module 6150;

The three-dimensional model generation module 6240 is configured to generate a three-dimensional model according to the trained neural radiation field, and send the three-dimensional model to the client, wherein the three-dimensional model includes a three-dimensional mesh and a texture map;

A real-time rendering module 6150 renders the three-dimensional model by a three-dimensional rendering method according to the viewpoint information to be rendered, to obtain a new viewpoint image;

The interaction module 6140 is also configured to display the new viewpoint image.

An embodiment of the present application further provides an electronic device, including one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the new viewpoint image synthesis method as described in any embodiment of the present application.

An embodiment of the present application further provides a computer storage medium, in which a computer program is stored, wherein the computer program is configured to execute the new viewpoint image synthesis method as described in any embodiment of the present application when running.

It can be seen that according to the new viewpoint image synthesis system architecture provided in the embodiment of the present application, a distributed deployment method is used to combine the computing power advantage of the server for neural radiation field model training, and the shooting convenience and richness of the client for initial sample collection, thereby avoiding the occupation of computing resources that are not powerful on the client itself, and providing a better customer experience for real-time rendering with high stability and reasonable distribution of computing overhead. Based on the first camera posture information obtained by the client, the camera posture is optimized through a spatial matching method before training, which can improve the camera posture accuracy of the sample image, reduce matching errors, and improve the training effect. Some exemplary implementations provide a server rendering solution, and other exemplary embodiments provide a client-side rendering solution based on a three-dimensional model, which fully meets flexible rendering requirements.

It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules/units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims (15)

A new viewpoint image synthesis method, comprising: Acquire a sample image and first camera posture information corresponding to the sample image; By using a spatial matching method, the first camera posture information of the sample image is optimized to obtain the second camera posture information corresponding to the sample image; Training the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field; The trained neural radiation field is used to perform new viewpoint rendering to obtain a new viewpoint image. The new viewpoint image synthesis method according to claim 1, wherein: The obtaining of a sample image and first camera posture information corresponding to the sample image includes: Acquire an image to be processed, and acquire camera posture information corresponding to the image to be processed by a real-time positioning and map building method; According to the viewpoint diversity principle, multiple frames of images in the image to be processed are selected as the sample images, and the camera posture information corresponding to the sample images is the first camera posture information. The new viewpoint image synthesis method according to claim 1, wherein: The method of optimizing the first camera posture information of the sample image by a spatial matching method to obtain the second camera posture information corresponding to the sample image includes: Extracting feature points of the sample image; Establishing an initial map according to the first camera posture information; According to the initial map, obtaining the spatial distance between the sample images; Matching the feature points of the sample image whose spatial distance is within the spatial distance threshold to obtain a matching result; According to the matching result, the first camera posture information is optimized to obtain the second camera posture information. The new viewpoint image synthesis method according to claim 3, wherein: The optimizing the first camera posture information according to the matching result to obtain the second camera posture information includes: The first camera posture information is optimized through bundle adjustment to obtain the second camera posture information. The new viewpoint image synthesis method according to claim 1, wherein: The step of training the initial neural radiation field according to the sample image and the second camera posture information to obtain the trained neural radiation field includes: Acquire a plurality of training samples according to the sample image and the second camera posture information, wherein each training sample is composed of light emitted by a pixel of the sample image and a color corresponding to the pixel; The initial neural radiation field is trained according to the multiple training samples. The new viewpoint image synthesis method according to claim 5, wherein the step of obtaining a plurality of training samples according to the sample image and the second camera posture information comprises: The light emitted by the pixel point is determined according to the second camera posture information corresponding to the sample image where the pixel point is located and the position of the pixel point in the sample image. The new viewpoint image synthesis method according to claim 1, wherein the performing new viewpoint rendering through the trained neural radiation field to obtain the new viewpoint image comprises: According to the viewpoint information to be rendered, obtaining the new viewpoint image through the trained neural radiation field; The viewpoint information to be rendered includes the camera posture of the viewpoint to be rendered and the preview image resolution, and the resolution of the new viewpoint image is not less than the preview image resolution. The new viewpoint image synthesis method according to claim 1, wherein the performing new viewpoint rendering through the trained neural radiation field to obtain the new viewpoint image comprises: Generate a three-dimensional model according to the trained neural radiation field, wherein the three-dimensional model includes a three-dimensional mesh and a texture map; The three-dimensional model is rendered by a three-dimensional rendering method to obtain the new viewpoint image. A new viewpoint image synthesis system, comprising: Client and server; The client is configured to obtain a sample image and first camera posture information corresponding to the sample image; The server is configured to optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image; The server is also configured to train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field. The new viewpoint image synthesis system according to claim 9, wherein: The client is also configured to obtain viewpoint information to be rendered and send it to the server; The server is also configured to perform rendering according to the viewpoint information to be rendered using the trained neural radiation field to obtain a new viewpoint image, and send the new viewpoint image to the client. The new viewpoint image synthesis system according to claim 9 or 10, wherein: The server is further configured to generate a three-dimensional model according to the trained neural radiation field, and send the three-dimensional model to the client, wherein the three-dimensional model includes a three-dimensional mesh and a texture map; The client is also configured to render the three-dimensional model by a three-dimensional rendering method to obtain a new viewpoint image. The new viewpoint image synthesis system according to claim 9, wherein: The client is further configured to obtain information of a plurality of candidate viewpoints to be rendered; and send the information of the plurality of candidate viewpoints to be rendered and the preview image resolution to the server; The server is also configured to render multiple candidate new viewpoint images based on multiple candidate viewpoint information to be rendered and preview image resolutions through the trained neural radiation field, and send the multiple candidate new viewpoint images to the client. The new viewpoint image synthesis system according to claim 12, wherein: The client is further configured to, in response to a user's selection instruction, determine a viewpoint information to be rendered from the multiple candidate new viewpoint images; and send the viewpoint information to be rendered and the target image resolution to the server; The server is further configured to render the trained neural radiation field according to the viewpoint information to be rendered and the target image resolution to obtain a new viewpoint image, and send the new viewpoint image to the client; The target image resolution is greater than or equal to the preview image resolution. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the new viewpoint image synthesis method as described in any one of claims 1 to 8. A computer storage medium having a computer program stored therein, wherein the computer program is configured to execute the new viewpoint image synthesis method according to any one of claims 1 to 8 when running.