WO2025060587A1 - Video jitter removal method, electronic device, system, and storage medium - Google Patents

Abstract

Provided are a video jitter removal method, an electronic device, a system, and a storage medium. The method comprises: obtaining a video to be processed and camera attitude information corresponding to the video to be processed (110); on the basis of the video to be processed and the camera attitude information, training an initial neural radiance field, and obtaining a trained neural radiance field (120); on the basis of the video to be processed and the camera attitude information, generating a smooth camera path (130); and on the basis of the smooth camera path, rendering a video scene of the video to be processed by using the trained neural radiance field, and generating a new video after jitter removal (140). According to the solution provided by the present application, neural radiance field technology is utilized and rendering is carried out on the basis of the generated smooth camera path, achieving the effect of jitter removal by means of processing of the video to be processed.

Description

Video jitter removal method, electronic device, system and storage medium

This application claims priority to the Chinese patent application filed on September 20, 2023, with application number 202311221184X, and invention name “A video jitter removal method, electronic device, system and storage medium”, the content of which should be understood as incorporated into this application by reference.

Technical Field

This article relates to but is not limited to the field of video processing technology.

Background Art

In daily life, shooting videos has become a habitual behavior for people to record information and share their beautiful lives. However, when ordinary users use handheld devices to shoot videos, jitter problems often occur, especially when moving. Therefore, in the field of video processing, the demand for jitter removal technology is becoming increasingly widespread and the requirements are becoming higher and higher.

Summary of the invention

The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

The embodiments of the present application provide a video jitter removal method, electronic device, system and storage medium, which utilize neural radiation field technology to perform rendering based on a generated smooth camera path to achieve processing of the video to be processed to achieve the effect of removing jitter.

The present application provides a method for removing video jitter, including:

Obtaining a video to be processed and camera posture information corresponding to the video to be processed;

Training the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field;

Generate a smooth camera path according to the video to be processed and the camera posture information;

According to the smoothed camera path, the video scene of the video to be processed is rendered using the trained neural radiance field to generate a new video with the jitter removed.

The present application also provides an electronic device, including one or more processors; a storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the video jitter removal method as described in any embodiment of the present application.

The embodiment of the present application also provides a video jitter removal system, including a client and a server;

The client is configured to obtain a video to be processed and camera posture information corresponding to the video to be processed, and send the video to be processed and the camera posture information to the server;

The server is configured to train the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field;

The server is further configured to generate a smooth camera path according to the video to be processed and the camera posture information;

The server is also configured to render the video scene of the video to be processed using the trained neural radiation field according to the smooth camera path to generate a new video with the jitter removed.

An embodiment of the present application further provides a computer storage medium, in which a computer program is stored, wherein the computer program is configured to execute the video jitter removal method described in any embodiment of the present application when running.

The video jitter removal system architecture proposed in the embodiment of the present application adopts a system architecture supported remotely by a server. The server is responsible for computing tasks that require high computing resources, such as neural radiation field training, camera posture correction, and rendering to generate new videos. This significantly alleviates the computing pressure on the client and ensures the practicality of the video jitter removal solution.

Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

FIG1 is a flow chart of a method for removing video jitter provided by an embodiment of the present application;

FIG2 is a schematic diagram of the structure of a video jitter removal system provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of another video jitter removal system provided in an embodiment of the present application;

FIG4 is a flow chart of another video jitter removal method provided in an embodiment of the present application;

FIG. 5 is a flow chart of another video jitter removal method provided in an embodiment of the present application.

Details

In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

It is understood that the terms "first" and "second" used in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

When used herein, the singular forms "a", "an", and "said/the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include/comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, in this specification The term "and/or" used in the present invention includes any and all combinations of the associated listed items.

The embodiment of the present application provides an implementable solution, which uses the IMU (Inertial Measurement Unit) data sent back by the shooting device to calculate the translation and rotation, and then distorts the video frame by frame according to the translation and rotation to remove the video jitter. The resulting video obtained by this type of method will lose a certain amount of frame.

The embodiment of the present application provides a video jitter removal method, as shown in FIG1 , comprising:

Step 110, obtaining a video to be processed and camera posture information corresponding to the video to be processed;

Step 120, training the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field;

Step 130, generating a smooth camera path according to the video to be processed and the camera posture information;

Step 140, according to the smoothed camera path, the video scene of the video to be processed is rendered using the trained neural radiation field to generate a new video with the jitter removed.

Among them, Neural Radiance Field (NERF) is an emerging new viewpoint synthesis technology. It implicitly models the input video through a multi-layer perceptron, so that realistic images can be rendered from a new viewpoint. This solution uses a fully connected network to represent a three-dimensional scene, inputs three-dimensional spatial position and viewing angle information, outputs the volume density at the spatial position and the color information related to the viewing angle, and further combines the volume rendering technology to render the output color information and volume density onto the 2D image, thereby realizing new view synthesis and obtaining a new viewpoint image.

It should be noted that the camera path refers to the camera position and motion trajectory used to generate each frame of a given video sequence in a three-dimensional scene. In computer graphics, a video can be viewed as a sequence of continuous image frames. In order to generate these image frames, it is necessary to determine the position, direction, and movement of the camera in the scene. The camera path corresponding to the video describes the change process of the camera in time, that is, how the camera moves from one position and posture to the next position and posture and captures each image frame. The camera path corresponding to the video can be obtained by a variety of methods, including manually setting camera parameters, using a motion capture system to record real camera motion, and creating a virtual camera path through mathematical models and interpolation calculations. By defining and controlling the camera path corresponding to the video, it is possible to achieve perspective transformation, object tracking, simulation of photographic effects, etc., and ultimately generate a video sequence with coherent motion and visual sense.

In some exemplary embodiments, the camera path can be represented by a rotation that describes the position and orientation of the camera in three-dimensional space. Rotations are usually represented using Euler angles (such as pitch, yaw, and roll) or quaternions, and translations or displacements are represented using three-dimensional vectors.

In some exemplary embodiments, the camera path can be described by a series of discrete path points, each of which contains the position and posture information of the camera. These path points can be set manually or generated by other methods, such as real camera motion data recorded by a motion capture system or virtual path points calculated based on an interpolation algorithm.

In some exemplary embodiments, the camera path can be described by curve parameterization. Common curve types include Bezier Curve, Spline Curve, etc. Curve parameterization describes the change of the camera over time, and determines the position and direction of the camera at different time points according to the curve shape and control points.

It can be understood that in the embodiment of the present application, the new video generated by rendering based on the smooth camera path is a video with more coherent pictures and smoother changes compared to the initial video obtained from the shooting device. Therefore, it is also called a video with removed jitter.

In some exemplary embodiments, the step of obtaining a video to be processed and camera posture information corresponding to the video to be processed includes:

The camera posture information corresponding to each frame image in the video to be processed is obtained through a simultaneous localization and mapping (SLAM) algorithm.

The frame image is also called an image frame, or frame for short.

In some exemplary embodiments, the step of obtaining a video to be processed and camera posture information corresponding to the video to be processed includes:

Obtaining initial camera posture information corresponding to each frame of the video to be processed through the SLAM algorithm;

The initial camera posture information is optimized by using the Structure from Motion (SFM) algorithm to obtain the camera posture information corresponding to each frame image in the video to be processed.

It can be understood that the camera posture information obtained after SFM algorithm optimization is more accurate than the initial camera posture information.

In some exemplary embodiments, generating a smooth camera path according to the video to be processed and the camera posture information includes:

Acquire multiple key frames in the video to be processed;

Interpolation processing is performed on the camera posture information corresponding to the multiple key frames to obtain a smooth camera path corresponding to the video to be processed.

In some exemplary embodiments, the step of obtaining a plurality of key frames in the video to be processed includes:

According to a preset frame interval, the plurality of key frames are determined from all frame images included in the video to be processed.

For example, if the preset frame interval is 5, a key frame is determined every 5 frames from the video to be processed, which is flexibly set according to the system processing performance needs and is not limited to a specific aspect. Alternatively, it is dynamically determined according to the motion information of the shooter. For example, in the case of intense exercise, the frame interval is small, and in the case of gentle action, the frame interval is large. More examples are not listed here one by one.

In some exemplary embodiments, the step of obtaining a plurality of key frames in the video to be processed includes: selecting a frame image satisfying a first condition as a key frame in the video to be processed, wherein the first condition is that a change in attribute information of the frame image on the original camera path compared to attribute information of a previous frame image of the frame image on the original camera path is greater than a first threshold;

The original camera path is a camera path determined according to the camera posture information corresponding to the video to be processed.

In some exemplary embodiments, the acquiring a plurality of key frames in the video to be processed includes: acquiring a plurality of key frames selected by a user from the video to be processed.

In some exemplary embodiments, a user interaction instruction is received through a user interaction module to determine the selected multiple key frames.

In some exemplary embodiments, the attribute information of the frame image on the camera path includes one or more of the following:

Location information, orientation information.

In some exemplary embodiments, the orientation information is simply referred to as orientation, also called rotation information, or direction information.

In some exemplary embodiments, when the attribute information includes position information and orientation information, the attribute information is also referred to as camera attitude information, or attitude information for short.

In some exemplary embodiments, the position information is coordinates; in some exemplary embodiments, the coordinates are three-dimensional coordinates.

According to different attribute information, a corresponding first threshold is set. For example, when the attribute information includes position information, the first threshold is a distance threshold, and when the change range of the position information of the frame image on the original camera path compared with the position information of the previous frame image on the original camera path is greater than the distance threshold, In this case, the frame image is determined as a key frame; that is, the distance between the position of the frame image on the original camera path and the position of the previous frame image of the frame image on the original camera path is greater than the distance threshold.

For another example, when the attribute information includes orientation information, the first threshold is the direction angle difference threshold. When the change amplitude of the orientation information of the frame image on the original camera path compared to the orientation information of the previous frame image on the original camera path is greater than the direction angle threshold, the frame image is determined as a key frame; that is, the angle difference between the orientation of the frame image on the original camera path and the orientation of the previous frame image on the original camera path is greater than the distance direction angle difference threshold.

For another example, when the attribute information includes position information and orientation information, the first threshold also correspondingly includes: a distance threshold and a direction angle difference threshold. When the change amplitude of the orientation information of the frame image on the original camera path compared to the orientation information of the previous frame image on the original camera path of the frame image is greater than the direction angle threshold, or when the change amplitude of the position information of the frame image on the original camera path compared to the position information of the previous frame image on the original camera path of the frame image is greater than the distance threshold, the frame image is determined to be a key frame.

Alternatively, when the change amplitude of the orientation information of the frame image on the original camera path compared to the orientation information of the previous frame image on the original camera path is greater than the direction angle threshold, and the change amplitude of the position information of the frame image on the original camera path compared to the position information of the previous frame image on the original camera path is greater than the distance threshold, the frame image is determined to be a key frame. More examples are not listed here one by one.

It is understandable that due to the presence of jitter during shooting, the original camera path corresponding to the video to be processed may not be smooth, and there may be large jumps in spatial position, orientation or posture. In some exemplary implementations, frame images at path points with large changes in attribute information are selected to form the multiple key frames, and then interpolation processing is performed based on these selected key frames to obtain a smooth camera path corresponding to the video to be processed.

In some exemplary embodiments, the duration corresponding to the smoothed camera path obtained after interpolation processing based on multiple key frames is the same as the duration of the video to be processed.

In some exemplary embodiments, the duration corresponding to the smoothed camera path obtained after interpolation processing based on multiple key frames is the same as the duration of the video to be processed. It can be understood that the new video rendered based on the smoothed camera with the same duration has the same duration as the video to be processed obtained in step 110.

In some exemplary embodiments, the position information is interpolated using a linear interpolation method, and interpolation processing is performed based on the position information corresponding to a plurality of key frames to obtain the position information of the insertion point.

In some exemplary embodiments, the orientation information is interpolated using a spherical linear interpolation method, and interpolation processing is performed based on the orientation information corresponding to a plurality of key frames to obtain the orientation information of the insertion point.

In some exemplary embodiments, the duration of the smoothed camera path obtained after interpolation processing based on multiple key frames is shorter than the duration of the video to be processed. It can be understood that the duration of the new video rendered based on the smoothed camera with a shorter duration is shorter than the duration of the video to be processed obtained in step 110.

In some exemplary embodiments, generating a smooth camera path according to the video to be processed and the camera posture information includes:

For each frame image in the video to be processed, according to a preset number of iterations, using multiple frames of images adjacent to the frame image, correct the camera posture information corresponding to the frame image;

A smooth camera path corresponding to the video to be processed is generated according to the corrected camera posture information corresponding to each frame image in the video to be processed.

In some exemplary embodiments, the camera posture information includes: coordinates and orientation.

In some exemplary embodiments, the coordinates include three-dimensional coordinates.

In some exemplary embodiments, the orientation comprises a quaternion.

In some exemplary embodiments, correcting the posture information includes: correcting coordinates and correcting orientation.

In some exemplary embodiments, for each frame image in the video to be processed, correcting the camera posture information corresponding to the frame image by using multiple frames of images adjacent to the frame image according to a preset number of iterations includes:

For each frame of the video to be processed, the camera posture information corresponding to the frame is used as the initial camera posture information to be corrected. According to the preset number of iterations, in each round of iteration, the following steps are performed respectively:

Acquire image parameters of the adjacent multi-frame images, and correct the to-be-corrected camera posture information of the frame image according to the image parameters of the adjacent multi-frame images, wherein the image parameters include coordinates, orientations, and weights;

The corrected camera posture information is used as the camera posture information to be corrected in the next round of iteration.

That is, multiple iterative corrections are performed on the camera posture information of each frame image in the video to be processed, and the total number of repetitions is the preset number of iterations N. The camera posture information corresponding to the frame image in the video to be processed is used as the initial camera posture information to be corrected, which is corrected in the first iteration, and the corrected camera posture information is used as the camera posture information to be corrected in the next iteration, until N corrections are completed to obtain the final corrected camera posture information.

In some exemplary embodiments, the preset number of iterations N=3, that is, each initial camera posture information to be corrected is iterated 3 times to obtain the final correction result. The specific value of the number of iterations can be flexibly set as needed and is not limited to the aspects of the embodiments of the present application.

In some exemplary embodiments, for each frame image in the video to be processed, correcting the camera posture information corresponding to the frame image by using multiple frames of images adjacent to the frame image according to a preset number of iterations includes:

Repeat the following method for calibration according to the preset number of iterations:

according to Determine the corrected coordinate information;

according to Determine the corrected orientation information;

Wherein, p is the three-dimensional coordinate to be corrected, d is the orientation quaternion to be corrected; n is the number of multiple frames adjacent to the frame image to be corrected; p _i is the three-dimensional coordinate of the i-th frame image, {p _i |i＝1…n}, d _i is the orientation quaternion of the i-th frame image, {d _i |i＝1…n}; _wi is the weight of the i-th frame image, {w _i |i＝1…n}; λ is the set correction degree coefficient, λ∈[0,1]; is the Frobenius norm, A(d) is the orthogonal attitude matrix of the quaternion d; p ^* is the corrected three-dimensional coordinate, and d ^* is the corrected orientation quaternion.

Wherein, in the first iteration, the three-dimensional coordinates to be corrected are the three-dimensional coordinates corresponding to the frame image to be corrected, and the orientation quaternion to be corrected is the quaternion corresponding to the frame image to be corrected;

In a non-first iteration, the three-dimensional coordinates to be corrected are the corrected three-dimensional coordinates obtained in the previous iteration, and the orientation quaternion to be corrected is the corrected quaternion obtained in the previous iteration.

In some exemplary embodiments, the weight _wi is determined according to the following method:

It should be noted that the multiple frames of images adjacent to the frame image in the above correction method are n frames of images adjacent to the frame image, where n is an integer greater than or equal to 1 and is flexibly set according to needs and is not limited to a specific aspect.

In some exemplary embodiments, the training of the initial neural radiation field according to the video to be processed and the camera posture information includes:

A plurality of training samples are obtained according to the video to be processed and the camera posture information, wherein each training sample is composed of The light emitted by the pixel points of the frame image in the video to be processed and the corresponding color composition;

The initial neural radiation field is trained according to the multiple training samples.

In some exemplary embodiments, the light emitted by a pixel point of a frame image in the video to be processed is determined based on the camera posture information corresponding to the frame image where the pixel point is located and the position of the pixel point in the frame image to which the pixel point belongs. That is, the light emitted by the pixel point is determined based on the camera posture information corresponding to the frame image where the pixel point is located and the position of the pixel point in the image.

In some exemplary embodiments, the step of obtaining a plurality of training samples according to the video to be processed and the camera posture information includes:

Determine a plurality of data groups according to the video to be processed and the camera posture information, each data group including a frame image and corresponding camera posture information;

For each data set including frame images and camera posture information, perform the following steps to obtain multiple training samples:

Analyze the frame image and camera posture information into the light emitted by each pixel;

The light emitted by each pixel and the color of the pixel form a sample.

In some exemplary embodiments, the color of each pixel point p of a frame image is c. Combining the camera posture information and pixel position corresponding to the frame image, the light emitted by the pixel point can be obtained and recorded as l(p, d), where p = (x, y, z) is the position coordinate of the pixel point in the three-dimensional Cartesian space coordinate system, and d = (θ, φ) is the solid angle parameter of the light direction in the spherical coordinate system, where θ represents the polar angle or latitude, which is the angle between the vector from the reference axis (usually the positive z axis) to the point and the reference axis, and the value range of θ is usually 0 to π. φ represents the azimuth or longitude, which is the angle between the projection from a certain reference direction on the reference plane (usually the positive x axis) to the point and the reference direction, and the value range of φ is usually 0 to 2π.

It can be understood that a frame image in the video to be processed includes multiple pixels, corresponding to multiple training samples composed of light and color emitted by the pixels, that is, one pixel corresponds to one training sample, and multiple image frames in the video to be processed obtain more training samples.

In some exemplary embodiments, training the initial neural radiation field according to the plurality of training samples comprises:

All training samples are randomly shuffled and then the initial neural radiation field is trained.

For each input sample (l, c), l is light and c is color. In the divided space, the coordinate p is spatially deformed and its position is queried through hash coding. The multi-layer perceptron of the node is used to encode it. The encoded features f and d are encoded together using a global multi-layer perceptron to output the color c _pred and the difference disp.

Among them, the loss function

In some exemplary embodiments, the spatial division in the neural radiation field may be a uniform grid-based spatial division or an octree-based spatial division.

In some exemplary embodiments, the spatial deformation in the neural radiation field may be a spatial deformation based on normalized device coordinates, or a spatial deformation based on a perspective projection coordinate system.

It can be understood that the video de-jittering solution provided in the embodiment of the present application obtains training samples from the video to be processed to train the neural radiation field, utilizes the new viewpoint synthesis technology based on the neural radiation field, and then renders and generates a new video with de-jittering corresponding to the video to be processed based on the smoothed camera path obtained after smoothing the camera path of the video to be processed. This can effectively remove video jitter and avoid the frame loss caused by some de-jittering solutions.

The present application also provides an electronic device, including:

one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the video jitter removal method as described in any embodiment of the present application.

The embodiment of the present application also provides a video jitter removal system, as shown in FIG2 , comprising:

Client 210 and server 220;

The client 210 is configured to obtain a video to be processed and camera posture information corresponding to the video to be processed, and send the video to be processed and the camera posture information to the server 220;

The server 220 is configured to train the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field;

The server 220 is further configured to generate a smooth camera path according to the video to be processed and the camera posture information;

The server 220 is further configured to render the video scene of the video to be processed using the trained neural radiation field according to the smoothed camera path, so as to generate a new video with the jitter removed.

In some exemplary embodiments, the server 220 is further configured to send the new video after the jitter is removed to the client 210;

Correspondingly, the client 210 is also configured to display the new video after the jitter is removed.

In some exemplary embodiments, the client 210 is further configured to obtain a plurality of key frames in the video to be processed, and send the plurality of key frames to the server 220 .

In some exemplary embodiments, the client 210 includes: a user interaction module, configured to receive a user operation instruction and determine the selected multiple key frames from the video to be processed.

In some exemplary embodiments, the server 220 is further configured to perform interpolation processing on camera posture information corresponding to multiple key frames in the video to be processed, so as to obtain a smooth camera path corresponding to the video to be processed.

In some exemplary embodiments, the server 220 or the client 210 selects the multiple key frames from the video to be processed, and the frame images that meet the first condition are used as key frames;

Among them, the first condition is that the change amplitude of the attribute information of the frame image on the original camera path compared with the attribute information of the previous frame image on the original camera path of the frame image is greater than a first threshold; the original camera path is a camera path determined according to the camera posture information corresponding to the video to be processed.

It can be seen that the key frames used to determine the smooth camera path can be determined by the server 220 according to the video to be processed, or by the client 210 according to the video to be processed, or by the client 210 according to the user's operation instruction. For example, during video shooting or video playback, the user uses the user interaction module to mark to determine multiple key frames.

The embodiment of the present application also provides a video jitter removal system, as shown in FIG3 , comprising:

Client 210 and server 220;

The client 210 includes: an image acquisition module 2110, a posture information acquisition module 2120, and a user interaction module 2130;

The server 220 includes: a neural radiation field training module 2210, a smooth path determination module 2220, a neural radiation field training module 2211, a smooth path determination module 2221, a neural radiation field training module 2212, a neural radiation field training module 2213, a neural radiation field training module 2214, a neural radiation field training module 2215, a neural radiation field training module 2216, a neural radiation field training module 2217, a Shooting field rendering module 2230.

The image acquisition module 2110 is configured to acquire a video to be processed; it can be understood that the video to be processed includes a plurality of frame images.

The posture information acquisition module 2120 is configured to acquire the camera posture information corresponding to the video to be processed; accordingly, the corresponding camera posture information includes the camera posture information corresponding to each frame of image.

The user interaction module 2130 is configured to receive a user operation instruction and determine the multiple key frames selected from the video to be processed.

The neural radiation field training module 2210 is configured to train the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field.

The smooth path determination module 2220 is configured to generate a smooth camera path according to the video to be processed and the camera posture information.

The neural radiation field rendering module 2230 is configured to render the video to be processed using the trained neural radiation field according to the smoothed camera path to obtain a rendering result.

In some exemplary embodiments, the posture information acquisition module 2120 is configured to acquire the camera posture information corresponding to each frame image in the video to be processed through a SLAM algorithm.

In some exemplary embodiments, the posture information acquisition module 2120 is configured to obtain the initial camera posture information corresponding to each frame image in the video to be processed through the SLAM algorithm; optimize the initial camera posture information through the SFM algorithm to obtain the camera posture information corresponding to each frame image in the video to be processed.

In some exemplary embodiments, the client 210 further includes: a first data transceiver module, configured to send the video to be processed and camera posture information corresponding to the video to be processed to the server 220 .

In some exemplary embodiments, the first data transceiver module is further configured to send the multiple key frames selected by the user to the server 220 .

In some exemplary embodiments, the first data transceiver module is further configured to receive a rendering result from the server 220 .

In some exemplary embodiments, the user interaction module 2130 is configured to generate a new video after de-jittering according to the rendering result.

In some exemplary embodiments, the server 220 further includes: a second data transceiver module, configured to receive the video to be processed and the camera posture information corresponding to the video to be processed from the client 210.

In some exemplary embodiments, the second data transceiver module is further configured to receive the selected key frame from the client 210 .

In some exemplary embodiments, the second data transceiver module is further configured to send the rendering result to the client 210 .

In some exemplary embodiments, the smooth path determination module 2220 is configured to obtain a plurality of key frames in the video to be processed; and perform interpolation processing on the camera posture information corresponding to the plurality of key frames to obtain a smooth camera path corresponding to the video to be processed.

In some exemplary embodiments, the smooth path determination module 2220 is configured to correct the camera posture information corresponding to each frame image in the video to be processed according to a preset number of iterations using multiple frame images adjacent to the frame image; and generate a smooth camera path corresponding to the video to be processed based on the corrected camera posture information corresponding to each frame image in the video to be processed.

The embodiment of the present application also provides a video jitter removal method, as shown in FIG4 , comprising:

Step 410, the image acquisition module acquires the video to be processed;

Step 420, the posture information acquisition module acquires the camera posture information corresponding to the video to be processed through the SLAM algorithm;

Step 430: The first data transceiver module sends the video to be processed and the camera posture information corresponding to the video to be processed to the server;

Step 440, the neural radiation field training module trains the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field;

Step 450: the user interaction module obtains a plurality of key frames and sends them to the server through the first data transceiver module;

Step 460: the smooth path determination module performs interpolation processing according to the multiple key frames to obtain a smooth camera path;

Step 470: The neural radiation field rendering module renders the video scene of the video to be processed using the trained neural radiation field according to the smoothed camera path to generate a new video after jitter removal;

Step 480, sending the new video to the client through the second transceiver module;

Step 490: The user interaction module displays the new video.

In some exemplary embodiments, steps 450-460 are replaced by steps 451-461, as shown in FIG5 :

Step 451, the smooth path determination module corrects the camera posture information corresponding to each frame image in the video to be processed by using multiple frames of images adjacent to the frame image according to a preset number of iterations;

Step 461: The smooth path determination module generates a smooth camera path corresponding to the video to be processed according to the corrected camera posture information corresponding to each frame image in the video to be processed.

It can be seen that the system architecture proposed in the disclosed embodiment is that the client executes the basic steps of video acquisition and camera posture information acquisition, and the server executes the neural radiation field training, camera posture information correction and rendering to generate new videos and other steps that require high computing resources. Fully considering the implementation characteristics of the neural radiation field solution, a distributed computing solution combining the server and the client is adopted to improve the feasibility and practicality of the solution. The powerful computing power of the server is utilized to avoid the insufficient computing power that may be faced by relying solely on the local execution of the solution by the video capture device, which affects the final jitter removal effect.

An embodiment of the present application further provides a computer storage medium, in which a computer program is stored, wherein the computer program is configured to execute the video jitter removal method as described in any embodiment of the present application when running.

The video jitter removal solution provided in the embodiments of the present application adopts a new neural radiation field solution, and implicitly models the input video through a multi-layer perceptron, so that a realistic picture can be rendered from a new viewpoint, overcoming the frame loss caused by some jitter removal solutions, and achieving a good jitter removal effect. In some exemplary embodiments, the solution that supports users to select key frames and then determine a smooth camera path can fully meet the user's viewpoint setting needs. In some exemplary embodiments, an automatic correction method is used to correct the camera posture of the video to be processed, and then a smooth camera path is obtained, which significantly improves the jitter removal effect. In some exemplary embodiments, a server remote support system architecture is adopted, and the server undertakes computing tasks that require high computing resources such as neural radiation field training, camera posture correction, and rendering to generate new videos, which significantly alleviates the computing pressure of the client and ensures the practicality of the solution of the present application.

Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules/units in the above disclosed methods may be implemented as software, firmware, hardware, or a suitable combination thereof. In hardware implementations, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Components or all components can be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media generally contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

Claims (15)

A video jitter removal method, comprising: Obtaining a video to be processed and camera posture information corresponding to the video to be processed; Training the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field; Generate a smooth camera path according to the video to be processed and the camera posture information; According to the smoothed camera path, the video scene of the video to be processed is rendered using the trained neural radiance field to generate a new video with the jitter removed. The video jitter removal method according to claim 1, wherein the step of obtaining the video to be processed and the camera posture information corresponding to the video to be processed comprises: The camera posture information corresponding to each frame image in the video to be processed is obtained by synchronously positioning and mapping the SLAM algorithm. The video jitter removal method according to claim 1, wherein the step of obtaining the video to be processed and the camera posture information corresponding to the video to be processed comprises: Acquire the initial camera posture information corresponding to each frame image in the video to be processed by synchronous positioning and mapping SLAM algorithm; The initial camera posture information is optimized by using the structure from motion (SFM) algorithm to obtain the camera posture information corresponding to each frame of the video to be processed. The video jitter removal method according to claim 1, wherein generating a smooth camera path according to the video to be processed and the camera posture information comprises: Acquire multiple key frames in the video to be processed; Interpolation processing is performed on the camera posture information corresponding to the multiple key frames to obtain a smooth camera path corresponding to the video to be processed. The video jitter removal method according to claim 4, wherein the obtaining of a plurality of key frames in the video to be processed comprises: Selecting, from the video to be processed, a frame image that meets a first condition as a key frame, wherein the first condition is that a change in attribute information of the frame image on the original camera path compared to attribute information of a previous frame image of the frame image on the original camera path is greater than a first threshold; The original camera path is a camera path determined according to the camera posture information corresponding to the video to be processed. The video jitter removal method according to claim 1, wherein generating a smooth camera path according to the video to be processed and the camera posture information comprises: For each frame image in the video to be processed, according to a preset number of iterations, using multiple frames of images adjacent to the frame image, correct the camera posture information corresponding to the frame image; A smooth camera path corresponding to the video to be processed is generated according to the corrected camera posture information corresponding to each frame image in the video to be processed. The video jitter removal method according to claim 6, wherein the step of correcting the camera posture information corresponding to each frame image in the video to be processed by using multiple frames of images adjacent to the frame image according to a preset number of iterations comprises: For each frame of the video to be processed, the camera posture information corresponding to the frame is used as the initial camera posture information to be corrected. According to the preset number of iterations, in each round of iteration, the following steps are performed respectively: Acquire image parameters of the adjacent multi-frame images, and correct the to-be-corrected camera posture information of the frame image according to the image parameters of the adjacent multi-frame images, wherein the image parameters include coordinates, orientations, and weights; The corrected camera posture information is used as the camera posture information to be corrected for the next round of iteration. The video jitter removal method according to any one of claims 1 to 7, wherein: The training of the initial neural radiation field according to the video to be processed and the camera posture information includes: Acquire a plurality of training samples according to the video to be processed and the camera posture information, wherein each training sample is composed of light emitted by a pixel point of a frame image in the video to be processed and a corresponding color; The initial neural radiation field is trained according to the multiple training samples. The video jitter removal method according to claim 8, wherein: The light emitted by the pixel point of the frame image in the video to be processed is determined according to the camera posture information corresponding to the frame image where the pixel point is located and the position of the pixel point in the belonging frame image. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video jitter removal method according to any one of claims 1 to 9. A video jitter removal system, comprising: Client and server; The client is configured to obtain a video to be processed and camera posture information corresponding to the video to be processed, and send the video to be processed and the camera posture information to the server; The server is configured to train the initial neural radiation field according to the video to be processed and the camera posture information to obtain a trained neural radiation field; The server is further configured to generate a smooth camera path according to the video to be processed and the camera posture information; The server is also configured to render the video scene of the video to be processed using the trained neural radiation field according to the smooth camera path to generate a new video with the jitter removed. The video jitter removal system according to claim 11, wherein the server is further configured to, based on multiple key frames in the video to be processed, perform interpolation processing on camera posture information corresponding to the multiple key frames to obtain a smooth camera path corresponding to the video to be processed. The video jitter removal system according to claim 11, wherein the client is further configured to obtain multiple key frames in the video to be processed and send the multiple key frames to the server. The video jitter removal system according to claim 13, wherein the client comprises: a user interaction module, configured to receive user operation instructions and determine the selected multiple key frames from the video to be processed. A computer storage medium having a computer program stored therein, wherein the computer program is configured to execute the video jitter removal method according to any one of claims 1 to 9 when running.