CN116503536A - A Light Field Rendering Method Based on Scene Layering - Google Patents

Abstract

The invention relates to a light field rendering method based on scene layering to solve the problem of waste of computing power caused by high-resolution rendering of a large depth part of a 3D scene in the prior art. The present invention includes: using a model rendering tool to conduct experiments to obtain the change curve of the perceived resolution with the depth of the light field display in the three-dimensional light field display; according to the change curve of the perceived resolution with the display depth of the light field, a depth threshold is selected, and the scene is divided into Different levels; render 3D scenes at different levels; determine the occlusion relationship between different levels to obtain low-resolution parallax images; perform super-resolution reconstruction and pixel encoding on parallax images to obtain high-resolution composite images. Through simulation experiments, the relationship between the display depth of the light field and the display resolution perceived by the human eye under ideal conditions is obtained, and the display resolution of the synthesized image is improved by combining the deep learning super-resolution method, so as to achieve high-quality display with low computing power loss Effect.

Description

A Light Field Rendering Method Based on Scene Layering

technical field

The invention belongs to the field of three-dimensional display, and in particular relates to a light field rendering method based on scene layering.

Background technique

Three-dimensional light field display technology produces stereoscopic vision through physiological factors or psychological factors. Common 3D display methods, such as grating 3D display, integrated imaging 3D display, volumetric 3D display and other 3D display technologies, use physiological and psychological factors such as binocular parallax to display parallax images on the 2D display screen, so that the audience can feel the screen. The depth information of the object presents a three-dimensional effect. The clarity of a 3D image is not only related to the display resolution of the light field display system, the distance from the lens array to the LCD, and the distance from the diffusion film to the lens array, but also to the display depth of the light field display system. For display content with a small screen-to-screen distance, the higher the rendering resolution and display resolution of the image, the higher the definition of the 3D image, and the better the viewing effect of the human eye; for the display content with a large screen-to-screen distance, if the resolution of the image If the rate is too high, it will cause ghosting and blurring due to large volume pixels, which will reduce the display clarity of the image, and it will also be an unnecessary waste of computing power.

The invention patent document with the publication number CN110956689A and the publication date of 2020-04-03 discloses a three-dimensional suspended light field display system and display method, refer to paragraphs [0048] to [0089], in an embodiment of the present invention A three-dimensional suspended light field display system is provided. FIG. 1 is a schematic structural diagram of the three-dimensional suspended light field display system provided by an embodiment of the present invention. The display system includes a web terminal, a cloud, and a dual-screen floating display terminal. Database... The embodiment of the present invention provides a three-dimensional floating light field display system and display method, and provides a set of management system for realizing three-dimensional floating light field display, which can realize orderly management of multi-user floating display and ensure three-dimensional floating of products displayed effect.

Although it can achieve the effect of three-dimensional floating display of the product, for the display content with a large distance between the entrance and exit screen, if the resolution of the image is too high, it will cause ghosting and blurring due to large volume pixels, which will reduce the clarity of the image display At the same time, the problem of unnecessary waste of computing power cannot be effectively solved.

Contents of the invention

The purpose of the present invention is to provide a light field rendering method based on scene layering, so as to solve the problem of waste of computing power caused by high-resolution rendering of a large depth part of a 3D scene in the prior art.

In order to achieve the above purpose, the present invention provides a specific technical solution of a light field rendering method based on scene layering as follows:

Step 1: Experimentally obtain the variation curve of the perceived resolution with the depth of the light field display in the 3D light field display.

The display depth of the light field, the display depth of the grating three-dimensional display refers to the screen-out and screen-entry range of the grating three-dimensional display displaying 3D images; spacing.

The experiments were carried out using model rendering tools. First place the 3D scene, set the ambient light, then determine the number and position of the virtual camera array, and set the resolution of the output disparity map; in order to obtain the ideal rendering effect, use the ray tracing algorithm to render the 3D scene; by adjusting the zero plane of the scene Position, so that the virtual camera array collects images of 3D models on different depth planes, and uses computational imaging technology to reproduce 3D images; measure the image resolution seen by human eyes, so as to obtain the change curve of perceived resolution and display depth.

Step 2: Select the zero plane of the 3D scene, and according to the obtained resolution change curve, select multiple out-screen and in-screen depths as thresholds to layer the scene, and obtain multiple scene levels;

The zero plane means that in the light field display imaging system, the light convergence plane is the plane with the best imaging definition, and the reconstructed three-dimensional scene presented is the clearest. This plane is called the zero plane, or the zero parallax plane.

For the 3D scene reproduced in the simulation experiment, the depth plane closer to the zero plane corresponds to a smaller scene voxel size, and the depth plane farther from the zero plane corresponds to a larger scene voxel size, so the image resolution observed by the human eye varies with the scene The depth from the zero plane varies: the farther the scene is from the zero plane, the lower the resolution. According to the change curve of the resolution, multiple screen-out and screen-in depths are selected as thresholds to implement layering of the scene, and multiple scene levels are obtained.

Step 3: Render 3D scenes of different levels;

The rendering refers to the process of two-dimensionally projecting the model in the three-dimensional scene into a digital image according to the set environment, lighting, material and rendering parameters.

Optionally, the rendering technology includes algorithms such as rasterization, ray tracing, and path tracing. Preferably, the ray tracing algorithm can more realistically simulate light effects, achieve correct rendering of reflections, transparency phenomena, shadows, and global illumination, and achieve good visual effects.

By adjusting the number of viewpoints at different levels and the rendering resolution parameters, high-definition and high-resolution rendering is performed on 3D scene levels with high complexity, high information content, and close to zero plane, so as to retain more model details and achieve High-quality display effect; low-definition, low-resolution rendering of 3D scenes with low complexity, low information content, and far away from the zero plane, thereby reducing computing time and saving computing power.

Step 4: Determine the occlusion relationship between different levels to obtain a parallax image;

The occlusion relationship between different levels refers to that after the scene is layered, since different levels have different distances from the viewpoint and are rendered separately, objects located in different scenes may have a front-to-back occlusion relationship. In order to form a parallax image with a correct occlusion relationship, each level is sorted from far to near according to its distance from the viewpoint, and the priority of the level closer to the viewpoint is higher than that farther away from the viewpoint. When drawing a parallax image, the layer image with a higher priority covers the scene image with a lower priority.

The parallax image refers to a plurality of different images obtained by shooting the same 3D scene from different angles with a camera array according to the principle of binocular parallax.

Step 5: Perform super-resolution reconstruction and pixel encoding on the disparity image to obtain a high-resolution composite image.

The pixel encoding refers to the process of arranging specific sub-pixels in a parallax image sequence according to a certain rule to generate a composite image according to the display characteristics of a three-dimensional light field display. The synthesized image is displayed on the display panel of the 3D light field display. According to the principle of binocular parallax, the light control unit makes the light emitted by the sub-pixels form different viewing point display areas to realize the reproduction of stereoscopic images.

The super-resolution reconstruction is a process of restoring image details and other data information based on known low-resolution image information by using machine learning and related optical knowledge. The structure of the super-resolution neural network is a generative confrontation network, which consists of a generative network and a discriminative network. The generative confrontation network refers to a machine learning algorithm in which a generative network model and a discriminative network model compete with each other to optimize each other.

Wherein, the generation network is used to extract features from the input low-resolution synthetic image and perform super-resolution reconstruction, which consists of multiple deformable convolutional layers, multiple activation units, a sub-pixel convolutional layer and a residual connection composition.

The deformable convolution refers to an unconventional convolution method that introduces an offset in the traditional convolution receptive field, wherein the offset varies with the adaptability of the shape and size of the object, so that the receptive field changes with the object free deformation.

For the input feature map x, the value at P ₀ in the output feature map y after deformable convolution is:

,

Where R={(-1,-1),(-1,0),...,(0,1),(1,1)} defines a 3×3 convolution kernel, W( P _n ) is the weight at P _n in the receptive field, and ΔP _n is the offset.

Since the position after adding the offset is not an integer, use bilinear interpolation to get the offset position: X(P):

,

Among them, any position P=P ₀ +P _n +ΔP _n , q is all integer spatial positions in the feature map x, and G(q,p) is the kernel of the bilinear interpolation algorithm.

The deformable convolution better takes into account the change in the shape and size of the object, and makes up for the shortcomings of the traditional convolution with a fixed model in extracting spatial information.

The sub-pixel convolution refers to the process of obtaining a high-resolution feature map from a low-resolution feature map through convolution and multi-channel recombination. In the last layer of the generation network, sub-pixel convolution is used to reorganize the multi-channel feature map of the low-resolution image into a high-resolution image, avoiding the huge amount of calculation caused by interpolating the low-resolution image first and then performing feature extraction. It is a choice to save computing power and reduce network time complexity.

The input data set of the discriminant network is composed of real data (marked as 1) and the output result of the generated network (marked as 0), and the parameters of the discriminant network are optimized by minimizing the error between the predicted value of the discriminant network and the true value. The discriminative network consists of multiple convolutional layers, BN layers, activation units and fully connected layers, which are used to optimize the parameters of the generated network model.

A light field rendering method based on scene layering provided by the present invention has the following advantages:

This application uses the model rendering tool to conduct experiments to obtain the change curve of the perceived resolution with the depth of the light field display in the 3D light field display; according to the change curve of the perceived resolution with the display depth of the light field, the depth threshold is selected, and the scene is divided into Different levels; render 3D scenes at different levels; determine the occlusion relationship between different levels to obtain low-resolution parallax images; perform super-resolution reconstruction and pixel encoding on parallax images to obtain high-resolution composite images. At the same time, the technical solution provided by this application obtains the relationship between the display depth of the light field in an ideal state and the display resolution perceived by the human eye through simulation experiments, and then layers the 3D scene, and further classifies the scene levels of different display depths and different amounts of information. Use different viewpoints and resolutions for layered rendering, and combine deep learning super-resolution methods to improve the display resolution of composite images, so as to achieve high-quality display effects with low computing power consumption.

Description of drawings

Fig. 1 is the schematic flow chart that the present invention provides;

Fig. 2 is a schematic structural diagram of the generation network provided by the present invention;

Fig. 3 is a schematic flow chart of the super-resolution neural network provided by the present invention.

In the picture:

1. Experimentally obtain the variation curve of the perceived resolution with the depth of the light field display in the 3D light field display;

2. According to the change curve of the perception resolution and the display depth of the light field, the depth threshold is selected to divide the scene into different levels;

3. Render 3D scenes of different levels;

4. Determine the occlusion relationship between different levels to obtain low-resolution parallax images;

5. Perform super-resolution reconstruction and pixel coding on the parallax image to obtain a high-resolution composite image.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Referring to Fig. 1 to Fig. 3, the present invention provides a light field rendering method based on scene layering, including five steps:

Step 1: Carry out optical simulation experiments, and use the open source 3D graphic image software Blender to conduct simulation experiments. The steps to render and generate stereoscopic content using ray tracing algorithm in Blender are as follows:

a) Create a 3D scene. Adjust the position of objects in the scene and set the appropriate lighting for the scene.

b) Place the camera array, and determine the array position of the overall camera by placing the main camera first. Adjust the position and quantity of the camera array to achieve the best shooting effect of the scene.

c) Set the path and resolution of the output disparity map, and render the scene.

d) Encoding the disparity image into a composite image.

In order to study the change curve of perceived resolution and display depth, a fixed number of virtual camera arrays are used to capture 3D scenes with an off-axis placement structure on multiple different depth planes to obtain multi-view parallax images. By comparing with the two-dimensional image under the same field of view conditions, the resolution of the light field acquisition image viewed by the human eye is measured every 50 mm of the display depth change, and the change curve of the perceived resolution and the display depth is obtained.

Step 2: Select the zero plane of the 3D scene as the plane with the clearest image when reconstructing the 3D scene, set ten depth thresholds according to the change curve of perceived resolution and display depth, and divide the entire 3D scene into nine levels.

Step 3: Render 3D scenes of different levels;

From the change curve of the perceptual resolution with the display depth of the light field, the change function of the perceptual resolution with the display depth of the light field can be obtained: , /> ,

Among them, h is the display depth of the light field, f(z) is the perceived resolution value when the display depth is h, h _c is the abscissa of the center of the curve, h _c =0, y ₀ is the offset, A is the amplitude , w is the full width corresponding to h equal to the standard deviation ±σ. H ₃ and H ₄ are Hermitian polynomials, where H ₃ =z ³ -3z, H ₄ =z _⁴ -6z ³ +3.

The number of viewpoints at each scene level is set to 15; according to the change function of the perceptual resolution with the display depth of the light field, the perceptual resolution value corresponding to the average display depth of each scene is selected as the rendering resolution of the scene level. Use the new viewpoint image generation method to process the multi-viewpoint images of the level where the zero plane is located and n nearby levels. Each scene level generates up to 15 new viewpoint images, and the scene levels far away from the zero plane do not generate new viewpoint images.

Preferably, the new viewpoint image generation technology includes but not limited to deep learning-based, optical flow-based, parallax or depth image-based rendering methods.

Preferably, when rendering the 3D scene of each level, only the scene content of the level between the near plane and the far plane is rendered; the rendering method includes but not limited to rasterization, path tracing, ray tracing algorithm .

Step 4: Determine the occlusion relationship between different levels to obtain a parallax image;

The depth priority of the layers is obtained according to the distance between the layers and the viewpoint. The layer farther away from the viewpoint has a lower priority and is drawn first, and the layer farther away from the viewpoint has a higher priority and is drawn later. Therefore, the scene content close to the viewpoint covers the scene content far away from the viewpoint, forming a parallax image with a correct occlusion relationship.

Step 5: Perform super-resolution reconstruction and pixel encoding on the disparity image to obtain a high-resolution composite image.

The parallax image obtained by rendering is used as the input of the super-resolution neural network, and the super-resolution scale factor r=2 is selected to realize the super-resolution reconstruction of the batch parallax image and obtain a high-resolution parallax image. According to the display of the 3D display device feature, which encodes the pixels in the disparity image to obtain a high-resolution composite image.

The super-resolution neural network is a generated confrontation network. Among them, the generative network consists of multiple deformable convolutional layers, activation units, a sub-pixel convolutional layer and a residual connection in series. For a low-resolution disparity image with an input size of H×W, multiple deformable convolution layers are used to extract its features, and the output feature map size is 4×H×W after multiple deformable convolution operations. Sub-pixel convolution is performed on the feature maps extracted after multiple convolutions to obtain a high-resolution feature map with a size of 2H×2W. The high-resolution feature map is concatenated with the image directly obtained by upsampling from the input to output a high-resolution disparity image.

The discriminant network is proposed in the paper "C. Ledig etc, Photo-realistic single image super-resolution using a generative adversarial network, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2017), pp.4681-4690." The discriminative network structure. Optionally, other methods can also be used or the neural network can be built by itself as the discriminant network.

According to the display characteristics of the 3D light field display, pixel encoding is performed on the sub-pixels in the parallax image sequence to obtain a high-resolution composite image. The three-dimensional light field display is a cylindrical lens grating three-dimensional display.

Preferably, the three-dimensional display hardware device is composed of a display panel and a backlight plate, including but not limited to a slit grating three-dimensional display, a cylindrical lens grating three-dimensional display, and an integrated imaging three-dimensional display.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (10)

1. A scene layering-based light field rendering method, comprising the steps of:

step 1, obtaining a change curve of perceived resolution along with the display depth of a light field in three-dimensional light field display through experiments;

step 2, selecting a depth threshold according to a change curve of the perceived resolution along with the display depth of the light field, and dividing a scene into different levels;

step 3, rendering the 3D scenes of different levels;

step 4, determining shielding relations among different layers to obtain a low-resolution parallax image;

and 5, performing super-resolution reconstruction and pixel coding on the parallax image to obtain a high-resolution synthetic image.

2. The scene hierarchy-based light field rendering method according to claim 1, wherein the specific steps of rendering 3D scenes of different levels are as follows:

obtaining a change function of the perceived resolution along with the light field display depth by using a change curve of the perceived resolution along with the light field display depth obtained through experiments, and setting rendering resolutions of scene levels positioned at different display depths according to the change function, so that the rendering resolution of each scene level is the perceived resolution at the current display depth;

generating new viewpoint images for scene levels where zero planes are located and scene levels near n zero planes, so that the levels have rendering results with high discretization degree and high definition;

for other layers far away from the zero plane, generating a new viewpoint image is not performed, and a low-resolution rendering result with low discretization degree and low definition is obtained;

the generation technology of the new viewpoint image comprises a drawing method based on deep learning, optical flow, parallax or depth image.

3. The scene layering-based light field rendering method according to claim 1, wherein occlusion relations among different layers are determined to obtain a low-resolution parallax image, and the specific steps are as follows:

and according to the distance between the hierarchy and the view point, sequencing the hierarchy to determine the priority, so that the scene hierarchy image close to the view point covers the scene hierarchy image far away from the view point, and obtaining the parallax image with the correct shielding relation.

4. The scene layering-based light field rendering method according to claim 1, wherein the step of experimentally obtaining a change curve of perceived resolution with light field display depth in three-dimensional light field display is as follows:

and performing a light field simulation experiment on the 3D scene by using a model rendering tool, changing the display depth of the scene by adjusting the position of a zero plane of the virtual scene, and measuring the perceived resolution observed by eyes in three-dimensional light field display, thereby obtaining a change curve of the perceived resolution along with the display depth of the light field in the three-dimensional light field display.

5. The scene hierarchy based light field rendering method of claim 1, wherein the perceived resolution is a curve of change in light field display depth, wherein:

the corresponding abscissa of the change curve represents the light field display depth, namely the distance between the displayed scene plane and the scene zero plane, and the display depth is a negative value if the displayed scene plane is behind the scene zero plane when the displayed scene plane is watched from the viewpoint direction; if the displayed scene plane is before the scene zero plane, the display depth is a positive value; the change curve corresponds to a numerical value whose ordinate represents the perceived resolution of the human eye.

6. The scene hierarchy based light field rendering method of claim 1, wherein the step of selecting a depth threshold and dividing the scene into different levels is as follows:

selecting a zero plane of the 3D scene, and selecting a plurality of screen-out and screen-in depths as thresholds according to the obtained resolution change curve to carry out depth division on the 3D scene in the model rendering tool so as to obtain a plurality of 3D scene levels;

two depth thresholds are needed for dividing any 3D scene level, wherein the plane where the depth threshold with a large value is located is the near plane of the scene level, and the plane where the depth threshold with a small value is located is the far plane of the scene level.

7. The scene hierarchy-based light field rendering method according to claim 1, wherein the specific steps of rendering 3D scenes of different levels are as follows:

when rendering the 3D scene of each level, only rendering the scene content of the level between the near plane and the far plane;

the rendering method comprises rasterization, path tracking and ray tracking algorithms.

8. The scene layering-based light field rendering method according to claim 1, wherein there are two ways to reconstruct the parallax image in super resolution and encode the pixels to obtain a high resolution synthetic image;

the method comprises the following specific steps:

taking the parallax image obtained by rendering as the input of the super-resolution neural network;

selecting a scale factor of super resolution, and realizing super resolution reconstruction of batch parallax images to obtain high-resolution parallax images;

according to the display characteristics of the three-dimensional display equipment, pixels in the parallax image are encoded, and a high-resolution synthetic image is obtained;

the method comprises the following specific steps:

encoding pixels in the low-resolution parallax image according to the display characteristics of the three-dimensional display device to obtain a low-resolution composite image;

and taking the single low-resolution synthetic image as the input of the super-resolution neural network, and selecting the scale factor of the super-resolution so that the super-resolution neural network outputs the single high-resolution synthetic image.

9. The scene hierarchy based light field rendering method of claim 8, wherein the super-resolution neural network is a generation countermeasure network comprising a generation network and a discrimination network; the generating network is composed of a plurality of deformable convolution layers, a plurality of activating units and a sub-pixel convolution layer and a residual error connection, wherein the plurality of deformable convolution layers are used for extracting the characteristics of a low-resolution image with the input size of H multiplied by W, the sub-pixel convolution layer changes the low-resolution image into a high-resolution characteristic image with the size of rH multiplied by rW through pixel recombination operation, the residual error connection is used for connecting the high-resolution characteristic image with an image obtained from an input end through up sampling directly, network identity mapping is initially input through learning residual error, the degradation problem of the network is avoided, and a final high-resolution reconstructed image is output; the discrimination network consists of a plurality of convolution layers, a BN layer, an activation unit and a full connection layer and is used for optimizing parameters for generating a network model.

10. The scene hierarchy based light field rendering method of claim 1, wherein the three-dimensional display hardware comprises a slit grating three-dimensional display, a lenticular grating three-dimensional display, or an integrated imaging three-dimensional display.