WO2015188666A1 - Three-dimensional video filtering method and device - Google Patents

Abstract

Provided are a three-dimensional video filtering method and device, the three-dimensional video filtering method comprising: projecting the pixels in the plane of an image into three-dimensional space; according to the coordinate values of the pixels to be filtered and the reference pixels in the three-dimensional space, calculating the spatial proximity of the pixels to be filtered and the reference pixels in the three-dimensional space; according to the texture pixel values of the pixels to be filtered and the reference pixels, calculating texture pixel value similarity of the pixels to be filtered and the reference pixels; according to the texture pixel values of the pixels to be filtered, the reference pixels, and the pixels in the same position in the previous frame of the image, calculating motion characteristics consistency of the pixels to be filtered and the reference pixels; calculating the weight of the pixels to be filtered; determining filtering weight according to the proximity, texture pixel value similarity and motion characteristics consistency, and conducting weighted averaging on the pixel values of the reference pixels to obtain a filtering result of the pixels to be filtered, thus improving three-dimensional video filtering accuracy.

Description

Three-dimensional video filtering method and device Technical field

Embodiments of the present invention relate to image processing technologies, and in particular, to a three-dimensional video filtering method and apparatus.

Background technique

With the continuous development of visual multimedia technology, 3D video gradually enters people's lives with its unique depth of field effect, and is applied in many fields such as education, military, entertainment and medical. The current 3D video is mainly divided into two categories according to the video content: pure color 3D video and depth based 3D video. Pure color 3D video directly presents multi-channel color video to users, and its viewpoint position and parallax are fixed, which brings certain limitations to people's viewing. Compared with pure color 3D video, due to the introduction of depth map, depth-based 3D video can synthesize virtual images of arbitrary viewpoints by depth image-based rendering technology. People can select viewpoints and adjust parallax according to personal preference, so as to enjoy better. 3D video brings fun. This free and flexible feature makes depth-based 3D video the currently accepted 3D video format.

The depth-based 3D video content consists of a sequence of texture maps and a sequence of depth maps that visually represent the texture features of the surface of the object, and the depth map reflects the distance between the object and the camera. The specified virtual view texture image can be synthesized using the above video content and depth image based rendering techniques. However, depth maps and texture maps introduce a lot of noise during acquisition, encoding, and transmission. In the view synthesis stage, the noise in the depth map and texture map will cause geometric distortion and texture distortion of the composite image, which will seriously affect people's visual experience. The filtering technology can effectively remove these noises and effectively improve the quality of 3D video.

In the prior art, the denoising method for the texture map is mainly a bilateral filter, and the filtering result is obtained by using the pixels around the pixel to be filtered as a reference and weighting and averaging the same. In the process of weight calculation, the similarity between pixel positional proximity and pixel value in the image is mainly referred to. The filtering method considers that the closer the distance between two pixel points in the image plane is, the stronger the correlation is; the more similar the pixel values of the two pixel points are, the stronger the correlation is.

1 is a schematic diagram of the computational proximity of a prior art bilateral filter. The problem of the prior art is that since the pixel points in the image are the reproduction of the points in the real three-dimensional space in the two-dimensional image plane, The bilateral filter does not start from the real three-dimensional scene when considering the pixel proximity. The calculation result is not accurate, as shown in Figure 1, where A', B', C' are three points in the real scene. The position in the image plane is A, B, C collected by the camera, and the distance between A and C on the plane is equal to the distance between B and C on the plane. If C is filtered, A and B are reference pixels. As shown in Fig. 1, it can be clearly seen that in three-dimensional space, the proximity of B and C is stronger, while the bilateral filter considers that the proximity of A and C and B is consistent with C, so the accuracy of the filtering result is not high.

Summary of the invention

The embodiment of the invention provides a three-dimensional video filtering method and device to overcome the problem that the filtering result in the prior art is not high.

In a first aspect, an embodiment of the present invention provides a three-dimensional video filtering method, including:

Projecting pixels in the image plane into a three-dimensional space; the pixels comprising a pixel to be filtered and a reference pixel set;

Calculating spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; The reference pixel set is in the same frame image as the pixel to be filtered;

Calculating texture pixel value similarity of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

Calculating the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel in the same position in the previous frame image of the frame in which the pixel to be filtered is located Consistency of motion characteristics;

Determining the weights of the filtering according to the spatial proximity, the texel value similarity, and the motion feature consistency, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered.

With reference to the first aspect, in a first implementation manner of the first aspect, when filtering the depth image, determining filtering according to the spatial proximity, the similarity of the texel values corresponding to the depth pixels, and the consistency of the motion features a weighted average of the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel value of the depth image to be filtered pixel; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the motion feature consistency, respectively, for the reference pixels in the reference pixel set The texture pixel values are weighted averaged to obtain a filtering result of the texture pixel values of the texture image to be filtered.

In conjunction with the first aspect, or the first implementation of the first aspect, in the second implementation of the first aspect, the projecting the pixels in the image plane into the three-dimensional space includes:

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the using the depth image information, the viewpoint position information, and the reference camera parameter information provided by the three-dimensional video to project the pixel from the image plane to Three-dimensional space, including:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

With reference to the first aspect, or the first to the third implementation of the first aspect, in a fourth implementation manner of the first aspect, the spatial proximity is through the pixel to be filtered in the three-dimensional space and the The distance of the reference pixel is calculated as an input value of the function; the output value of the function increases as the input value decreases;

The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases;

The motion feature consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, including:

When the difference between the texel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame is And determining, by the difference between the reference pixel and the texel value of the pixel of the corresponding position in the previous frame, that the difference between the reference pixel and the pixel of the reference pixel is greater than or less than a preset threshold; The motion states of the pixel to be filtered and the reference pixel are inconsistent.

With reference to the first aspect, or the first to fourth implementation manners of the first aspect, in the fifth implementation manner of the first aspect, the spatial proximity, texture pixel value similarity, and motion feature are The consistency determines the weight of the filtering, and performs weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered, including:

计算获得所述待滤波像素的深度像素值的滤波结果；或，According to formula (1):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果；According to formula (2):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating texture pixel value similarity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的运动特征一致性；

And calculating a motion feature consistency of the pixel to be filtered and the reference pixel;

Where p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, P, Q are p, q in three-dimensional space In the coordinate values, T _p , T _q are the texel values of p and q, T _{p '} and T _{q '} are the texel values of p and q at the same position in the previous frame, and T _p ' is the texture of the p-filtered texture. The pixel value, th is the preset texture pixel difference threshold.

In a second aspect, an embodiment of the present invention provides a three-dimensional video filtering method, including:

Projecting pixels in the image plane into a three-dimensional space; the pixels comprising a pixel to be filtered and a reference pixel set;

Calculating spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; The reference pixel set is in the same frame image and the adjacent multi-frame image as the pixel to be filtered;

Calculating texture pixel value similarity of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

Calculating a time domain proximity of the pixel to be filtered and the reference pixel according to a time interval of a frame in which the pixel to be filtered and a reference pixel in the reference pixel set are located;

Determining the weights of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered.

With reference to the second aspect, in a first implementation manner of the second aspect, when filtering the depth image, determining filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the time domain proximity a weighted average of the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel value of the depth image to be filtered pixel; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the time domain proximity, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel.

With reference to the second aspect, or the first implementation of the second aspect, in the second implementation manner of the second aspect, the projecting the pixels in the image plane into the three-dimensional space includes:

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

With reference to the second implementation manner of the second aspect, in a third implementation manner of the second aspect, the using the depth image information, the viewpoint location information, and the reference camera parameter information provided by the three-dimensional video, the pixel from the image plane Projected into 3D space, including:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

With reference to the second aspect, or the first to the third implementation of the second aspect, in a fourth implementation manner of the second aspect, the spatial proximity is through the pixel to be filtered in the three-dimensional space and the The distance of the reference pixel is calculated as an input value of the function; the output value of the function increases as the input value decreases;

The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases;

The time domain proximity is calculated by using a time interval of the pixel to be filtered and a frame in which the reference pixel is located as a function of an input value; an output value of the function increases as the input value decreases.

With reference to the second aspect, or the first to fourth implementation manners of the second aspect, in the fifth implementation manner of the second aspect, the spatial proximity, texture pixel value similarity, and time domain are And determining, by the weighting average, the weighted average of the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered, including:

计算获得所述待滤波像素的深度像素值的滤波结果；或，According to formula (3):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果； According to formula (4):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的纹理像素值相似性；

Means for calculating texture pixel value similarity between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel;

第i帧中q的深度像素值，P、Q_i为p、第i帧中q在三维空间中的坐标值，T_p、

分别为p、第i帧中q的纹理像素值，T_p'为p滤波后的纹理像素值。Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, i is an integer in the interval [Nm, N+n], and m and n are respectively before the frame in which the pixel to be filtered is located And the number of reference frames, m, n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the ith frame, K _i is the reference pixel set in the ith frame, and D _p ' is p Filtered depth pixel value,

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value.

In a third aspect, an embodiment of the present invention provides a three-dimensional video filtering apparatus, including:

a projection module, configured to project pixels in an image plane into a three-dimensional space; the pixels include a pixel to be filtered and a reference pixel set;

a calculation module, configured to calculate spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the reference pixel in the to-be-filtered pixel and the reference pixel set in the three-dimensional space And wherein the reference pixel set is in the same frame image as the pixel to be filtered;

The calculating module is further configured to calculate, according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set, a texture pixel value similarity between the pixel to be filtered and the reference pixel;

The calculating module is further configured to calculate, according to the texel value of the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel of the same position in the previous frame image of the frame where the pixel to be filtered is located, Consistency of motion characteristics of the pixel to be filtered and the reference pixel;

a filtering module, configured to determine a weight of the filtering according to the spatial proximity, the texel value similarity, and the motion feature consistency, and respectively perform pixel values of the reference pixels in the reference pixel set A weighted average obtains a filtering result of the pixel to be filtered.

With reference to the third aspect, in a first implementation manner of the third aspect, the filtering module is specifically configured to:

When filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the motion feature consistency, respectively, the depth of the reference pixel in the reference pixel set Performing a weighted average of the pixel values to obtain a filtering result of the depth pixel value of the pixel to be filtered of the depth image; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the motion feature consistency, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel.

With reference to the third aspect, or the first implementation manner of the third aspect, in the second implementation manner of the third aspect, the

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

With the second implementation of the third aspect, in a third implementation manner of the third aspect, the projection module is specifically configured to:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

With reference to the third aspect, or the first to the third implementation of the third aspect, in a fourth implementation manner of the third aspect, the spatial proximity is through the pixel to be filtered in the three-dimensional space and the The distance of the reference pixel is calculated as an input value of the function; the output value of the function increases as the input value decreases;

The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases;

The motion feature consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, including:

a difference between a texel value of the pixel to be filtered and a pixel at a corresponding position in the previous frame, and a difference between the reference pixel and a texel value of the pixel at a corresponding position in the previous frame, which are simultaneously larger or smaller than The threshold value is determined to be consistent with the motion state of the pixel to be filtered and the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered and the reference pixel are inconsistent.

With reference to the third aspect, or the first to the fourth implementation manner of the third aspect, in the fifth implementation manner of the third aspect, the filtering module is specifically configured to:

计算获得所述待滤波像素的深度像素值的滤波结果；或，According to formula (1):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果；According to formula (2):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating texture pixel value similarity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的运动特征一致性；

And calculating a motion feature consistency of the pixel to be filtered and the reference pixel;

Where p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, P, Q are p, q in three-dimensional space In the coordinate values, T _p , T _q are the texel values of p and q, T _{p '} and T _{q '} are the texel values of p and q at the same position in the previous frame, and T _p ' is the texture of the p-filtered texture. The pixel value, th is the preset texture pixel difference threshold.

In a fourth aspect, an embodiment of the present invention provides a three-dimensional video filtering apparatus, including:

a projection module, configured to project pixels in an image plane into a three-dimensional space; the pixels include a pixel to be filtered and a reference pixel set;

a calculation module, configured to calculate spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the reference pixel in the to-be-filtered pixel and the reference pixel set in the three-dimensional space And the reference pixel set is in the same frame image and the adjacent multi-frame image as the pixel to be filtered;

The calculating module is further configured to calculate, according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set, a texture pixel value similarity between the pixel to be filtered and the reference pixel;

The calculating module is further configured to calculate a time domain proximity of the pixel to be filtered and the reference pixel according to a time interval of a frame in which the reference pixel in the pixel to be filtered and the reference pixel in the reference pixel set are located;

And a filtering module, configured to determine a weight of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, and perform weighted averaging on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be-filtered The filtering result of the pixel.

With reference to the fourth aspect, in a first implementation manner of the fourth aspect, the filtering module is specifically configured to:

When filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the time domain proximity, respectively, the depth of the reference pixel in the reference pixel set Performing a weighted average of the pixel values to obtain a filtering result of the depth pixel value of the pixel to be filtered of the depth image; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the time domain proximity, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel.

With reference to the fourth aspect, or the first implementation manner of the fourth aspect, in the second implementation manner of the fourth aspect, the

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

With reference to the second implementation manner of the fourth aspect, in a third implementation manner of the fourth aspect, the

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

With reference to the fourth aspect, or the first to the third implementation of the fourth aspect, in a fourth implementation manner of the fourth aspect, the spatial proximity is through the pixel to be filtered in the three-dimensional space and the The distance of the reference pixel is calculated as an input value of the function; the output value of the function increases as the input value decreases;

The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases;

The time domain proximity is calculated by using a time interval of the pixel to be filtered and a frame in which the reference pixel is located as a function of an input value; an output value of the function increases as the input value decreases.

With reference to the fourth aspect, or the first to the fourth implementation manner of the fourth aspect, in the fifth implementation manner of the fourth aspect, the filtering module is specifically configured to:

计算获得所述待滤波像素的深度像素值的滤波结果；或，According to formula (3):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果；According to formula (4):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的纹理像素值相似性；

Means for calculating texture pixel value similarity between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel;

第i帧中q的深度像素值，P、Q_i为p、第i帧中q在三维空间中的坐标值，T_p、

分别为p、第i帧中q的纹理像素值，T_p'为p滤波后的纹理像素值。Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, i is an integer in the interval [Nm, N+n], and m and n are respectively before the frame in which the pixel to be filtered is located And the number of reference frames, m, n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the ith frame, K _i is the reference pixel set in the ith frame, and D _p ' is p Filtered depth pixel value,

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value.

The three-dimensional video filtering method and apparatus of the embodiment of the present invention calculates the spatial proximity and texture pixels of the pixel to be filtered and the reference pixel in the three-dimensional space by using the relationship between the pixel to be filtered and the reference pixel in the real three-dimensional space. Value similarity, motion feature consistency, and time domain proximity; determine the weight of the filter based on spatial proximity, texture pixel value similarity, and motion feature consistency, or based on spatial proximity, texture pixel value similarity, and time domain proximity Sex determination of the weight of the filter, respectively The weighted average of the pixel values of the reference pixels in the reference pixel set is used to obtain the filtering result of the pixel to be filtered, and the spatial proximity, texture pixel value similarity, motion feature consistency and time domain proximity are comprehensively considered when calculating the weight. Since the space proximity is used to calculate the position in the real three-dimensional space, and since the three-dimensional video is composed of a series of images collected from different moments, the pixels between different frames also have correlation, so the weight considers the time domain. The continuity between the frame and the frame after the proximity filtering is strong, and the motion feature consistency between the pixel points is considered, the accuracy of the filtering result is improved, and the problem that the filtering result is not high in the prior art is solved. .

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief description of the drawings used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any inventive labor.

1 is a schematic diagram of computational proximity of a prior art bilateral filter;

2 is a flowchart of Embodiment 1 of a three-dimensional video filtering method according to the present invention;

3 is a schematic diagram of pixel projection according to Embodiment 1 of the method of the present invention;

4 is a flowchart of Embodiment 2 of a three-dimensional video filtering method according to the present invention;

FIG. 5 is a schematic diagram of reference pixel selection according to Embodiment 2 of the method of the present invention; FIG.

6 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device according to the present invention;

FIG. 7 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device according to the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. It is a partial embodiment of the invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

FIG. 2 is a flowchart of Embodiment 1 of a method for filtering a three-dimensional video according to the present invention, and FIG. 3 is a schematic diagram of pixel projection according to Embodiment 1 of the method of the present invention. As shown in FIG. 2, the method in this embodiment may include:

Step 201: Project a pixel in an image plane to a three-dimensional space; the pixel includes a pixel to be filtered and a reference pixel set.

Optionally, projecting pixels in the image plane into the three-dimensional space includes:

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

Optionally, the projecting the pixels from the image plane to the three-dimensional space by using the depth image information, the viewpoint position information and the reference camera parameter information provided by the three-dimensional video comprises:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

应用三维投影技术，将像素投影到世界坐标系中的P点，所述像素在所世界坐标系中的坐标值为

该坐标值可以通过公式P＝R^-1(dA^-1p-t)计算得到，其中R和t为参考相机的旋转矩阵和平移矢量，d为像素的深度像素值，可以由三维视频提供的深度图信息得到，A为参考相机参数矩阵

f_x和f_y分别为水平和竖直方向的归一化焦距， r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。As shown in FIG. 3, the plane where the uv coordinates are located is an image plane, the pixel positions in the three-dimensional space are represented by coordinates in the world coordinate system, p is a pixel in the image plane, and the coordinates of the pixels in the image plane are for

Applying 3D projection technology to project pixels to P points in the world coordinate system, the coordinates of the pixels in the world coordinate system

The coordinate value can be calculated by the formula P=R ^-1 (dA ^-1 pt), where R and t are the rotation matrix and translation vector of the reference camera, d is the depth pixel value of the pixel, and the depth map can be provided by the 3D video. Information is obtained, A is the reference camera parameter matrix

f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, r is a radial distortion coefficient, and (o _x , o _y ) is a coordinate value of a reference point on the image plane; The intersection of the optical axis of the reference camera and the image plane.

Step 202: Calculate spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the reference pixel in the pixel to be filtered and the reference pixel in the reference pixel set; wherein, the reference pixel set and the pixel to be filtered are In the same frame image.

Optionally, the spatial proximity is calculated by using a distance of the pixel to be filtered and the reference pixel in a three-dimensional space as a function of an input value; an output value of the function increases as the input value decreases.

Specifically, the spatial distance between two points may reflect the spatial proximity thereof. The closer the distance is, the stronger the correlation is, the larger the spatial proximity is, that is, the reference pixels in the pixel to be filtered and the reference pixel set in the three-dimensional space. The coordinate value calculates the spatial distance, and the spatial distance is used as the input value, for example, the spatial proximity is calculated by a Gaussian function. The function for calculating the spatial proximity may also be other functions, but it is necessary to ensure that the output value of the function decreases with the input value. Small and increasing; wherein the reference pixel set in this embodiment is in the same frame as the pixel to be filtered.

Step 203: Calculate texture pixel value similarity of the pixel to be filtered and the reference pixel according to the texel value of the reference pixel in the pixel to be filtered and the reference pixel set.

Optionally, the texture pixel value similarity is calculated by using a difference between a texel value of the pixel to be filtered and the reference pixel as a function of an input value; and an output value of the function decreases with an input value. And increase.

Specifically, the degree of difference in the texture features between the two points reflects the degree of similarity. The more similar the texture is, the stronger the correlation is, the greater the similarity of the texel values, that is, the texture of the pixel to be filtered and the reference pixel. The pixel value is calculated as a difference value, and the difference value is used as an input value to calculate the similarity of the texel value, for example, by a Gaussian function. The function for calculating the similarity of the texel value may also be other functions, but it is necessary to ensure that the output value of the function follows The input value decreases as the value decreases.

Step 204: Calculate motion feature consistency of the pixel to be filtered and the reference pixel according to the texel value of the pixel in the same position in the pixel to be filtered, the reference pixel in the reference pixel set, and the previous frame image of the frame in which the pixel to be filtered is located.

Optionally, the motion feature consistency is obtained by calculating whether the motion feature of the pixel to be filtered and the reference pixel are consistent, including:

a difference between a texel value of the pixel to be filtered and a pixel at a corresponding position in the previous frame, and a difference between the reference pixel and a texel value of a pixel at a corresponding position in the previous frame, and greater than Or less than a preset threshold, determining that the motion characteristics of the pixel to be filtered and the reference pixel are consistent; otherwise determining that the motion characteristics of the pixel to be filtered and the reference pixel are inconsistent.

Specifically, the relationship between the relative motions between the two points also reflects its similarity in motion, and the more similar the motion, the stronger the correlation. Since it is difficult to obtain the motion information of the pixel from the three-dimensional video sequence, the embodiment of the present invention determines whether the pixel moves by using the difference between the pixels of the two positions of the pixels in the image plane at the same position in the image plane, when the difference is greater than a certain value. When a preset threshold is used, the motion feature of the pixel is considered to be motion, and conversely, the motion feature of the pixel is considered to be motionless; further, the pixel to be filtered and the reference pixel of the two frames before and after are in the same position in the image plane. The difference is used to determine whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent. When the difference is greater than or less than a certain threshold, the motion characteristics of the pixel to be filtered and the reference pixel are considered to be the same. Inconsistent features. If the motion characteristics of the pixels are consistent, it is considered to be relevant, and vice versa.

Step 205: Determine a weight of the filter according to spatial proximity, texture pixel value similarity, and motion feature consistency, and perform weighted average on the pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the pixel to be filtered.

Optionally, when filtering the depth image, determining, according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the motion feature consistency, determining a weight of the filtering, respectively, in the reference pixel set Performing a weighted average of the depth pixel values of the reference pixels to obtain a filtering result of the depth pixel values of the pixels to be filtered by the depth image; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the motion feature consistency, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the pixel to be filtered of the image to be textured.

Optionally, determining the weight of the filtering according to the spatial proximity, the texel value similarity, and the motion feature consistency, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered, including:

计算获得待滤波像素的深度像素值的滤波结果；或， According to formula (1):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得待滤波像素的纹理像素值的滤波结果；According to formula (2):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算待滤波像素和参考像素的空间邻近性；among them,

Used to calculate spatial proximity of the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T ⁽ ||T _p −T _q ||) is used for calculating texture pixel value similarity of the pixel to be filtered and the reference pixel;

用于计算待滤波像素和参考像素的运动特征一致性；

Used to calculate motion feature consistency of the pixel to be filtered and the reference pixel;

Where p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, P, Q are p, q in three-dimensional space In the coordinate values, T _p , T _q are the texel values of p and q, T _{p '} and T _{q '} are the texel values of p and q at the same position in the previous frame, and T _p ' is the texture of the p-filtered texture. The pixel value, th is the preset texture pixel difference threshold.

对参考像素集合中的参考像素进行加权平均，计算获得待滤波像素的深度像素值的滤波结果；根据公式

对参考像素集合中的参考像素进行加权平均，计算获得待滤波像素的纹理像素值的滤波结果；Specifically, according to the formula

Performing weighted averaging on the reference pixels in the reference pixel set, and calculating a filtering result of obtaining the depth pixel value of the pixel to be filtered; according to the formula

Performing a weighted average on the reference pixels in the reference pixel set, and calculating a filtering result of obtaining the texel value of the pixel to be filtered;

用于计算待滤波像素和参考像素的空间邻近性；该函数的输入值为所述待滤波像素和所述参考像素的空间距离；所述函 数的输出值随着输入值的减小而增大；among them,

Means for calculating spatial proximity of the pixel to be filtered and the reference pixel; an input value of the function is a spatial distance between the pixel to be filtered and the reference pixel; an output value of the function increases as the input value decreases ;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating the texel value similarity of the pixel to be filtered and the reference pixel; the input value of the function is the pixel to be filtered a difference from a texel value of the reference pixel; an output value of the function increases as the input value decreases;

用于计算待滤波像素和参考像素的运动特征一致性；即当所述待滤波像素与前一帧中对应位置的像素的纹理像素值的差值，以及所述参考像素与前一帧中对应位置的像素的纹理像素值的差值，同时大于或小于预设的阈值时确定为所述待滤波像素和所述参考像素的运动特征一致；否则确定为所述待滤波像素和所述参考像素的运动特征不一致。

For calculating the motion feature consistency of the pixel to be filtered and the reference pixel; that is, the difference between the texel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame, and the reference pixel corresponding to the previous frame The difference between the texel values of the pixels of the location is greater than or less than the preset threshold, and is determined to be consistent with the motion features of the pixel to be filtered and the reference pixel; otherwise determined as the pixel to be filtered and the reference pixel The motion characteristics are inconsistent.

Where p is the pixel to be filtered, q is the reference pixel, and K is the reference pixel set. This set is usually taken as a square region centered on the pixel to be filtered, and the size is 5*5 or 7*7, and D _p ' is filtered by p Depth pixel value, D _q is the depth pixel value of q, P, Q are the coordinate values of p, q in three-dimensional space, T _p , T _q are texture pixel values of p, q, T _{p '} , T _{q '} The texture pixel values of p and q at the same position in the previous frame, where the same position refers to the same position in the image plane, T _p ' is the filtered tex value of p, and th is the preset texture pixel difference Threshold. Th is a threshold value for judging whether the motion characteristics of the pixel points are consistent, and may be selected according to different contents of the three-dimensional video sequence, generally 6 to 20. When the selection is appropriate, the boundary of the moving object can be better distinguished, so that the boundary of the filtered object is further improved. obvious.

It should be noted that, in this embodiment, step 202, step 203, and step 204 are in no particular order.

In this embodiment, the spatial proximity, the texel value similarity, and the motion feature of the pixel to be filtered and the reference pixel in the three-dimensional space are calculated by using a relationship between the pixel to be filtered and the reference pixel in a real three-dimensional space. Determining, according to the spatial proximity, the texel value similarity and the motion feature consistency, determining the weight of the filtering, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered, Space is considered when calculating weights Proximity, texture pixel value similarity, motion feature consistency, the position in the real three-dimensional space is utilized when calculating the spatial proximity, and the motion feature consistency between the pixel points is increased, and the accuracy of the filtering result is improved. The problem that the filtering result in the prior art is not high is solved.

4 is a flowchart of a second embodiment of a three-dimensional video filtering method according to the present invention. FIG. 5 is a schematic diagram of reference pixel selection according to a second embodiment of the present invention. As shown in FIG. 4, the method in this embodiment may include:

Step 401: Project a pixel in an image plane to a three-dimensional space; the pixel includes a pixel to be filtered and a reference pixel set.

Optionally, projecting pixels in the image plane into the three-dimensional space includes:

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

Optionally, using the depth image information, the viewpoint location information, and the reference camera parameter information provided by the three-dimensional video to project the pixel from the image plane to the three-dimensional space, including:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

Step 402: Calculate spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the reference pixel in the to-be-filtered pixel and the reference pixel set in the three-dimensional space; The reference pixel set is in the same frame image and the adjacent multi-frame image as the pixel to be filtered.

Optionally, the spatial proximity is calculated by using a distance of the pixel to be filtered and the reference pixel in a three-dimensional space as a function of an input value; an output value of the function increases as the input value decreases.

Step 403: Calculate texture pixel value similarity between the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set.

Optionally, the texture pixel value similarity is calculated by using a difference between a texel value of the pixel to be filtered and the reference pixel as a function of an input value; and an output value of the function decreases with an input value. Increase

The implementation principles in the steps 401, 402, and 403 are similar to those in the first embodiment, and are not described here.

Step 404: Calculate time domain proximity of the pixel to be filtered and the reference pixel according to a time interval of a frame in which the pixel to be filtered and a reference pixel in the reference pixel set are located.

Optionally, the time domain proximity is calculated by using a time interval of the pixel to be filtered and a frame where the reference pixel is located as a function input value; an output value of the function increases as the input value decreases. .

Specifically, since the pixels between different frames collectively reflect the state of the object in a period of time, there is a certain correlation, and the correlation can also be used in the filtering method of the image in the three-dimensional video. Therefore, the present invention extends the selection range of the reference pixel from the frame where the pixel to be filtered is located to its adjacent frame (filtered reference frame) to increase the filtered frame and frame, based on the weight calculation method of the first embodiment. Continuity between. As shown in FIG. 5, in each filtering reference frame, the selected range of the reference pixel is consistent with the selected range of the reference pixel in the frame to be filtered, wherein the Nth frame is the frame where the pixel to be filtered is currently located, and the previous m frame and the subsequent n are selected. The frame is used as the filtering reference frame, and the reference pixel window of each frame is the same as the coordinate and size of the reference pixel window of the Nth frame (here, the coordinates and the size on the image plane are the same), and K _i is the reference pixel set in the ith frame, and the value of i is For an integer in the interval [Nm, N+n], m and n are non-negative integers, and when n=0, it means that only the frame that has been encoded or decoded before the frame to be filtered is acquired.

The distance between the two points in the time domain reflects the degree of proximity of the time domain. The closer the time domain distance is, the stronger the correlation is, the larger the time domain proximity is, that is, the pixel to be filtered and the reference pixel can be located. The frame calculates the time interval and uses the time interval as an input value to calculate the time domain proximity, for example, by a Gaussian function. The function for calculating the time domain proximity may also be other functions, but it is necessary to ensure that the output value of the function follows the input value. The decrease increases.

Step 405: Determine weights of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, and perform weighted averaging on the pixel values of the reference pixels in the reference pixel set respectively to obtain the pixels to be filtered. Filter the result.

Optionally, when filtering the depth image, according to the spatial proximity, the depth pixel corresponds to The texel value similarity and the time domain proximity determine the weight of the filtering, respectively performing weighted averaging on the depth pixel values of the reference pixels in the reference pixel set to obtain the depth pixel value of the depth image to be filtered pixel Filtered result; or,

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the time domain proximity, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel.

Optionally, determining the weight of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the to-be-filtered pixel The filtering results include:

计算获得所述待滤波像素的深度像素值的滤波结果；或，According to formula (3):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果；According to formula (4):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；；among them,

Used to calculate spatial proximity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的纹理像素值相似性；

Means for calculating texture pixel value similarity between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel;

第i帧中q的深度像素值，P、Q_i为p、第i帧中q在三维空间中的坐标值，T_p、

分别为p、第i帧中q的纹理像素值，T_p'为p滤波后的纹理像素值。Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, i is an integer in the interval [Nm, N+n], and m and n are respectively before the frame in which the pixel to be filtered is located And the number of reference frames, m, n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the ith frame, K _i is the reference pixel set in the ith frame, and D _p ' is p Filtered depth pixel value,

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value.

对参考像素集合中的参考像素进行加权平均，计算获得待滤波像素的深度像素值的滤波结果；根据公式

对参考像素集合中的参考像素进行加权平均，计算获得待滤波像素的纹理像素值的滤波结果；Specifically, according to the formula

Performing weighted averaging on the reference pixels in the reference pixel set, and calculating a filtering result of obtaining the depth pixel value of the pixel to be filtered; according to the formula

Performing a weighted average on the reference pixels in the reference pixel set, and calculating a filtering result of obtaining the texel value of the pixel to be filtered;

用于计算待滤波像素和参考像素的空间邻近性；该函数的输入值为所述待滤波像素和所述参考像素的空间距离；所述函数的输出值随着输入值的减小而增大；among them,

Means for calculating spatial proximity of the pixel to be filtered and the reference pixel; an input value of the function is a spatial distance between the pixel to be filtered and the reference pixel; an output value of the function increases as the input value decreases ;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating the texel value similarity of the pixel to be filtered and the reference pixel; the input value of the function is the pixel to be filtered a difference from a texel value of the reference pixel; an output value of the function increases as the input value decreases;

f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel; an input value of the function is the pixel to be filtered and the The time interval of the frame of the reference pixel; the output value of the function increases as the input value decreases;

第i帧中q的深度像素值，P、Q_i为p、第i帧中q在三维空间中的坐标值，T_p、

分别为p、第i帧中q的纹理像素值，T_p'为p滤波后的纹理像素值。Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, and m and n are the number of reference frames before and after the frame in which the pixel to be filtered is located, usually m and n may be 1 ～3, because as the time interval increases, the correlation between frames and frames becomes very small, which can be ignored. p is the pixel to be filtered, q _i is the reference pixel in the ith frame, and K _i is the ith a set of reference pixels in the frame, which is usually taken as a square region centered on the pixel to be filtered, and has a size of 5*5 or 7*7, and _Dp ' is a depth pixel value after p filtering.

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value.

It should be noted that in the embodiment, step 402, step 403, and step 404 are in no particular order.

In this embodiment, the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space, the similarity of the texture pixel value, and the time domain are calculated by using the relationship between the pixel to be filtered and the reference pixel in the real three-dimensional space. Proximity; determining weights of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered When calculating weights, spatial proximity, texel value similarity and time domain proximity are considered. Since the spatial proximity is used to calculate the position in the real three-dimensional space, and because the three-dimensional video is collected from different moments The image composition, the pixel points between different frames also have correlation, so the weight value considers the continuity between the frame and the frame after the time domain proximity filtering is strong, which improves the accuracy of the filtering result and solves the prior art. The result of the filtering result is not high.

FIG. 6 is a schematic structural diagram of an embodiment of a three-dimensional video filtering apparatus according to the present invention. As shown in FIG. 6, the three-dimensional video filtering apparatus 60 of the present embodiment may include: a projection module 601, a calculation module 602, and a filtering module 603;

The projection module 601 is configured to project pixels in the image plane into the three-dimensional space; the pixels include a pixel to be filtered and a reference pixel set;

The calculating module 602 is configured to calculate a space of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate value of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space Proximity; wherein the reference pixel set is in the same frame image as the pixel to be filtered;

The calculating module 602 is further configured to calculate, according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set, a texture pixel value similarity between the pixel to be filtered and the reference pixel;

The calculating module 602 is further configured to calculate, according to the texel value of the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel in the same position in the previous frame image of the frame where the pixel to be filtered is located, Having a consistency of motion characteristics of the filtered pixel and the reference pixel;

a filtering module 603, configured to determine a weight of the filter according to the spatial proximity, the texel value similarity, and the motion feature consistency, and perform weighted averaging on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be-checked The filtered result of the filtered pixel.

Optionally, the filtering module 603 is specifically configured to:

When filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the motion feature consistency, respectively, the depth of the reference pixel in the reference pixel set Performing a weighted average of the pixel values to obtain a filtering result of the depth pixel value of the pixel to be filtered of the depth image; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the motion feature consistency, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel.

Optionally, the projection module 601 is specifically configured to:

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

Optionally, the projection module 601 is specifically configured to:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。 Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

Optionally, the spatial proximity is calculated by using an input value of the distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; an output value of the function increases as the input value decreases;

The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases;

The motion feature consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, including:

a difference between a texel value of the pixel to be filtered and a pixel at a corresponding position in the previous frame, and a difference between the reference pixel and a texel value of the pixel at a corresponding position in the previous frame, which are simultaneously larger or smaller than The threshold value is determined to be consistent with the motion state of the pixel to be filtered and the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered and the reference pixel are inconsistent.

Optionally, the filtering module 603 is specifically configured to:

计算获得所述待滤波像素的深度像素值的滤波结果；或，According to formula (1):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果；According to formula (2):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating texture pixel value similarity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的运动特征一致性；

And calculating a motion feature consistency of the pixel to be filtered and the reference pixel;

Where p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, P, Q are p, q in three-dimensional space In the coordinate values, T _p , T _q are the texel values of p and q, T _{p '} and T _{q '} are the texel values of p and q at the same position in the previous frame, and T _p ' is the texture of the p-filtered texture. The pixel value, th is the preset texture pixel difference threshold.

The device in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 2, and the implementation principle and technical effects are similar, and details are not described herein again.

In the second embodiment of the three-dimensional video filtering device of the present invention, the device of the present embodiment is based on the device structure shown in FIG. 6. Further, the projection module 601 in the three-dimensional video filtering device 60 of the present embodiment is used for the image. a pixel in a plane is projected into a three-dimensional space; the pixel includes a pixel to be filtered and a reference pixel set;

The calculating module 602 is configured to calculate a space of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate value of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space Proximity; wherein the reference pixel set is in the same frame image and adjacent multi-frame image as the pixel to be filtered;

The calculating module 602 is further configured to calculate, according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set, a texture pixel value similarity between the pixel to be filtered and the reference pixel;

The calculating module 602 is further configured to calculate a time domain proximity of the to-be-filtered pixel and the reference pixel according to a time interval between the to-be-filtered pixel and a frame in which the reference pixel in the reference pixel set is located;

a filtering module 603, configured to determine a weight of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, and perform weighted averaging on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be-determined The filtered result of the filtered pixel.

Optionally, the filtering module 603 is specifically configured to:

When filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the time domain proximity, respectively, the depth of the reference pixel in the reference pixel set Performing a weighted average of the pixel values to obtain a filtering result of the depth pixel value of the pixel to be filtered of the depth image; or

When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the time domain proximity, respectively, the texture of the reference pixel in the reference pixel set The pixel values are weighted and averaged to obtain a filtering result of the texture pixel values of the texture image to be filtered.

Optionally, the projection module 601 is specifically configured to:

The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels.

Optionally, the projection module 601 is specifically configured to:

Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt);

为所述像素在所述图像平面中的坐标值，

为所述像素在所述三维空间中的坐标值，d为所述像素的深度像素值；f_x和f_y分别为水平和竖直方向的归一化焦距，r为径向畸变系数，(o_x,o_y)为所述图像平面上的基准点的坐标值；所述基准点为所述参考相机的光轴和所述图像平面的交点。Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

Optionally, the spatial proximity is calculated by using an input value of the distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; an output value of the function increases as the input value decreases;

The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases;

The time domain proximity is calculated by using a time interval of the pixel to be filtered and a frame in which the reference pixel is located as a function of an input value; an output value of the function increases as the input value decreases.

Optionally, the filtering module 603 is specifically configured to:

计算获得所述待滤波像素的深度像素值的滤波结果；或， According to formula (3):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or

计算获得所述待滤波像素的纹理像素值的滤波结果；According to formula (4):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered;

用于计算所述待滤波像素和所述参考像素的空间邻近性；；among them,

Used to calculate spatial proximity of the pixel to be filtered and the reference pixel;

用于计算所述待滤波像素和所述参考像素的纹理像素值相似性；

Means for calculating texture pixel value similarity between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel;

第i帧中q的深度像素值，P、Q_i为p、第i帧中q在三维空间中的坐标值，T_p、

分别为p、第i帧中q的纹理像素值，T_p'为p滤波后的纹理像素值。Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, i is an integer in the interval [Nm, N+n], and m and n are respectively before the frame in which the pixel to be filtered is located And the number of reference frames, m, n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the ith frame, K _i is the reference pixel set in the ith frame, and D _p ' is p Filtered depth pixel value,

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value.

The device in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 4, and the implementation principle and technical effects are similar, and details are not described herein again.

FIG. 7 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device according to the present invention. As shown in FIG. 7, the three-dimensional video filtering device 70 provided in this embodiment includes a processor 701 and a memory 702. The memory 702 is configured to store execution instructions. When the three-dimensional video filtering device 70 is in operation, the processor 701 communicates with the memory 702, and the processor 701 calls an execution instruction in the memory 702 for executing the method described in any of the method embodiments. The technical solution has similar implementation principles and technical effects, and will not be described here.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit or module is only a logical function division, and may be implemented in actual implementation. There are additional ways of dividing, for example multiple units or modules may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or otherwise.

The modules described as separate components may or may not be physically separated. The components displayed as modules may or may not be physical modules, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

One of ordinary skill in the art will appreciate that all or part of the steps to implement the various method embodiments described above may be accomplished by hardware associated with the program instructions. The aforementioned program can be stored in a computer readable storage medium. The program, when executed, performs the steps including the foregoing method embodiments; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that The technical solutions described in the foregoing embodiments may be modified, or some or all of the technical features may be equivalently replaced; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present invention. range.

Claims (24)

A three-dimensional video filtering method, comprising: Projecting pixels in the image plane into a three-dimensional space; the pixels comprising a pixel to be filtered and a reference pixel set; Calculating spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; The reference pixel set is in the same frame image as the pixel to be filtered; Calculating texture pixel value similarity of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set; Calculating the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel in the same position in the previous frame image of the frame in which the pixel to be filtered is located Consistency of motion characteristics; Determining the weights of the filtering according to the spatial proximity, the texel value similarity, and the motion feature consistency, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered. The method according to claim 1, wherein when filtering the depth image, determining the weight of the filtering according to the spatial proximity, the similarity of the texel values corresponding to the depth pixels, and the consistency of the motion features, Performing a weighted average on the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel value of the depth image to be filtered pixel; or When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the motion feature consistency, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel. The method according to claim 1 or 2, wherein the projecting pixels in the image plane into the three-dimensional space comprises: The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels. The method according to claim 3, wherein said using said depth image information, viewpoint position information and reference camera parameter information provided by the three-dimensional video, said pixel from said image plane Projected into 3D space, including: Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt); Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane. The method according to any one of claims 1 to 4, wherein the spatial proximity is calculated by inputting a distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; The output value increases as the input value decreases; The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases; The motion feature consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, including: a difference between a texel value of the pixel to be filtered and a pixel at a corresponding position in the previous frame, and a difference between the reference pixel and a texel value of the pixel at a corresponding position in the previous frame, which are simultaneously larger or smaller than The threshold value is determined to be consistent with the motion state of the pixel to be filtered and the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered and the reference pixel are inconsistent. The method according to any one of claims 1 to 5, wherein the determining the weight of the filtering according to the spatial proximity, the texel value similarity and the motion feature consistency, respectively, the reference pixel set Performing a weighted average of the pixel values of the reference pixels within the obtained pixel to obtain the filtering result of the pixel to be filtered, including: According to formula (1):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or According to formula (2):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered; among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel; f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating texture pixel value similarity of the pixel to be filtered and the reference pixel;

And calculating a motion feature consistency of the pixel to be filtered and the reference pixel; Where p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, P, Q are p, q in three-dimensional space In the coordinate values, T _p , T _q are the texel values of p and q, T _{p '} and T _{q '} are the texel values of p and q at the same position in the previous frame, and T _p ' is the texture of the p-filtered texture. The pixel value, th is the preset texture pixel difference threshold. A three-dimensional video filtering method, comprising: Projecting pixels in the image plane into a three-dimensional space; the pixels comprising a pixel to be filtered and a reference pixel set; Calculating spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; The reference pixel set is in the same frame image and the adjacent multi-frame image as the pixel to be filtered; Calculating texture pixel value similarity of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set; Calculating a time domain proximity of the pixel to be filtered and the reference pixel according to a time interval of a frame in which the pixel to be filtered and a reference pixel in the reference pixel set are located; Determining the weights of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, respectively performing weighted averaging on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered. The method according to claim 7, wherein when filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the time domain proximity, Performing a weighted average on the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel value of the depth image to be filtered pixel; or When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the time domain proximity, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel. The method according to claim 7 or 8, wherein the projecting pixels in the image plane into the three-dimensional space comprises: The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels. The method according to claim 9, wherein the projecting the pixel from the image plane to the three-dimensional space by using the depth image information, the viewpoint position information and the reference camera parameter information provided by the three-dimensional video comprises: Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt); Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane. Method according to any of claims 7-10, wherein said spatial proximity The property is calculated by inputting the distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; the output value of the function increases as the input value decreases; The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases; The time domain proximity is calculated by using a time interval of the pixel to be filtered and a frame in which the reference pixel is located as a function of an input value; an output value of the function increases as the input value decreases. The method according to any one of claims 7 to 11, wherein the determining the weight of the filtering according to the spatial proximity, the texel value similarity and the time domain proximity, respectively, the reference pixel set Performing a weighted average of the pixel values of the reference pixels within the obtained pixel to obtain the filtering result of the pixel to be filtered, including: According to formula (3):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or According to formula (4):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered; among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

Means for calculating texture pixel value similarity between the pixel to be filtered and the reference pixel; f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel; Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, i is an integer in the interval [Nm, N+n], and m and n are respectively before the frame in which the pixel to be filtered is located And the number of reference frames, m, n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the ith frame, K _i is the reference pixel set in the ith frame, and D _p ' is p Filtered depth pixel value,

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value. A three-dimensional video filtering device, comprising: a projection module, configured to project pixels in an image plane into a three-dimensional space; the pixels include a pixel to be filtered and a reference pixel set; a calculation module, configured to calculate spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the reference pixel in the to-be-filtered pixel and the reference pixel set in the three-dimensional space And wherein the reference pixel set is in the same frame image as the pixel to be filtered; The calculating module is further configured to calculate, according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set, a texture pixel value similarity between the pixel to be filtered and the reference pixel; The calculating module is further configured to calculate, according to the texel value of the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel of the same position in the previous frame image of the frame where the pixel to be filtered is located, Consistency of motion characteristics of the pixel to be filtered and the reference pixel; a filtering module, configured to determine a weight of the filtering according to the spatial proximity, the texel value similarity, and the motion feature consistency, and perform weighted averaging on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be-filtered The filtering result of the pixel. The device according to claim 13, wherein the filtering module is specifically configured to: When filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the motion feature consistency, respectively, the depth of the reference pixel in the reference pixel set Performing a weighted average of the pixel values to obtain a filtering result of the depth pixel value of the pixel to be filtered of the depth image; or When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the motion feature consistency, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set Obtaining a filter of the texture pixel value of the texture image to be filtered pixel Wave results. The device according to claim 13 or 14, wherein the projection module is specifically configured to: The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels. The device according to claim 15, wherein the projection module is specifically configured to: Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt); Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane. The apparatus according to any one of claims 13 to 16, wherein the spatial proximity is calculated by inputting a distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; the function The output value increases as the input value decreases; The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases; The motion feature consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, including: a difference between a texel value of the pixel to be filtered and a pixel at a corresponding position in the previous frame, and a difference between the reference pixel and a texel value of the pixel at a corresponding position in the previous frame, which are simultaneously larger or smaller than The threshold value is determined to be consistent with the motion state of the pixel to be filtered and the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered and the reference pixel are inconsistent. The device according to any one of claims 13-17, wherein the filtering module is specifically configured to: According to formula (1):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or According to formula (2):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered; among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel; f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used for calculating texture pixel value similarity of the pixel to be filtered and the reference pixel;

And calculating a motion feature consistency of the pixel to be filtered and the reference pixel; Where p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, P, Q are p, q in three-dimensional space In the coordinate values, T _p , T _q are the texel values of p and q, T _{p '} and T _{q '} are the texel values of p and q at the same position in the previous frame, and T _p ' is the texture of the p-filtered texture. The pixel value, th is the preset texture pixel difference threshold. A three-dimensional video filtering device, comprising: a projection module, configured to project pixels in an image plane into a three-dimensional space; the pixels include a pixel to be filtered and a reference pixel set; a calculation module, configured to calculate spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to coordinate values of the reference pixel in the to-be-filtered pixel and the reference pixel set in the three-dimensional space And wherein the reference pixel set is in the same manner as the pixel to be filtered One frame of image and adjacent multi-frame image; The calculating module is further configured to calculate, according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set, a texture pixel value similarity between the pixel to be filtered and the reference pixel; The calculating module is further configured to calculate a time domain proximity of the pixel to be filtered and the reference pixel according to a time interval of a frame in which the reference pixel in the pixel to be filtered and the reference pixel in the reference pixel set are located; And a filtering module, configured to determine a weight of the filtering according to the spatial proximity, the texel value similarity, and the time domain proximity, and perform weighted averaging on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be-filtered The filtering result of the pixel. The device according to claim 19, wherein the filtering module is specifically configured to: When filtering the depth image, determining the weight of the filtering according to the spatial proximity, the texel value similarity corresponding to the depth pixel, and the time domain proximity, respectively, the depth of the reference pixel in the reference pixel set Performing a weighted average of the pixel values to obtain a filtering result of the depth pixel value of the pixel to be filtered of the depth image; or When filtering the texture image, determining the weight of the filtering according to the spatial proximity, the texture pixel similarity, and the time domain proximity, respectively performing weighted average on the texel values of the reference pixels in the reference pixel set And obtaining a filtering result of the texel value of the texture image to be filtered pixel. The device according to claim 19 or 20, wherein the projection module is specifically configured to: The pixels are projected from the image plane to the three-dimensional space using depth image information, viewpoint position information, and reference camera parameter information provided by the three-dimensional video; the depth image information includes depth pixel values of the pixels. The device according to claim 21, wherein the projection module is specifically configured to: Calculating coordinate values of the pixel after being projected into the three-dimensional space according to the formula P=R ^-1 (dA ^-1 pt); Where R and t are the rotation matrix and translation vector of the reference camera, and A is the reference camera parameter matrix.

For the coordinate values of the pixels in the image plane,

a coordinate value of the pixel in the three-dimensional space, d is a depth pixel value of the pixel; f _x and f _y are normalized focal lengths in horizontal and vertical directions, respectively, and r is a radial distortion coefficient, ( o _x , o _y ) is a coordinate value of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane. The apparatus according to any one of claims 19-22, wherein the spatial proximity is calculated by inputting a distance of the pixel to be filtered and the reference pixel in a three-dimensional space as a function; The output value increases as the input value decreases; The texture pixel value similarity is calculated by using a difference value of the texel value of the pixel to be filtered and the reference pixel as a function of an input value; an output value of the function increases as the input value decreases; The time domain proximity is calculated by using a time interval of the pixel to be filtered and a frame in which the reference pixel is located as a function of an input value; an output value of the function increases as the input value decreases. The device according to any one of claims 19 to 23, wherein the filtering module is specifically configured to: According to formula (3):

Calculating a filtering result of obtaining a depth pixel value of the pixel to be filtered; or According to formula (4):

Calculating a filtering result of obtaining a texel value of the pixel to be filtered; among them,

And configured to calculate spatial proximity of the pixel to be filtered and the reference pixel;

Means for calculating texture pixel value similarity between the pixel to be filtered and the reference pixel; f _tem (i, N)=f _tem (||iN||) for calculating a time domain proximity of the pixel to be filtered and the reference pixel; Where N is the frame number of the frame in which the pixel to be filtered is located, i is the frame number of the frame in which the reference pixel is located, i is an integer in the interval [Nm, N+n], and m and n are respectively before the frame in which the pixel to be filtered is located And the number of reference frames, m, n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the ith frame, K _i is the reference pixel set in the ith frame, and D _p ' is p Filtered depth pixel value,

The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the three-dimensional space in the i-th frame, T _p ,

They are p, the texel value of q in the i-th frame, and T _p ' is the p-filtered texel value.

Images