CN105046657A

CN105046657A - Image stretching distortion adaptive correction method

Info

Publication number: CN105046657A
Application number: CN201510354230.2A
Authority: CN
Inventors: 冯华君; 杨波
Original assignee: Zhejiang University ZJU
Current assignee: Zhejiang University ZJU
Priority date: 2015-06-23
Filing date: 2015-06-23
Publication date: 2015-11-11
Anticipated expiration: 2035-06-23
Also published as: CN105046657B

Abstract

The invention proposes an image stretching distortion self-adaptive correction method. First, the image is divided into uniform square grids and the input and output coordinates are established; secondly, the character area and the background area are determined through the face detection algorithm, and homography constraints are established for each area to correct the stretching distortion in the character area. And establish homography compatibility constraints for adjacent regions to ensure the continuity of each region; determine the straight region through the straight line detection algorithm, and establish straight line maintenance constraints for the straight line area to ensure that the corrected straight line does not appear curved; through global coordinate smoothing and Uniformity constraints ensure that the entire image is consistent and continuous. Then express the above constraints as energy functions, add weights, and add up the energy equations. The least square method is used to calculate the coordinate value when the energy is minimum, that is, the corrected image coordinate value is obtained. Finally, the bilinear map is used for rendering to obtain the final rectified image.

Description

An Adaptive Correction Method for Image Stretch Distortion

技术领域technical field

本发明涉及一种数字图像处理技术，尤其涉及一种图像拉伸畸变自适应校正方法。The invention relates to a digital image processing technology, in particular to an image stretching distortion adaptive correction method.

背景技术Background technique

照相机广角镜头能获取较大视场的图像，但也容易引入图像畸变。其中一种畸变是当广角镜头遵循针孔成像模型进行成像，并且成像场景中有物体靠近相机并处在视场边缘时，拍摄后的图像中该物体由于透视效应会被严重拉伸，造成该物体几何形状的失真。特别是该物体是人脸或人物时，由于人眼对人脸和人物图像的畸变较为敏感，该拉伸畸变更为明显。因此，需要对出现物体拉伸畸变的图像进行校正，使得图像符合人眼视觉。The wide-angle lens of the camera can obtain images with a larger field of view, but it is also easy to introduce image distortion. One of the distortions is that when the wide-angle lens follows the pinhole imaging model for imaging, and there is an object in the imaging scene that is close to the camera and at the edge of the field of view, the object in the captured image will be severely stretched due to the perspective effect, causing the object Distortion of geometric shapes. Especially when the object is a human face or a person, since the human eye is more sensitive to the distortion of the image of the human face or person, the stretching distortion becomes more obvious. Therefore, it is necessary to correct the image with stretching distortion of the object so that the image conforms to human vision.

然而，图像场景中的直线会因为校正了物体的拉伸而受到弯曲，因此，需要在在校正拉伸畸变同时使得其他场景或物体保持不受影响。However, the straight lines in the image scene will be bent due to the correction of the stretching of the object. Therefore, it is necessary to correct the stretching distortion while keeping other scenes or objects unaffected.

此外，全景图像本质上也是一种广角图像。当全景图像采用线性透视投影时，该图像也会出现拉伸畸变。In addition, a panoramic image is essentially a wide-angle image. When a panoramic image is projected from a linear perspective, the image will also appear stretched.

在现有的技术中有通过改变投影中心的全局投影校正方法，有通过将图像背景与人物相分离再分别校正的方法，有优化的保持图像内容的投影校正方法。In the existing technology, there is a global projection correction method by changing the projection center, there is a method by separating the image background from the person and correcting them separately, and there is an optimized projection correction method that maintains the image content.

发明内容Contents of the invention

本发明的目的在于针对现有技术的不足，提供一种图像拉伸畸变自适应校正方法，在保持图像中背景直线笔直的同时校正图像中物体尤其是人物出现的拉伸畸变。The object of the present invention is to address the deficiencies in the prior art and provide an adaptive correction method for image stretching distortion, which corrects the stretching distortion of objects in the image, especially characters, while keeping the background straight in the image.

本发明的目的是通过以下技术方案实现的：一种图像拉伸畸变自适应校正方法，所述方法包括以下步骤：The purpose of the present invention is achieved by the following technical solutions: a method for self-adaptive correction of image stretching distortion, said method comprising the following steps:

步骤A：输入带有物体拉伸畸变的图像。Step A: Input an image with object stretching distortion.

步骤B：建立网格及坐标，将带有物体拉伸畸变的图像划分为正方形均匀网格，网格交点坐标记为x_i,j＝(x_i,j,y_i,j)，i,j为网格交点序号；设网格交点坐标x_i,j校正后的坐标为u_i,j＝(u_i,j,v_i,j)。Step B: Establish the grid and coordinates, divide the image with object stretching distortion into a square uniform grid, and the grid intersection coordinates are marked as x _i,j = ( _xi,j ,y _i,j ), i, j is the serial number of the grid intersection point; suppose the corrected coordinates of the grid intersection point x _i,j are u _i,j =(u _i,j ,v _i,j ).

步骤C：构造u_i,j的约束能量函数。Step C: Construct the constrained energy function of u _i,j .

步骤C1：构造单应性约束能量函数。计算出将人物区域旋转至视场中心需要旋转的角度计算公式为：Step C1: Construct a homography-constrained energy function. Calculate the angle needed to rotate the character area to the center of the field of view The calculation formula is:

其中L为人物区域中心到视场中心的水平偏移距离，f为焦距，L和f均转化为以像素数为单位。则人物区域的单应性变换可写为：Among them, L is the horizontal offset distance from the center of the character area to the center of the field of view, f is the focal length, and both L and f are converted to the number of pixels. Then the homography transformation of the character area can be written as:

${u u}_{i i,, j j} = = f f ((\frac{U u}{W W})) + + L L$

${v v}_{i i,, j j} = = f f ((\frac{V V}{W W}))$

将上式描述的单应性变换记为H(x_i,j)，则可以定义人物区域网格点单应性约束的能量函数：Denote the homography transformation described by the above formula as H( _xi,j ), then the energy function of the homography constraint of grid points in the character area can be defined:

${E E.}_{h h 11} = = \underset{{x x}_{i i,, j j} &Element; &Element; {P P}_{11}}{Σ Σ} | | {u u}_{i i,, j j} - - H h (({x x}_{i i,, j j})) {| |}^{22}$

其中，P₁为人物区域网格点的集合。Among them, P ₁ is the set of grid points in the character area.

背景区域网格点的单应性约束能量函数定义为：The homography-constrained energy function of the grid points in the background region is defined as:

${E E.}_{h h 22} = = \underset{{x x}_{i i,, j j} &Element; &Element; {P P}_{22}}{Σ Σ} | | {u u}_{i i,, j j} - - {x x}_{i i,, j j} {| |}^{22}$

其中，P₂为背景区域网格点的集合。Among them, _P2 is the set of grid points in the background area.

总的单应性约束能量函数即为The total homography-constrained energy function is

${E E.}_{h h} = = {E E.}_{h h 11} + + {E E.}_{h h 22} = = \underset{{x x}_{i i,, j j} &Element; &Element; {P P}_{11}}{Σ Σ} | | {u u}_{i i,, j j} - - H h (({x x}_{i i,, j j})) {| |}^{22} + + \underset{{x x}_{i i,, j j} &Element; &Element; {P P}_{22}}{Σ Σ} | | {u u}_{i i,, j j} - - {x x}_{i i,, j j} {| |}^{22}$

步骤C2：构造单应性兼容约束能量函数。Step C2: Construct a homography compatible constrained energy function.

定义P_hc为人物区域和背景区域间共有边界的网格点，H₁、H₂分别为背景区域和人物区域的单应性变换。则单应性兼容约束的能量函数为：Define P _hc as the grid point sharing the boundary between the character area and the background area, and H ₁ , H ₂ are the homography transformations of the background area and the character area respectively. Then the energy function of the homography compatible constraint is:

${E E.}_{h h c c} = = \underset{{x x}_{i i,, j j} &Element; &Element; {P P}_{h h c c}}{Σ Σ} ((| | {u u}_{i i,, j j} - - {H h}_{11} (({x x}_{i i,, j j})) {| |}^{22} + + | | {u u}_{i i,, j j} - - {H h}_{22} (({x x}_{i i,, j j})) {| |}^{22}))$

步骤C3：构造直线保持约束能量函数。首先使用直线段检测算法检测图像中除人物之外背景区域中的直线段。将检测到的直线集合定义为L。Step C3: Construct a straight line preserving constraint energy function. Firstly, a straight line segment detection algorithm is used to detect the straight line segment in the background area of the image except the person. Define the set of detected lines as L.

在每一个有直线经过的网格中定义一个虚拟点 Define a virtual point in each grid through which a line passes

${u u}_{i i,, j j}^{L L} = = {au au}_{i i,, j j} + + {bu bu}_{i i + + 11,, j j} + + {cu cu}_{i i,, j j + + 11} + + {du du}_{i i + + 11,, j j + + 11}$

其中系数a,b,c,d由与u-v坐标相对应的x-y坐标值经双线性插值计算得到。计算公式为：The coefficients a, b, c, d are calculated by bilinear interpolation from the x-y coordinate values corresponding to the u-v coordinates. The calculation formula is:

$a a = = \frac{(({x x}_{i i + + 11,, j j} - - {x x}_{i i,, j j}^{L L})) (({y the y}_{i i,, j j + + 11} - - {y the y}_{i i,, j j}^{L L}))}{(({x x}_{i i + + 11,, j j} - - {x x}_{i i,, j j})) (({y the y}_{i i,, j j + + 11} - - {y the y}_{i i,, j j}))}$

$b b = = \frac{(({x x}_{i i,, j j}^{L L} - - {x x}_{i i,, j j})) (({y the y}_{i i,, j j + + 11} - - {y the y}_{i i,, j j}^{L L}))}{(({x x}_{i i + + 11,, j j} - - {x x}_{i i,, j j})) (({y the y}_{i i,, j j + + 11} - - {y the y}_{i i,, j j}))}$

$c c = = \frac{(({x x}_{i i,, j j}^{L L} - - {x x}_{i i,, j j})) (({y the y}_{i i,, j j}^{L L} - - {y the y}_{i i,, j j}))}{(({x x}_{i i + + 11,, j j} - - {x x}_{i i,, j j})) (({y the y}_{i i,, j j + + 11} - - {y the y}_{i i,, j j}))}$

$d d = = \frac{(({x x}_{i i + + 11,, j j} - - {x x}_{i i,, j j}^{L L})) (({y the y}_{i i,, j j}^{L L} - - {y the y}_{i i,, j j}))}{(({x x}_{i i + + 11,, j j} - - {x x}_{i i,, j j})) (({y the y}_{i i,, j j + + 11} - - {y the y}_{i i,, j j}))}$

其中，取为网格块内线段的中点。in, Taken as the midpoint of the line segment within the grid block.

对于任意一条直线，定义其起始点和终点分别为u_start和u_end。For any straight line, define its starting point and end point as u _start and u _end respectively.

计算曲线上的点u_i,j与该点到直线上投影点之间的距离D，并将D约束为零。由几何关系可以得到，该距离D的计算式为Calculate the distance D between the point u _i,j on the curve and the projected point on the straight line, and constrain D to be zero. It can be obtained from the geometric relationship that the calculation formula of the distance D is

D＝|(u_i,j-u_start)^Tn|D＝|(u _i,j -u _start ) ^T n|

$n no = = [\begin{matrix} 00 & - - 11 \\ 11 & 00 \end{matrix}] \frac{(({u u}_{e e n no d d} - - {u u}_{s the s t t a a r r t t}))}{| | | | {u u}_{e e n no d d} - - {u u}_{s the s t t a a r r t t} | | | |}$

其中，n＝(n₁,n₂)为直线的单位法向量。计算出距离D，则可定义直线保持约束的能量函数为Wherein, n=(n ₁ , n ₂ ) is the unit normal vector of the straight line. After calculating the distance D, the energy function of the line-keeping constraint can be defined as

${E E.}_{l l} = = \underset{{x x}_{i i,, j j} &Element; &Element; L L}{Σ Σ} {(({(({u u}_{i i,, j j} - - {u u}_{s the s t t a a r r t t}))}^{T T} n no))}^{22} = = \underset{{x x}_{i i,, j j} &Element; &Element; L L}{Σ Σ} {(({n no}_{11} (({u u}_{i i,, j j} - - {u u}_{s the s t t a a r r t t})) + + {n no}_{22} (({v v}_{i i,, j j} - - {v v}_{s the s t t a a r r t t}))))}^{22};;$

步骤C4：构造全局坐标平滑约束能量函数。定义u-v坐标的雅可比矩阵为Step C4: Construct a global coordinate smoothing constrained energy function. The Jacobian matrix defining the u-v coordinates is

$J J = = \frac{\partial \partial u u}{\partial \partial x x} = = [\begin{matrix} \frac{\partial \partial u u}{\partial \partial x x} & \frac{\partial \partial u u}{\partial \partial y the y} \\ \frac{\partial \partial v v}{\partial \partial x x} & \frac{\partial \partial v v}{\partial \partial y the y} \end{matrix}];;$

当雅可比矩阵中的各项满足柯西-黎曼方程：When the terms in the Jacobian matrix satisfy the Cauchy-Riemann equation:

$\begin{matrix} \frac{\partial u}{\partial x} = \frac{\partial v}{\partial y} \\ \frac{\partial u}{\partial y} = - \frac{\partial v}{\partial x} \end{matrix}$ 时， $\begin{matrix} \frac{\partial u}{\partial x} = \frac{\partial v}{\partial the y} \\ \frac{\partial u}{\partial the y} = - \frac{\partial v}{\partial x} \end{matrix}$ hour,

可实现坐标的连续平滑。将上式离散化得：Continuous smoothing of coordinates is possible. Discretize the above formula to get:

u_i+1,j-u_i,j＝v_i,j+1-v_i,j u _i+1,j -u _i,j =v _i,j+1 -v _i,j

u_i,j+1-u_i,j＝-(v_i+1,j-v_i,j)；u _i,j+1 -u _i,j =-(v _i+1,j -v _i,j );

通过上式可计算坐标平滑约束能量函数：The coordinate smoothing constraint energy function can be calculated by the above formula:

E_s＝Σ((u_i+1,j-u_i,j)-(v_i,j+1-v_i,j))²+Σ((v_i+1,j-v_i,j)+(u_i,j+1-u_i,j))²；E _s ＝Σ((u _i+1,j -u _i,j )-(v _i,j+1 -v _i,j )) ² +Σ((v _i+1,j -v _i,j ) +(u _i,j+1 -u _i,j )) ² ;

步骤C5：构造全局坐标均匀约束能量函数。将拉普拉斯算子离散化，可定义全局坐标均匀约束的能量函数为Step C5: Construct a global coordinate uniformly constrained energy function. By discretizing the Laplacian operator, the energy function of the global coordinate uniform constraint can be defined as

${E E.}_{u u} = = Σ Σ | | | | [\begin{matrix} {u u}_{i i + + 11,, j j} + + {u u}_{i i - - 11,, j j} + + {u u}_{i i,, j j +1 +1} + + {u u}_{i i,, j j -1 -1} {-4u -4u}_{i i,, j j} \\ {v v}_{i i + + 11,, j j} {+v +v}_{i i - - 11,, j j} {+v +v}_{i i,, j j +1 +1} {+v +v}_{i i,, j j -1 -1} {-4v -4v}_{i i,, j j} \end{matrix}] | | {| |}^{22};;$

步骤D：线性最小二乘求解。为每一个能量函数添加权重系数并将各能量函数相加得到总能量函数为：Step D: Linear least squares solution. Add weight coefficients for each energy function and add up the energy functions to get the total energy function as:

$E E. = = {w w}_{h h}^{22} {E E.}_{h h} + + {w w}_{h h c c}^{22} {E E.}_{h h c c} + + {w w}_{l l}^{22} {E E.}_{l l} + + {w w}_{s the s}^{22} {E E.}_{s the s} + + {w w}_{u u}^{22} {E E.}_{u u}$

使E＝0，则E_h,E_hc,E_l,E_s,E_u均需为0，由此可以构造出线性方程组Make E=0, then E _h , E _hc , E _l , E _s , E _u must all be 0, and thus a linear equation system can be constructed

Ax＝bAx=b

其中A为约束矩阵，包含约束能量函数中的u_i,j前的各个系数。x为包含待求未知量u_i,j和v_i,j的列向量，b包含各个方程的常数项。Wherein A is the constraint matrix, including the coefficients before u _{i, j} in the constraint energy function. x is a column vector containing the unknown quantities u _{i, j} and v _{i, j} , and b contains the constant terms of each equation.

用线性最小二乘法对x进行求解，求解公式为Use the linear least squares method to solve x, and the solution formula is

x＝(A^TA)^-1A^Tbx＝(A ^T A) ^-1 A ^T b

步骤E：求解出网格点坐标u_i,j和v_i,j后，采用双线性映射计算出除网格点之外任意点(x,y)校正后的坐标为Step E: After solving the grid point coordinates u _{i, j} and v _{i, j} , use bilinear mapping to calculate the corrected coordinates of any point (x, y) other than the grid point as

(u,v)＝(1-x)(1-y)p1+x(1-y)p2+(1-x)yp3+xyp4。(u,v)=(1-x)(1-y)p1+x(1-y)p2+(1-x)yp3+xyp4.

其中，p1,p2,p3,p4分别为点(x,y)所在网格的4个网格点坐标，p1为左上角网格点坐标，p2为右上角网格点坐标，p3为左下角网格点坐标，p4为右下角网格点坐标。Among them, p1, p2, p3, and p4 are the four grid point coordinates of the grid where the point (x, y) is located, p1 is the grid point coordinates of the upper left corner, p2 is the grid point coordinates of the upper right corner, and p3 is the lower left corner Grid point coordinates, p4 is the grid point coordinates in the lower right corner.

步骤F：计算出校正坐标(u,v)后。通过从输入坐标值(x,y)到计算得到的坐标值(u,v)的重映射和插值，得到校正图像。Step F: After calculating the corrected coordinates (u, v). The rectified image is obtained by remapping and interpolating from input coordinate values (x, y) to calculated coordinate values (u, v).

本发明公开的方法相比现有的校正方法有以下几个特点：(1)既能校正视场边缘处物体的拉伸畸变，又能保持背景直线不受弯曲；(2)校正结果图像内容连续一致，无切割想象出现；(3)校正后图像的视场损失较小；(4)校正方法自动完成，无需人工干预。Compared with the existing correction methods, the method disclosed in the present invention has the following characteristics: (1) It can not only correct the stretching distortion of the object at the edge of the field of view, but also keep the background straight line from bending; (2) Correct the image content of the result Continuous and consistent, no cutting image appears; (3) The field of view loss of the corrected image is small; (4) The correction method is automatically completed without manual intervention.

附图说明Description of drawings

图1是本发明一种图像拉伸畸变校正方法的流程图；Fig. 1 is a flow chart of an image stretching distortion correction method of the present invention;

图2是带有本发明所描述的拉伸畸变的示例图像；Figure 2 is an example image with stretch distortion as described in the present invention;

图3是网格的建立和坐标的定义示意图；Fig. 3 is the establishment of grid and the definition schematic diagram of coordinate;

图4是图像区域的划分示意图；FIG. 4 is a schematic diagram of division of image regions;

图5是虚拟点的定义示意图；Fig. 5 is the definition sketch map of virtual point;

图6是距离D的计算示意图；Fig. 6 is the calculation schematic diagram of distance D;

图7是双线性映射示意图；Fig. 7 is a schematic diagram of bilinear mapping;

图8是经本发明方法校正后的示例图像；Fig. 8 is an example image corrected by the method of the present invention;

图9是本发明方法校正后的示例图像剪裁为矩形后的图像。FIG. 9 is an example image corrected by the method of the present invention and clipped into a rectangular image.

具体实施方式Detailed ways

本发明所述的图像拉伸畸变自适应校正方法包括接受带有物体拉伸畸变的图像；用均匀网格对图像进行划分并建立输入输出坐标系；将输入图像通过人脸检测算法检测人脸位置和大小，并通过该信息确定出人物所处的矩形区域和余下的背景区域；分别为各个区域建立单应性约束，以校正人物区域中出现的拉伸畸变，并为相邻区域建立单应性兼容约束，保证各个区域的连续性；通过直线检测算法确定直线区域，为直线区域建立直线保持约束，以保证校正后的直线不出现弯曲；建立全局坐标平滑和均匀性约束保证整幅图像一致连续；将上述约束表述为能量函数，并加入权重，将各个能量方程相加；将能量函数最小化以构造出线性方程组，使用最小二乘法计算该线性方程组，得到校正后的图像坐标值；使用双线性纹理映射进行渲染得到最终图像。The self-adaptive correction method for image stretching distortion of the present invention includes accepting an image with stretching distortion of an object; dividing the image with a uniform grid and establishing an input and output coordinate system; detecting the face of the input image through a face detection algorithm Position and size, and use this information to determine the rectangular area where the character is located and the remaining background area; establish homography constraints for each area to correct the stretching distortion in the character area, and establish a homography for the adjacent area Responsive compatibility constraints to ensure the continuity of each area; determine the straight line area through the straight line detection algorithm, and establish a straight line maintenance constraint for the straight line area to ensure that the corrected straight line does not appear curved; establish global coordinate smoothness and uniformity constraints to ensure the entire image Consistent and continuous; express the above constraints as energy functions, add weights, and add each energy equation; minimize the energy function to construct a linear equation system, use the least square method to calculate the linear equation system, and obtain the corrected image coordinates Value; the final image is rendered using bilinear texture mapping.

为更好地理解本发明的实现过程，下面将结合附图详细阐述本发明的一种图像拉伸畸变自适应校正方法。具体实施步骤的流程图参见图1，应用本发明的方法实现图像拉伸畸变校正的具体步骤如下：In order to better understand the implementation process of the present invention, an image stretching distortion adaptive correction method of the present invention will be described in detail below in conjunction with the accompanying drawings. Referring to Fig. 1 for the flow chart of specific implementation steps, the specific steps of applying the method of the present invention to realize image stretching distortion correction are as follows:

步骤A：输入带有物体拉伸畸变的图像。图2是带有人物拉伸畸变的图像的一个示例，畸变位置用双箭头标明。Step A: Input an image with object stretching distortion. Figure 2 is an example of an image with stretching distortion of a person, and the location of the distortion is marked with a double arrow.

步骤B：建立网格及坐标。以图2为例，如图3所示，将图像划分为正方形均匀网格，网格交点坐标记为x_i,j＝(x_i,j,y_i,j)，i,j为网格交点序号。经图像扭曲后，待求解的坐标记为u_i,j＝(u_i,j,v_i,j)。Step B: Establish grid and coordinates. Taking Figure 2 as an example, as shown in Figure 3, the image is divided into square uniform grids, and the grid intersection coordinates are marked as x _i,j = ( _xi,j ,y _i,j ), where i,j is the grid Intersection number. After the image is warped, the coordinates to be solved are marked as u _i,j =(u _i,j ,v _i,j ).

步骤C：构造约束能量函数。Step C: Construct the constrained energy function.

步骤C1：构造单应性约束能量函数。出现拉伸畸变的原因是物体处在图像视场边缘，当旋转照相机使之正对物体并使物体处在视场中央拍摄时，则物体无拉伸畸变。因此，需要通过单应性变换(透视变换)改变图像局部区域的视角，以校正该区域中物体的拉伸畸变。首先，使用图像人脸检测算法检测人脸位置和大小，通过该信息确定出人物所处的矩形区域和余下的背景区域，以图2为例确定出的人物区域和背景区域如图4所示。计算出将人物区域旋转至视场中心需要旋转的角度计算公式为：Step C1: Construct a homography-constrained energy function. The reason for stretching distortion is that the object is at the edge of the image field of view. When the camera is rotated to face the object and the object is shot in the center of the field of view, the object has no stretching distortion. Therefore, it is necessary to change the viewing angle of a local area of the image through homography transformation (perspective transformation) to correct the stretching distortion of objects in this area. First, use the image face detection algorithm to detect the position and size of the face, and use this information to determine the rectangular area where the person is located and the remaining background area. Taking Figure 2 as an example, the figure area and background area determined are shown in Figure 4 . Calculate the angle needed to rotate the character area to the center of the field of view The calculation formula is:

${u u}_{i i,, j j} = = f f ((\frac{U u}{W W})) + + L L$

${v v}_{i i,, j j} = = f f ((\frac{V V}{W W}))$

其中P₁为人物区域网格点的集合。背景区域不需单应性变换，因此该区域网格点的单应性约束能量函数定义为：Among them, P ₁ is the set of grid points in the character area. The background area does not require homography transformation, so the homography constraint energy function of the grid points in this area is defined as:

其中P₂为背景区域网格点的集合。总的单应性约束能量函数即为where _P2 is the set of grid points in the background area. The total homography-constrained energy function is

步骤C2：构造单应性兼容约束能量函数。人物区域和背景区域有公共的边。为了防止各个区域经过单应性变换之后造成区域相接部分出现错位现象，必须在区域的邻接部分加入单应性兼容约束。定义P_hc为区域间共有边界的网格点，H₁、H₂分别为人物和背景区域的单应性变换。则单应性兼容约束的能量函数为：Step C2: Construct a homography compatible constrained energy function. The character region and the background region have common edges. In order to prevent the misalignment of the adjacent parts of the regions after each region undergoes homography transformation, it is necessary to add homography compatibility constraints to the adjacent parts of the regions. Define P _hc as the grid point with a common boundary between regions, and H ₁ and H ₂ are the homography transformations of the character and background regions respectively. Then the energy function of the homography compatible constraint is:

所述单应性变换H₁为：The homography transformation H ₁ is:

${u u}_{i i,, j j} = = f f ((\frac{U u}{W W})) + + L L$

${v v}_{i i,, j j} = = f f ((\frac{V V}{W W}))$

所述单应性变换H₂为3×3的单位矩阵。The homography transformation H ₂ is a 3×3 identity matrix.

步骤C3：构造直线保持约束能量函数。保证图像中直线经校正后仍然保持笔直，需要加入直线保持约束。首先使用直线段检测算法检测图像中除人物之外背景区域的直线段。将检测到的直线集合定义为L。Step C3: Construct a straight line preserving constraint energy function. To ensure that the straight lines in the image remain straight after correction, it is necessary to add straight line maintenance constraints. Firstly, the straight line segment detection algorithm is used to detect the straight line segment in the background area of the image except the person. Define the set of detected lines as L.

由于并不是任何一条直线都经过网格点，因此如图5所示，在每一个有直线经过的网格中定义一个虚拟点 Since not any straight line passes through the grid points, as shown in Figure 5, a virtual point is defined in each grid where a straight line passes through

对于任意一条直线，定义其起始点和终点分别为u_start和u_end。如图6所示，为了将曲线拉直为直线，需要计算曲线上的点u_i,j与该点到直线上投影点之间的距离D，并将D约束为零。由几何关系可以得到，该距离D的计算式为For any straight line, define its starting point and end point as u _start and u _end respectively. As shown in Figure 6, in order to straighten the curve into a straight line, it is necessary to calculate the distance D between the point u _i,j on the curve and the projected point on the straight line, and constrain D to be zero. It can be obtained from the geometric relationship that the calculation formula of the distance D is

D＝|(u_i,j-u_start)^Tn|D＝|(u _i,j -u _start ) ^T n|

${E E.}_{l l} = = \underset{{x x}_{i i,, j j} &Element; &Element; L L}{Σ Σ} {(({(({u u}_{i i,, j j} - - {u u}_{s the s t t a a n no}))}^{T T} n no))}^{22} = = {\underset{{x x}_{i i,, j j} &Element; &Element; L L}{Σ Σ} (({n no}_{11} (({u u}_{i i,, j j} - - {u u}_{s the s t t a a r r t t})) + + {n no}_{22} (({ν ν}_{i i,, j j} - - {v v}_{s the s t t a a r r t t}))))}^{22}$

步骤C4：构造全局坐标平滑约束能量函数。要使图像内容能够一致连续，需定义全局坐标平滑约束，定义u-v坐标的雅可比矩阵为Step C4: Construct a global coordinate smoothing constrained energy function. In order to make the image content consistent and continuous, it is necessary to define the global coordinate smoothing constraints, and define the Jacobian matrix of u-v coordinates as

$J J = = \frac{\partial \partial u u}{\partial \partial x x} = = [\begin{matrix} \frac{\partial \partial u u}{\partial \partial x x} & \frac{\partial \partial u u}{\partial \partial y the y} \\ \frac{\partial \partial v v}{\partial \partial x x} & \frac{\partial \partial v v}{\partial \partial y the y} \end{matrix}]$

当雅可比矩阵中的各项满足柯西-黎曼方程：When the terms in the Jacobian satisfy the Cauchy-Riemann equation:

$\frac{\partial \partial u u}{\partial \partial x x} = = \frac{\partial \partial v v}{\partial \partial y the y}$

$\frac{\partial \partial u u}{\partial \partial y the y} = = - - \frac{\partial \partial v v}{\partial \partial x x}$

时，可实现坐标的连续平滑。由于网格是离散的，将上式离散化得When , the continuous smoothing of coordinates can be realized. Since the grid is discrete, the above formula is discretized to get

u_i+1,j-u_i,j＝v_i,j+1-v_i,j u _i+1,j -u _i,j =v _i,j+1 -v _i,j

u_i,j+1-u_i,j＝-(v_i+1,j-v_i,j)u _i,j+1 -u _i,j ＝-(v _i+1,j -v _i,j )

E_s＝Σ((u_i+1,j-u_i,j)-(v_i,j+1-v_i,j))²+Σ((v_i+1,j-v_i,j)+(u_i,j+1-u_i,j))² E _s ＝Σ((u _i+1,j -u _i,j )-(v _i,j+1 -v _i,j )) ² +Σ((v _i+1,j -v _i,j ) +(u _i,j+1 -u _i,j )) ²

步骤C5：构造全局坐标均匀约束能量函数。仅考虑坐标平滑约束会使得最终校正图像外围网格比图像中部网格大很多。因此，有必要加入全局坐标均匀约束，使得图像内外的网格比例保持一致。拉普拉斯方程的解具有均匀性质，因此这里采用拉普拉斯算子。将拉普拉斯算子离散化，可定义坐标均匀约束的能量函数为Step C5: Construct a global coordinate uniformly constrained energy function. Only considering coordinate smoothing constraints will make the peripheral grid of the final corrected image much larger than the central grid of the image. Therefore, it is necessary to add global coordinate uniform constraints to keep the grid ratio inside and outside the image consistent. The solution of the Laplace equation has a uniform property, so the Laplace operator is used here. By discretizing the Laplacian operator, the energy function of coordinate uniform constraints can be defined as

${E E.}_{u u} = = Σ Σ | | | | [\begin{matrix} {u u}_{i i + + 11,, j j} + + {u u}_{i i - - 11,, j j} + + {u u}_{i i,, j j + + 11} + + {u u}_{i i,, j j - - 11} - - 44 {u u}_{i i,, j j} \\ {v v}_{i i + + 11,, j j} + + {v v}_{i i - - 11,, j j} + + {v v}_{i i,, j j + + 11} + + {v v}_{i i,, j j - - 11} - - {vu v u}_{i i,, j j} \end{matrix}] | | {| |}^{22}$

要使能量函数最小，需使E＝0。E＝0，则E_h,E_hc,E_l,E_s,E_u均需为0，由此可以构造出线性方程组To minimize the energy function, it is necessary to make E=0. E=0, then E _h , E _hc , E _l , E _s , E _u must all be 0, from which a system of linear equations can be constructed

Ax＝bAx=b

其中A为约束矩阵，包含约束能量函数中的u_i,j前的各个系数。_X为包含待求未知量u_i,j和v_i,j的列向量，b包含各个方程的常数项。由于该线性方程组是超定的，可用线性最小二乘法对_X进行求解，求解公式为Wherein A is the constraint matrix, including the coefficients before u _{i, j} in the constraint energy function. _X is a column vector containing the unknown quantities u _{i, j} and v _{i, j} , and b contains the constant terms of each equation. Since the system of linear equations is overdetermined, the linear least square method can be used to solve _X , and the solution formula is

x＝(A^TA)^-1A^Tbx＝(A ^T A) ^-1 A ^T b

步骤E：求解出网格点坐标u_i,j和v_i,j后，需采用双线性映射计算出除网格点之外的其余坐标。如图7所示，已知网格点p1,p2,p3,p4坐标，网格内任意点(u,v)的计算式为Step E: After solving the coordinates u _i,j and v _i,j of the grid points, it is necessary to use bilinear mapping to calculate other coordinates except the grid points. As shown in Figure 7, the coordinates of grid points p1, p2, p3, and p4 are known, and the calculation formula of any point (u, v) in the grid is

(u,v)＝(1-x)(1-y)p1+x(1-y)p2+(1-x)yp3+xyp4(u,v)=(1-x)(1-y)p1+x(1-y)p2+(1-x)yp3+xyp4

步骤F：计算出校正坐标(u,v)后。经过从输入坐标值到计算得到的坐标值的重映射和插值，即可得到最终校正图像，如图8所示。校正图像边界不规则，需剪裁为矩形后输出，如图9所示。Step F: After calculating the corrected coordinates (u, v). After remapping and interpolation from the input coordinate value to the calculated coordinate value, the final corrected image can be obtained, as shown in FIG. 8 . The corrected image has irregular borders and needs to be cropped into a rectangle for output, as shown in Figure 9.

Claims

1. An adaptive correction method for image stretching distortion, characterized in that the method comprises the following steps:

step A: an image with object stretch distortion is input.

And B: establishing grids and coordinates, dividing an image with object stretching distortion into square uniform grids, and recording the coordinates of grid intersection points as x_i,j＝(x_i,j,y_i,j) I, j are grid intersection point serial numbers; let grid intersection point coordinate x_i,jThe corrected coordinate is u_i,j＝(u_i,j,v_i,j)。

And C: structure u_i,jThe constraint energy function of (2).

Step C1: a homography constraining energy function is constructed. Calculating the angle of the figure region to the center of the visual fieldThe calculation formula is as follows:

wherein L is the horizontal offset distance from the center of the human area to the center of the field of view, f is the focal length, and L and f are converted into units of pixel numbers. The homography transformation of the character area can be written as:

u_{i, j} = f (\frac{U}{W}) + L

v_{i, j} = f (\frac{V}{W})

the homography described by the above equation is denoted as H (x)_i,j) Then an energy function of the homography constraint for the person region grid points may be defined:

<math> <mrow> <msub> <mi>E</mi> <mrow> <mi>h</mi> <mn>1</mn> </mrow> </msub> <mo>=</mo> <munder> <mo>Σ</mo> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <msub> <mi>P</mi> <mn>1</mn> </msub> </mrow> </munder> <msup> <mrow> <mo>|</mo> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <mi>H</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>|</mo> </mrow> <mn>2</mn> </msup> </mrow> </math>

wherein, P₁A set of human area grid points.

The homography constraint energy function of the background region grid points is defined as:

<math> <mrow> <msub> <mi>E</mi> <mrow> <mi>h</mi> <mn>2</mn> </mrow> </msub> <mo>=</mo> <munder> <mo>Σ</mo> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <msub> <mi>P</mi> <mn>2</mn> </msub> </mrow> </munder> <mo>|</mo> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <msup> <mo>|</mo> <mn>2</mn> </msup> </mrow> </math>

wherein, P₂Is a set of background area grid points.

The total homography constraint energy function is

<math> <mrow> <msub> <mi>E</mi> <mi>h</mi> </msub> <mo>=</mo> <msub> <mi>E</mi> <mrow> <mi>h</mi> <mn>1</mn> </mrow> </msub> <mo>+</mo> <msub> <mi>E</mi> <mrow> <mi>h</mi> <mn>2</mn> </mrow> </msub> <mo>=</mo> <munder> <mo>Σ</mo> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <msub> <mi>P</mi> <mn>1</mn> </msub> </mrow> </munder> <mo>|</mo> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <mi>H</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>)</mo> </mrow> <msup> <mo>|</mo> <mn>2</mn> </msup> <mo>+</mo> <munder> <mo>Σ</mo> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <msub> <mi>P</mi> <mn>2</mn> </msub> </mrow> </munder> <mo>|</mo> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <msup> <mo>|</mo> <mn>2</mn> </msup> </mrow> </math>

Step C2: a homography compatibility constraint energy function is constructed.

Definition P_hcGrid points for common boundaries between human and background regions, H₁、H₂Homography transformations of the background region and the person region, respectively. The energy function of the homography compatibility constraint is then:

<math> <mrow> <msub> <mi>E</mi> <mrow> <mi>h</mi> <mi>c</mi> </mrow> </msub> <mo>=</mo> <munder> <mi>Σ</mi> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <msub> <mi>P</mi> <mrow> <mi>h</mi> <mi>c</mi> </mrow> </msub> </mrow> </munder> <mrow> <mo>(</mo> <mrow> <msup> <mrow> <mo>|</mo> <mrow> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>H</mi> <mn>1</mn> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>)</mo> </mrow> </mrow> <mo>|</mo> </mrow> <mn>2</mn> </msup> <mo>+</mo> <msup> <mrow> <mo>|</mo> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>H</mi> <mn>2</mn> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>|</mo> </mrow> <mn>2</mn> </msup> </mrow> <mo>)</mo> </mrow> </mrow> </math>

step C3: constructing a straight line preserves the constraint energy function. First, a straight-line segment detection algorithm is used to detect straight-line segments in a background region of the image except for a human figure. The set of detected straight lines is defined as L.

Defining a virtual point in each grid with straight lines passing through

u_{i, j}^{L} = {au}_{i, j} + {bu}_{i + 1, j} + {cu}_{i, j + 1} + {du}_{i + 1, j + 1}

The coefficients a, b, c and d are obtained by bilinear interpolation calculation of x-y coordinate values corresponding to u-v coordinates. The calculation formula is as follows:

a = \frac{(x_{i + 1, j} - x_{i, j}^{L}) (y_{i, j + 1} - y_{i, j}^{L})}{(x_{i + 1, j} - x_{i, j}) (y_{i, j + 1} - y_{i, j})}

b = \frac{(x_{i, j}^{L} - x_{i, j}) (y_{i, j + 1} - y_{i, j}^{L})}{(x_{i + 1, j} - x_{i, j}) (y_{i, j + 1} - y_{i, j})}

c = \frac{(x_{i, j}^{L} - x_{i, j}) (y_{i, j}^{L} - y_{i, j})}{(x_{i + 1, j} - x_{i, j}) (y_{i, j + 1} - y_{i, j})}

d = \frac{(x_{i + 1, j} - x_{i, j}^{L}) (y_{i, j}^{L} - y_{i, j})}{(x_{i + 1, j} - x_{i, j}) (y_{i, j + 1} - y_{i, j})}

wherein,taken as the midpoint of the line segment within the grid block.

For any straight line, defining the starting point and the end point of the straight line as u_startAnd u_end。

Calculating the point u on the curve_i,jThe distance D from the point to the projected point on the straight line is about DThe beam is zero. The distance D is calculated from the geometric relationship

D＝|(u_i,j-u_start)^Tn|

n = [\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix}] \frac{(u_{e n d} - u_{s t a r t})}{|| u_{e n d} - u_{s t a r t} ||}

Wherein n ═ n (n)₁,n₂) Is the unit normal vector of the straight line. The distance D is calculated as an energy function which defines a line hold constraint as

<math> <mrow> <msub> <mi>E</mi> <mi>l</mi> </msub> <mo>=</mo> <munder> <mo>Σ</mo> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <mi>L</mi> </mrow> </munder> <msup> <mrow> <mo>(</mo> <mrow> <msup> <mrow> <mo>(</mo> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>u</mi> <mrow> <mi>s</mi> <mi>t</mi> <mi>a</mi> <mi>r</mi> <mi>t</mi> </mrow> </msub> <mo>)</mo> </mrow> <mi>T</mi> </msup> <mi>n</mi> </mrow> <mo>)</mo> </mrow> <mn>2</mn> </msup> <mo>=</mo> <munder> <mi>Σ</mi> <mrow> <msub> <mi>x</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>&Element;</mo> <mi>L</mi> </mrow> </munder> <msup> <mrow> <mo>(</mo> <msub> <mi>n</mi> <mn>1</mn> </msub> <mo>(</mo> <mrow> <msub> <mi>u</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>u</mi> <mrow> <mi>s</mi> <mi>t</mi> <mi>a</mi> <mi>r</mi> <mi>t</mi> </mrow> </msub> </mrow> <mo>)</mo> <mo>+</mo> <msub> <mi>n</mi> <mn>2</mn> </msub> <mrow> <mo>(</mo> <msub> <mi>v</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mi>v</mi> <mrow> <mi>s</mi> <mi>t</mi> <mi>a</mi> <mi>r</mi> <mi>t</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>)</mo> </mrow> <mn>2</mn> </msup> <mo>;</mo> </mrow> </math>

Step C4: a global coordinate smoothing constraint energy function is constructed. Defining a Jacobian matrix of u-v coordinates as

When the terms in the jacobian matrix satisfy the cauchy-riemann equation:

when the temperature of the water is higher than the set temperature,

continuous smoothing of the coordinates can be achieved. Discretizing the above formula to obtain:

\begin{matrix} u_{i + 1, j} - u_{i, j} = v_{i, j + 1} - v_{i, j} \\ u_{i, j + 1} - u_{i, j} = - (v_{i + 1, j} - v_{i, j}) \end{matrix};

the coordinate smoothing constraint energy function can be calculated by the above equation:

E_s＝Σ((u_i+1,j-u_i,j)-(v_i,j+1-v_i,j))²+Σ((v_i+1,j-v_i,j)+(u_i,j+1-u_i,j))²；

step C5: and constructing a global coordinate uniform constraint energy function. Discretizing the Laplace operator, an energy function defining a uniform constraint on global coordinates can be defined as

Step D: and (5) solving linear least squares. Adding a weight coefficient to each energy function and adding the energy functions to obtain a total energy function as follows:

E = w_{h}^{2} E_{h} + w_{h c}^{2} E_{h c} + w_{l}^{2} E_{l} + w_{s}^{2} E_{s} + w_{u}^{2} E_{u}

if E is equal to 0, then E_h,E_hc,E_l,E_s,E_uAre all required to be 0, so that a linear equation system can be constructed

Ax＝b

Wherein A is a constraint matrix containing u in a constraint energy function_i,jThe previous coefficients. x is a value containing the unknown quantity u to be determined_i,jAnd v_i,jB contains constant terms of the respective equations.

Solving x by linear least square method to obtain the formula

x＝(A^TA)^-1A^Tb

Step E: solving the coordinates u of the grid points_i,jAnd v_i,jThen, the bilinear mapping is adopted to calculate the coordinates of any point (x, y) except the grid point after correction into

(u,v)＝(1-x)(1-y)p1+x(1-y)p2+(1-x)yp3+xyp4。

Wherein p1, p2, p3 and p4 are respectively coordinates of 4 grid points of a grid where the point (x, y) is located, p1 is coordinates of a grid point at the upper left corner, p2 is coordinates of a grid point at the upper right corner, p3 is coordinates of a grid point at the lower left corner, and p4 is coordinates of a grid point at the lower right corner.

Step F: after the corrected coordinates (u, v) are calculated. The corrected image is obtained by remapping and interpolating from the input coordinate values (x, y) to the calculated coordinate values (u, v).