Image Stitching by Line-guided Local Warping with Global Similarity Constraint
Tian-Zhu Xiang Gui-Song Xia Xiang Bai Liangpei Zhang
































































Image Stitching by Line-guided Local Warping
with Global Similarity Constraint
    Tianzhu Xiang^1, Gui-Song Xia^1, Xiang Bai^2, Liangpei Zhang^1



^1State Key Lab. LIESMARS, Wuhan University, Wuhan, China.

^2Electronic Information School, Huazhong University of Science and Technology, China.

    
=========================================================================================================================================================================================================================


Low-textured image stitching remains a challenging problem. It is difficult to achieve good alignment and it is easy to break image structures due to insufficient and unreliable point correspondences. Moreover, because of the viewpoint variations between multiple images, the stitched images suffer from projective distortions. To solve these problems, this paper presents a line-guided local warping method with a global similarity constraint for image stitching. Line features which serve well for geometric descriptions and scene constraints, are employed to guide image stitching accurately. On one hand, the line features are integrated into a local warping model through a designed weight function. On the other hand, line features are adopted to impose strong geometric constraints, including line correspondence and line colinearity, to improve the stitching performance through mesh optimization. To mitigate projective distortions, we adopt a global similarity constraint, which is integrated with the projective warps via a designed weight strategy. This constraint causes the final warp to slowly change from a projective to a similarity transformation across the image. Finally, the images undergo a two-stage alignment scheme that provides accurate alignment and reduces projective distortion. We evaluate our method on a series of images and compare it with several other methods. The experimental results demonstrate that the proposed method provides a convincing stitching performance and that it outperforms other state-of-the-art methods.








§ INTRODUCTION



Because images are limited by a camera's narrow field of view (FOV), image stitching combines a group of images with overlapping regions to generate a single, but larger, mosaic with a wider FOV. Image stitching has been widely used in many tasks in photogrammetry <cit.>, remote sensing <cit.> and computer vision <cit.>. 

In the literature <cit.>, there are typically two main approaches that have been attempted to produce image stitching with satisfactory visual results: (1) developing better alignment models and (2) employing image composition algorithms, such as seam cutting <cit.> and blending <cit.>. Image alignment is the first and  most crucial step in image stitching. Although advanced image composition methods can reduce stitching artifacts and improve the stitching performance, they cannot address obvious misalignments. When a seam or blending area coincides with misaligned areas, the current image composition schemes will fail to provide a satisfactory stitched image <cit.>. 

Most previous image stitching methods estimate global geometric transformations (e.g., similarity, affine or projective transformation) to bring the overlapping images into alignment. However, these methods require the camera rotation to have a fixed projective center or the scenes to have limited depth variance <cit.>, which are restrictive assumptions that are often violated in practice, resulting in artifacts in the stitched images, e.g., misalignments or ghosting.

To compensate for these geometric assumptions, some spatially-varying warping methods for image stitching have been proposed in recent years that can be roughly categorized into two groups: multiple homographies and mesh-based warping. The former estimates multiple homographies that are compatible with local geometries to align the input images, e.g., as-projective-as-possible (APAP) warping <cit.>. Mesh-based warping first pre-warps the image using global homography; then, it adopts some energy functions to optimize the alignment, treating it as a mesh warping problem, e.g., content-preserving warping (CPW) <cit.>. The high degrees of freedom (DoFs) involved in these methods can better handle parallax than can global transformations; thus, they can provide satisfactory stitching results. However, some challenges remain to be addressed:


	
  - The current methods often fail to achieve satisfactory alignment in low-texture images. Due to the high DoFs, these methods inevitably depend heavily on point correspondences <cit.>. However, keypoints are difficult to detect in some low-texture images because the homogeneous regions, such as indoor walls, sky, artificial structures, are not distinctive enough to provide rich and reliable correspondences. Hence, these methods often erroneously estimate the warping model, which causes misalignments. 
	
  - The influence of projective distortions has not been fully considered. Because many methods are based on projective transformations, e.g., CPW <cit.>, APAP <cit.>, the stitched results of images taken under various photographing viewpoints may suffer from projective distortions <cit.> in the non-overlapping regions, including both shape and perspective distortions. For instance, some regions in the stitched image may be stretched or non-uniformly enlarged, and it is difficult to preserve the perspective of each image (Fig. <ref>(b), Fig. <ref>(a)).
	
  - The image structure distortion has not been fully considered. Some local warping models, e.g., CPW <cit.>, APAP <cit.>, may bend line structures, especially when stitching low-texture images. For instance, insufficient or unreliable keypoints cause APAP to erroneously estimate some local transformations, which results in misalignment of the local regions and distorts the line structures that span multiple local regions, while CPW employs  only feature correspondences and content smoothness to optimize the global transformation and does not consider structural constraints.






The challenges to image stitching can be clearly seen in Fig. 1. Fig. 1 (a) shows the original images and the detected features (points and lines). In some homogeneous regions, only a few points are detected and matched, making it difficult to estimate an accurate transformation. Fig. 1 (b) shows the stitching results from global homography <cit.>, CPW <cit.>, APAP <cit.> and the proposed method. When the restrictive imaging conditions are violated, the global homography model does not fit the data correctly; thus, it results in obvious misalignments (the red boxes). In low-textured areas with insufficient correspondence (red boxes), CPW lacks sufficient data to align the pre-warping result, and APAP cannot estimate accurate local homographies, causing obvious misalignments. The lack of point correspondences also leads to structural deformations in CPW and APAP (blue boxes), where straight lines are deformed into curves. Due to the projective transformation used in these three models and the fact that no measures are taken to eliminate distortions, the stitched image results of these methods suffer from severe projective distortions (the yellow boxes), where the chairs are enlarged non-uniformly. 

The above problems provide strong motivation for improving the performance of image stitching. To our knowledge, only a few studies have been conducted to address either of the aforementioned problems; consequently, additional efforts are needed. Recent studies (<cit.> and <cit.>) have reported that line features can be used to improve the alignment performance, and <cit.> and <cit.> recently showed that similarity transformations are advantageous in reducing distortions. Inspired by these studies, our work is based on the following two assumptions: 


	
  - In most man-made environments, line features are relatively abundant, thus they can be regarded as effective supplements that can provide rich correspondences for accurate warping model estimation <cit.>. Furthermore, line features depict the geometrical and structural information of scenes <cit.>; thus, they can also be used to preserve the image structures.
	
  - Similarity transformation <cit.> does not introduce shape distortion because it consists only of translation, rotation and uniform scaling. A similarity transformation can be regarded as a combination of panning, zooming and in-plane camera rotation; therefore, it preserves the viewing direction. 


It is thus of great interest to investigate how to integrate line features and global similarity transformation to improve the image stitching performance. To this end, this paper presents a line-guided local warping model for image stitching with a global similarity constraint. More precisely, this method adopts a two-stage scheme to achieve good alignment. First, pre-warping is jointly estimated using both point and line features. Then, extended mesh-based warping is used to further align the pre-warping result. Line features are integrated into mesh-based warping framework and act as structural constraints to preserve image structures. Finally, to prevent  undesirable distortions, the global similarity transformation is adopted as a similarity constraint and used to adjust the estimated warping model. The contributions of our work are as follows:


	
  - We introduce line features to guide image stitching, especially in low-texture cases. Line features play a significant role mainly in two aspects: 1) they are integrated into the local warping model using a weight function to achieve accurate alignment; 2) they are employed to impose strong geometric constraints (i.e. line correspondence and line collinearity) to refine the stitching performance.
	
  - We present a weight integration strategy to combine the global similarity constraint with models of global homography or multiple homographies. Using this strategy, the resultant warp achieves a smooth transition from a projective to a similarity transformation across the image, which significantly mitigates the projective distortions in non-overlapping regions.
	
  - We propose a robust and effective two-stage stitching framework that combines the local multiple homographies model and the mesh-based warping model with line and global similarity constraints. The proposed method addresses local variation well to ensure image alignment by local stitching and flexible refinement. The method also preserves image structures and multi-perspective through strong geometrical and structural constraints.	The proposed method achieves a state-of-the-art performance.


The remainder of this paper is organized as follows. Section <ref> gives a brief review of the related works. Section <ref> describes the proposed method in detail. The experimental results and analyses are reported in Section <ref>. Finally, we draw some conclusions and provide remarks in Section <ref>.







§ RELATED WORKS



Numerous studies have been devoted to image stitching; a comprehensive survey can be found in <cit.>. The global homography model <cit.> works well for planar scenes or for scenes acquired with parallax-free camera motion, but violation of these assumptions may lead to ghosting artifacts.

Recently, spatially-varying warping methods have been proposed that flexibly address parallax. Liu et al. <cit.> proposed the content-preserving warping (CPW) method, which was first used in video stabilization. CPW adopts registration error and content smoothness to refine the pre-warping result obtained by global homography. A simple extension of global homography method was presented in <cit.>, called dual-homography warping (DHW), which divides the entire scene into two planes: a distant plane and a ground plane. The final warping is obtained by a linear combination of these two homographies estimated by the point correspondences of each plane. However, this method has difficulties on complex scenes. Lin et al. <cit.> proposed the smoothly varying affine (SVA) warping method for image stitching. SVA can handle local deformations while preserving global affinity. However, because there are insufficient DoFs in the affine model, SVA cannot achieve projective warping. Zaragoza et al. <cit.> extended the previous method and proposed an as-projective-as-possible (APAP) warping method for image stitching. APAP achieves a smoothly varying projective stitching field estimated by a moving direct linear transformation (DLT) <cit.>. It maintains a global projection while allowing local non-projective deviations. Zhang et al. <cit.> proposed a parallax-tolerant image stitching method that seeks the optimal homography evaluated by the seam cost and uses CPW to refine the alignment. However, except for SVA, these methods are based on projective transformations, thus the stitched images often suffer from projective distortions. In addition, the resulting images may suffer from structural deformations because of the nonlinear loceal transformations in the model.

In recent years, similarity transformation, which is composed of translation, rotation and scaling, was introduced. Similarity transformation constructs a combined warping with projective transformations to constrain the projective distortions. Chang et al. <cit.> proposed a shape-preserving half-projective (SPHP) warping for image stitching that adopts projective, transition and similarity transformation to achieve a gradual change from a projective to a similarity transformation across the image. SPHP can significantly reduce the distortions and preserve the image shape; however, it may introduce structural deformations, e.g., line distortions, when the scene is dominated by line structures. Lin et al. <cit.> proposed an adaptive as-natural-as-possible (AANAP) warping that linearizes the homography in the non-overlapping regions and combines these homographies with a global similarity transformation using a direct and simple distance-based weight strategy to mitigate perspective distortions. However, some distortions still exist locally when stitching images (Fig. <ref>(b)).

It is worth noting that spatially-varying warping-based image stitching is highly dependent on point correspondences. When there are insufficient reliable keypoints (such as in low-texture images), the effects of the estimated models will degrade. More recently, Joo et al. <cit.> introduced line correspondences into the local warping model, but this approach requires a user to annotate the straight lines, and setting the parameters for this method is complex. Li et al. <cit.> proposed a dual-feature warping method for motion model estimation that combines line segments and points to estimate the global homography. However, this method still suffers from projective distortions.




§ THE PROPOSED APPROACH



This section introduces the proposed method for image stitching in detail. The main idea is to integrate line constraints and a global similarity constraint into a two-stage alignment framework. The outline of our method is illustrated in Fig. <ref>. The first-stage alignment (presented in Section <ref>) involves estimating an accurate warping model using line guidance. Linear features are adopted as alignment constraints to jointly estimate both global and local homography with point correspondences, which provide rich and reliable correspondences even in low-texture images. To further improve the stitching performance, we adopt mesh optimization based on the extended content-preserving warping framework presented in Section <ref>. Then, the linear feature constraints (i.e., line correspondence and line collinearity) are combined to further refine the alignment and preserve the image structures. Finally, to mitigate the projective distortions, a global similarity transformation, estimated by a set of selected points in the approximate image projection plane, is employed to constrain the distortions caused by projective warping via a weighted integration strategy (Section <ref>). Based on the proposed warping model, we are able to achieve accurate and distortion-free image stitching.





 §.§ Line-guided warping model



Point features are often adopted for image alignment. Given the target and reference images I, I^', ℝ×ℝ↦ℝ, and a pair of matching points: 𝐩=[x,y,1] and 𝐩^'=[x^',y^',1] where x,y∈ℝ, the global homography, 𝐇∈ℝ^3 × 3: 𝐩^' = 𝐇𝐩, can be estimated by minimizing the algebraic distance ∑_i 𝐩_i^'×𝐇𝐩_i^2 between a set of matching points, where i is the index of matching points.

However, as stated previously, keypoints extracted from images are rare in some low-texture scenarios, thus it is difficult to estimate an accurate global homography for image stitching. Hence, line features, which are salient in artificial scenarios, are adopted as the alignment constraint to guide the global homography estimation.

Let 𝐥=[ a,b,c ]^T, 𝐥^' = [ a^',b^',c^' ]^T, with a,b,c∈ℝ be a pair of matching lines in the target and reference images respectively. Here, 𝐩^0,1=[x^0,1,y^0,1,1] denotes the two endpoints of line 𝐥. They satisfy 𝐥^'^T𝐇𝐩^0,1=0, which means that the endpoints transformed by 𝐇 from 𝐥 should lie on the corresponding line 𝐥^'. Therefore, 𝐇 can be estimated by minimizing the algebraic distance ∑_j 𝐥_j^'^T ×𝐇𝐩_j^0,1^2 using a set of matching lines, where j is the index of the matching lines.	

The homography is then estimated jointly by point and line correspondences:

    ĥ    = min_𝐡( ∑_i 𝐩_i^'×𝐇𝐩_i ^2  + ∑_j 𝐥_j^'^T×𝐇𝐩_j^0,1^2)  
        = min_𝐡( ∑_i 𝐀_i𝐡^2  + ∑_j 𝐁_j𝐡^2),    s.t. 𝐡 = 1,

where 𝐡=[h_1,h_2,h_3,h_4,h_5,h_6,h_7,h_8,h_9] is the column vector representation of 𝐇, and 𝐀_i, 𝐁_j ∈ℝ^2 × 9 are the coefficient matrixes computed by the i–th matching point and j–th matching line, respectively. 

Stacking all the coefficient matrices of points (𝐀_i) and lines (𝐁_j) vertically into a unified matrix, 𝐂 = [𝐀;𝐁], and Eq. (<ref>) can be rewritten as follows:

    ĥ
    	= min_𝐡𝐂𝐡^2,    s.t. 𝐡 = 1,

The global homography 𝐇 is the smallest significant right singular vector of 𝐂. Note that before estimation, all the entries of the stacked matrices [ A_i;B_j] should be normalized for numerical stability. In this study, we adopt the point-centric normalization approach proposed in <cit.>.

Local homography can handle parallax better than global homography due to the higher DoFs <cit.>. Therefore, we extend the line-guided global homography to local homographies. The input images are first divided into uniform grid meshes. The local homography 𝐡_k of the k–th mesh located at 𝐩_* = [x_*,y_*] is estimated by

    𝐡_k =
    	𝐡min𝐖_k 𝐂𝐡^2, s.t  h  = 1,

where 𝐖_k=diag( [ 𝐰^p,𝐰^l] ), 𝐰^p∈ℝ^2N, and 𝐰^l∈ℝ^2M denote the weight factors for the point and line correspondences, respectively. Specifically, 𝐰^p=[w^p_1 w^p_1 ... w^p_N w^p_N], and 𝐰^l=[w^l_1 w^l_1 ... w^l_M w^l_M]. Therefore, the solution is the smallest significant right singular vector of 𝐖𝐂.

The point weight factor 𝐰^p is calculated by the Gaussian weighted Euclidean distance:

    𝐰^p_i=
    	max( exp( -𝐩_*-𝐩_i^2/σ^2),η),

where 𝐩_i is the i–th keypoint, σ is the scale parameter, and η∈ [0,1] is used to avoid the numerical issues caused by the small weights when the mesh center 𝐩_* is far away from keypoint 𝐩_i, as shown in Fig. <ref>(a).

The line weight factor 𝐰^l is calculated as follows:

    𝐰^l_j=
    	max( exp( -d_l( 𝐩_*,𝐥_j)^2/σ^2),η),

where d_l(𝐩_*,𝐥_j) is the shortest distance between the mesh center 𝐩_* and line 𝐥_j, calculated as follows:


    d_l(𝐩_*,𝐥_j)=
    	{   min (𝐩_*-𝐩_j^0,𝐩_*-𝐩_j^1)    ( a ) 
       | a_jx_*+b_jy_*+c_j |/√(a_j^2+b_j^2)   ( b ) 
     ,.

where 𝐩_j^0, 𝐩_j^1 are the endpoints of line 𝐥_j: 𝐥_j=[a_j,b_j,c_j]. As shown in Fig. <ref>(b), when 𝐩_* is in the R_1 or R_2 region, the d_l is calculated by (a), and when 𝐩_* is in the R_3 region, d_l is calculated by (b). From Eq.(<ref>) and (<ref>), the weight is greater when the keypoint or line is closer to the mesh center 𝐩_*, which causes the local homography to be a better fit for the local structure around 𝐩_*.








 §.§ Alignment refinement with line constraints



This section describes the adoption of mesh optimization as the second step of the two-stage alignment scheme to further improve the performance of image stitching. Content-preserving warping is a mesh-based warping method that was first used for video stabilization in <cit.> and, later, successfully applied to image stitching <cit.>. It is well-suited for small local adjustments. In our work, the line feature constraints (e.g., the line correspondence constraint and line colinearity constraint) are integrated into the content-preserving warping framework to both maintain the image structures and refine the alignment satisfactorily.

The target image I is first divided into a regular grid mesh. In our case, the grid mesh is used to guide the image warping. Supposing 𝐕 denotes the vertices of the grid mesh in the pre-warping image transformed by the line-guided warping model. Alignment refinement is performed to find a group of deformed vertices 𝐕 using energy optimization.
An arbitrary point 𝐩 in the pre-warping image can be represented by a linear combination of four mesh vertices 𝐕 = [ 𝐕_1,𝐕_2,𝐕_3,𝐕_4]^T in its locating quad: 𝐩 = 𝐰^T𝐕, and 𝐰 = [ w_1,w_2,w_3,w_4]^T are calculated by inverse bilinear interpolation <cit.> and sum to 1. Therefore, the image warping problem can be formulated as a mesh warping problem. In fact, it is an optimization problem in which the objective is to accurately align the pre-warping image to the reference image while avoiding obvious distortions. The energy terms used in this paper are detailed below.




  §.§.§ Content-preserving warping


Content-preserving warping <cit.> includes three energy terms: a point alignment term, a global alignment term and a smoothness term. 

The point alignment term E_p is used to align the feature points in the target image or pre-warping image to the corresponding points in the reference image as much as possible. It is defined as follows:

    E_p = ∑_i𝐰_i^T𝐕_i  - 𝐩_i^' ^2,

where 𝐩_i^' is the matching point in the reference image. This term ensures the alignment of the overlapping region.

The global alignment term E_g is used to constrain the image regions without feature correspondences to be as consistent as possible with the pre-warping result:  

    E_g = ∑_i𝐕_i - 𝐕_i  ^2,

where 𝐕_i is the corresponding vertex in the pre-warping result.

The smoothness term E_s encourages each grid in the pre-warping result to preserve similarity during warping to avoid shape distortions as much as possible. Precisely, given a triangle ▵𝐕_0 𝐕_1 𝐕_2 in the pre-warping result, the vertex 𝐕_0 can be represented by 𝐕_1 and 𝐕_2 as shown below:

    [ 𝐕_1 = 𝐕_2 + μ (𝐕_3-𝐕_2) + ν𝐑 (𝐕_3-𝐕_2), ]
    	𝐑 = [
    	    0     1
        -1     0
    	],

where μ, ν are the coordinate values of 𝐕_0 in the coordinated system defined by the other two vertices. During warping, the triangle uses a similarity transformation to preserve the relative relationship of the three vertices and avoid local distortions. The smoothness term is

    E_s(𝐕_1) = φ𝐕_1 - ( 𝐕_2 + μ (𝐕_3-𝐕_2) + ν𝐑 (𝐕_3-𝐕_2))   ^2,

where φ is a weight used to measure the salience of the triangle as in <cit.>. The weight more strongly preserves the shapes of high-salience regions than those of low-salience regions. The full smoothness energy term is formed by summing Eq. (<ref>) over all the vertices.




  §.§.§ Line correspondence term


However, content-preserving warping terms only ensure the point alignment in the overlapping regions; thus, the line correspondences are taken into consideration to further improve the alignment.

A line correspondence term is utilized to ensure that the line correspondences are well aligned. Let 𝐥_j, 𝐥_j^' be a pair of corresponding lines in the target and reference images, respectively. Line 𝐥_j is cut into several short line segments by the edges of mesh if the line 𝐥_j traverses this mesh. The endpoints of the short line segments from 𝐥_j are denoted by 𝐩_j,k, where k is the index of the endpoints, and 𝐩_j,k^' denotes the endpoints in the pre-warping image transformed from 𝐩_j,k by the preceding warping process, 𝐩_j,k^' = 𝐰_j,k^T𝐕_j,k. The line correspondence term can be expressed by the idea that the distance from 𝐩_j,k^' to the corresponding line 𝐥_j^' should be the minimum distance:

    E_l = ∑_j,k(𝐥_j^'^T·𝐰_j,k^T𝐕_j,k)/√(a_j^'^2 + b_j^'^2) ^2.


The line correspondence term not only enhances the image alignment but also, together with line collinearity term below, preserves the straightness of line structures.




  §.§.§ Line collinearity term


However, the above terms may not reduce the distortions (e.g., line structure distortions) in the non-overlapping regions where there are few point or line correspondences. To capitalize on the line features and preserve the line structure, we adopt the line collinearity constraint.

The line collinearity term is used to preserve the straightness of linear structures in the target image as much as possible. Let 𝐩_i,k denote the endpoints and intersecting points of line 𝐥_i in the non-overlapping regions with the grid. Assume that 𝐩_i,k^' denotes the corresponding points of 𝐩_i,k in the pre-warping result. The line should maintain its straightness after warping, that is, the transformed points 𝐩_i,k^' should lie on the same line. This can be represented by the distance from the endpoints 𝐩_i,k^' to the line 𝐥̂_i which should be the minimum distance. Line 𝐥̂_i is calculated by the head and tail endpoints of 𝐩_i,k^'. The term is defined as follows:

    E_c =
    	∑_i,k(𝐥̂_i^T ·𝐰_i,k^T𝐕_i,k) / √(â_i^2 + b̂_i^2) ^2.


Together, the line collinearity term and the line correspondence term maintain the line structures well.




  §.§.§ Objective function

The above five energy terms are then combined as an energy optimization problem in which the objective function is

    E = αE_p + βE_g + γE_s + δE_l + ρE_c,

where α, β, γ, δ, ρ are the weight factors for each energy term. In our implementation, α = 1, β = 0.001, γ =0.01, δ = 1, and  ρ = 0.001. The above function is quadratic; consequently, it can be solved by a sparse linear solver. The final result is obtained through texture mapping.




 §.§ Distortion reduction by global similarity constraint



To reduce the projective distortions in the non-overlapping regions, the global similarity transformation is adopted to adjust the local warping model.

Chang et al. <cit.> has shown that similarity transformation is effective in mitigating distortions. If we can find a similarity transformation that approximately represents the camera motion of the image projection plane, that transformation can be applied to offset the camera motion <cit.>. RANSAC <cit.> is used to iteratively segment the matching points. Each group of point correspondences can be used to estimate a similarity transformation. The estimation with the smallest rotation angle is selected as the optimal candidate <cit.>. As shown in Fig. <ref>, the group of points in green is chosen to estimate the global similarity transformation. The plane composed of green points approximates the image projection plane because the camera is nearly perpendicular to the ground when shooting.







  §.§.§ Similarity constraint

An image patch can be transformed by a projective transformation (e.g. homography), which provides good alignment but may cause distortions, such as stretching. An image patch can also be warped by the similarity transformation, which, although it introduces no distortions, may result in poor alignment due to the limited DoFs. Integrating two types of transformations using weights, can therefore both ensure good alignment and reduce distortions. The similarity constraint procedure is described in Algorithm <ref>. The global similarity transformation is combined with global or local homographies using weight factors. To create a smooth transition, the whole image should be considered. The weight integration is calculated as follows:

    𝐇_i^' = τ𝐇_i + ξ𝐒,

where 𝐇_i is the homography in the i–th grid mesh, and 𝐇_i^' is the final homography in the i–th grid mesh. Here, 𝐒 is the similarity transformation, and τ and ξ are weight coefficients with τ + ξ = 1. The calculation of these two weights will be described later. In a global homography model, the homography of every grid mesh is the same.

The corresponding warping procedure should also be applied to the reference image because the similarity transformation also adjusts the overlapping regions. The warping procedure for the reference image can be formulated as follows:

    𝐓_i^' = 𝐇_i^'𝐇_i^-1,

where 𝐓_i^' is the warping procedure for the reference image in the i–th grid mesh.





As shown in Fig. <ref>, when a point is far from the overlapping regions (especially the distorted non-overlapping regions) the procedure assigns a high weight for the similarity transformation to mitigate the distortions as much as possible. In contrast, for points near the overlapping regions, it assigns a high weight for the homography to ensure accurate alignment. Using this weight combination, the final warp smoothly changes from a projective to a similarity transformation across the image, which preserves the image shape and maintains the multi-perspective.







  §.§.§ Weighting strategy

The weight coefficient calculation stems from the analysis of projective transformation. According to <cit.>, let 𝐑 be a rotation transformation that transforms the image coordinate (x,y) to a new coordinate (u,v). Based on 𝐩^' = 𝐇𝐩, a new projective transformation 𝐐 that transforms (u,v) to (x^',y^') meets 𝐩^' = 𝐐 [u, v, 1]^T = 𝐇𝐑 [u, v, 1]^T, where 𝐇 = [h_1, h_2, h_3; h_4, h_5, h_6; h_7, h_8, 1], and 𝐐 = [q_1, q_2, q_3; q_4, q_5, q_6; q_7, q_8, 1].

Supposing that the rotation angle is θ  = arctan( h_8/h_7), we can obtain q_8 =  - h_7sinθ + h_8cosθ = 0. Then, 𝐐 can be decomposed as follows:

    [    q_1   q_2   q_3
       q_4   q_5   q_6
        -c     0     1
    	] =
    	[    q_1 + cq_3   q_2   q_3
       q_4 + cq_6   q_5   q_6
        0     0     1
    		]_Q_a[     1     0     0
        0     1     0
        -c     0     1
    		]_Q_p,

where c = √(h_7^2 + h_8^2). Here, 𝐐_a is the affine transformation, and 𝐐_p is the projective transformation. Defining the local scale change <cit.> at point (u,v) under the projective transformation as the determinant of the Jacobian of 𝐐 at point (u,v), the local scale change is calculated as follows:

    𝐉( u,v) =
    	𝐉_a(u,v) ·𝐉_p( u,v) = λ_a·1/( 1 - cu)^3,

where det denotes the determinant, and λ_a is independent of u and v. It can be seen that the local area change derived from 𝐐 relies only on the u direction. In other words, the distortions of projective transformation occur only along the u–axis. Therefore, the distortions can be effectively eliminated if the weight coefficients are calculated along the u direction in the (u,v) coordinate system.




The weight coefficients are designed based on the distance of grid points in the u direction; the goal is to provide a gradual change from a projective to a similarity transformation across the image to preserve the image content in non-overlapping regions. As shown in Fig. <ref>, the center of the reference image is used as the origin of coordination o, and the unit vector on the u–axis denotes ou =( 1,0 ). For the arbitrary mesh center 𝐩, d is the projected length of vector o𝐩 on the vector ou. The projected point 𝐩_max with a maximum length of d and the projected point 𝐩_min with a minimum length of d can be calculated. For the i–th grid, the weight coefficients are calculated as follows:

    ξ  =
    	< 𝐩_min𝐩_i·𝐩_min𝐩_max >  /| 𝐩_min𝐩_max|,

where < 𝐩_min𝐩_i·𝐩_max𝐩_min > denotes the projection length of 𝐩_min𝐩_i on 𝐩_max𝐩_min, and τ  = 1 - ξ.




As shown in Fig. <ref>, APAP adopts the local homographies for alignment, which aims to be both globally projective while allowing local deviations. However, the stitched image suffers from projective distortions; for instance, the buildings are undesirably stretched and not parallel to the temples, in addition, the perspective distortions in the non-overlapping regions are obvious. In contrast, using a global similarity constraint, the proposed warping model preserves the shapes of objects and maintains the perspective of each image.





§ EXPERIMENTAL RESULTS AND ANALYSIS



This section describes several experiments conducted to assess the performance of the proposed method on a series of challenging images. In our experiments, the testing images were acquired casually, using different shooting positions and angles.

Given a pair of input images, the keypoints are detected and matched by SIFT <cit.> in the VLFeat library <cit.>. The line features are detected by a line segment detector (LSD) <cit.> and matched by line-point invariants <cit.> or line-junction-line <cit.>. Then, RANSAC is used to remove the mismatches, and the remaining inliers are input to the stitching algorithms. We compared our approach with several other methods. The parameters of the other methods were set as suggested in the respective papers and we used the source code provided by the authors of the papers to obtain the compared results. For our method, σ is 8.5, and η is 0.01. The experiments were conducted on a PC with an Intel i3-2120 3.3 Ghz CPU and 8 GB of RAM. Not considering feature detection and matching, the proposed method takes 20–30 s to stitch together two images with a resolution of 800×600.

To better compare the methods and reduce interference, we avoided post-processing methods such as blending or seam cutting as detailed in <cit.>. Instead, the aligned images are simply blended by intensity average so that any misalignments remain obvious.

To assess the accuracy of the image stitching alignment quantitatively, the metrics of correlation (Cor) <cit.> and mean geometric error (Err_mg) <cit.> are adopted. Cor is defined as one minus the normalized cross correlation (NCC) over the neighborhood of a 3 × 3 window, that is

    Cor( I,I^') =
    	√(1/N∑_π(  1 - NCC( 𝐩,𝐩^') )^2 ) ,

where N is the number of pixels in the overlapping region π, and 𝐩 and 𝐩^' are the pixels in image I and I^', respectively. Cor reflects the similarity of two images in the overlapping regions. The smaller the Cor value is, the better the stitching result is.

Err_mg is defined as the mean geometric error on points and lines, that is

    [                  Err_mg^(p)=1/M∑_i=1^M f(𝐩_i) - 𝐩_i^'; Err_mg^(l)=
    		1/2K∑_j=1^K∑_i=0^1d_l(f(𝐩_l_j^i),𝐥_j^');            Err_mg=(Err_mg^(p)*M+Err_mg^(l)*2K)/(M+2K) ] ,

where f: ℝ^2 ↦ℝ^2 is the estimated warping, M is the number of point correspondences, 𝐩_i and 𝐩_i^' are a pair of point correspondences, K is the number of line correspondences, and d_l denotes the projected distance of the endpoints of 𝐥_j to its correspondence line 𝐥_j^'. A smaller Err_mg value indicates a better stitching result.

In the following subsections, we first verify the performance of the proposed method on image alignment and distortion reduction. Then, we report the experimental comparison results including the comparison with the global-based methods and the local-based methods.




 §.§ Image alignment
  

Fig. <ref> illustrates the performance of each constraint in the proposed method, including the line-guided local warping estimation, the line correspondence constraint, and the line colinearity constraint. Fig. <ref>(b) shows the result of line-guided warping combined with APAP (LAPAP), which largely improves the alignment compared to APAP, as can be clearly seen in the closeup. However, LAPAP introduces structural distortions, e.g., the bent lines on the buildings, shown by red circle in the blue closeup. With CPW optimization, LAPAP+CPW refines the alignment performance (shown in Fig. <ref>(c)), but some slight misalignments still exist. Combined with line correspondence (LineCorr) constraint, LAPAP+CPW+LineCorr provides good alignment (Fig. <ref>(d)). However, structural distortions, e.g., line deformations, are not handled well as can be clearly seen in the blue closeup. By adding the line collinearity constraint to restrain the structural deformation, the proposed method provides a good stitching result with less distortion in this example (Fig. <ref>(e)). Quantitative evaluations of  Cor and Err_mg are shown in Table <ref>, which demonstrates conclusions consistent with the visual effect.






Fig. <ref> shows a comparison of the original point-based CPW model <cit.> and the proposed CPW model on the Rooftops[The Rooftops images were acquired from the open dataset of <cit.>.] images. Some errors or distortions are highlighted by the red circles. The stitching process is based on the proposed two-stage alignment. Fig. <ref>(a) shows the results from the original CPW model, in which misalignments are obvious, especially on the rooftops (red circle). Additionally, the roadside trees are stretched. Under the constraints of line features, Fig. <ref>(b) improves the alignment performance and produces more accurate results. As can be seen, line features provide a better geometric description than do point features alone, and the line features function as strong constraints for image stitching. Fig. <ref>(c) shows the final stitching results. Due to the global similarity constraint, the distortions around the roadside trees are largely mitigated, and the proposed method achieves a satisfactory stitching result. Table <ref> shows the quantitative comparison. The improved CPW model largely reduces the alignment errors (mainly line errors and total error). By using the similarity constraint, the proposed method obtains a lower Cor.







Next, we compared the proposed method with other flexible warping methods to evaluate the alignment performance, namely, global homography (baseline) <cit.>, CPW (using global warping for the initial alignment) <cit.>, and APAP <cit.>. For completeness, the proposed method is also compared with the Image Composite Editor (ICE) <cit.> (a common commercial tool for image stitching) by inputting two images at once. For ICE, we used the final post-processed results for the comparison because the original alignment results are not obtainable in the standard version of ICE. In addition, no quantitative comparison of ICE is provided.


Fig. <ref> shows the Desk image pair and the detected feature. For most of the low-textured areas, the keypoints are difficult to extract, resulting in insufficient matching points for the estimation of warping model. However, line features can be used as an effective complement for alignment purposes.






The comparison results are shown in Fig. <ref>. Because the images violate the assumptions, the baseline warp is unable to align them properly; it produces obvious misalignments (see the red boxes in Fig. <ref> (a)). ICE, CPW, and APAP provide relatively better stitching results, but a non-negligible number of ghost artifacts remain. In Fig. <ref>(b), although ICE uses blending and pixel selection to conceal the misalignments, the post-processing is clearly not completely successful; for instance, there are obvious misalignments on the vertical edge of the desk. Due to an insufficient number of corresponding keypoints along the vertical edge of the desk, CPW and APAP cannot provide an accurate warping model for image alignment; consequently, ghosting occurs in these regions (see the red boxes in Fig. <ref> (c) and (d)). With the help of line correspondences and the two-stage robust alignment scheme, our method results in satisfactory stitching performance with accurate alignment and few ghost artifacts (Fig. <ref>(e)). Note that our method also reduces the need for post-processing.

Table <ref> depicts the Cor and Err_mg values of the compared methods on the Desk image pair. As listed, CPW's stronger constraint on point correspondences results in a smaller alignment error on point Err_mg^(p); however, the alignment errors on line Err_mg^(l) and Err_mg remain large. The proposed method reduces the geometric error and results in better accuracy than do the other tested methods.






 §.§ Distortion reduction
 



To investigate the distortion reduction performance, SPHP <cit.> and AANAP <cit.> were compared with the proposed method on the Railtracks and Temple Square image pairs[The Railtracks and Temple Square images were acquired from the open dataset of <cit.>.].

Fig. <ref> shows the stitching results of the four methods, APAP <cit.>, SPHP <cit.>, SPHP with an assumption of no rotation (SPHP1) <cit.>, and our method. Due to its simple extrapolation of projective transformation to non-overlapping regions, the APAP (Fig. <ref>(a)) result shows projective distortions in the non-overlapping regions. In the blue box in the closeup, the car is enlarged, and the palm tree is obviously slanted. By introducing the similarity transformation, SPHP can largely mitigate these projective distortions. In Fig. <ref>(b), SPHP preserves the shape of the car, but it has a problem with the unnatural rotation. In addition, the construction site (in the red box) is tilted to the left. In contrast, SPHP1 preserves the shape and reduces the perspective distortion, but the construction site is now tilted slightly to the right (<ref>(c)). Using the global similarity constraint, the proposed method largely eliminates all these distortions, providing a pleasing stitching result, as is clearly shown in <ref>(d).




Fig. <ref> shows a comparison of the proposed method with AANAP <cit.> on distortion reduction. Fig. <ref>(a) shows that APAP achieves good alignment, but it suffers from shape and perspective distortion problems, for example, in the stretched and tilted buildings at the right of the image. By linearizing the homography and using the similarity transformation, AANAP provides an attractive result in which the projective distortions have been largely mitigated (Fig. <ref>(b)). However, as shown in the red circle of the enlarged view, the lines on the ground are slightly deformed. Our method yields more appealing stitching results in this example (Fig. <ref>(c)).




 §.§ Comparisons with global-based methods















In this section, the proposed method is compared with three global-based methods: global homography (Baseline) <cit.>, ICE <cit.>, and SPHP <cit.>. For our method (called the global version), global homography is adopted during the first alignment stage and jointly estimated by point and line correspondences to pre-warp the source images.

Fig. <ref> shows the two pairs of original images for stitching: Ceiling and Temple [The Temple images were acquired from the open dataset of <cit.>.]. The low-textured content of Ceiling results in the detection of only a limited number of unevenly distributed  keypoints, which may degrade the warping model's  estimations. However, line correspondences are abundant, which can improve the image alignment. Temple provides rich point correspondences, but the scene contains multiple distinct planes, which is a challenge for the global-based methods.

Fig. <ref> and <ref> show the results of the global-based methods on the Ceiling and Temple image pairs. As shown, due to the model deficiencies, the Baseline warp cannot provide satisfactory stitching results; there are numerous misalignments and projective distortions. The ICE and SPHP methods improve the stitching performance, especially in the aspect of the reduction of projective distortions. For instance, the door in Fig. <ref> and the people in Fig. <ref> have few distortions, but the bricks of the ceiling in the non-overlapping area in the ICE result (Fig. <ref>(b)) are slightly stretched. In addition, alignment errors in these two pairs of images (the red circles in Fig. <ref> and <ref>) remain obvious. In contrast, the proposed method is more flexible and robust in handling the alignment not only because of the line-guided warping estimation but also because of the alignment constraints in the mesh-based framework. With the similarity constraint, our method provides good stitching results with minimal distortions.

Table <ref> and Table <ref> contains a quantitative comparison of Ceiling and Temple, showing that our method provides the results with the fewest errors. On Ceiling, our method performs the best because the line features play an important role in scenes without reliable keypoint correspondences. On the Temple image, which has rich and reliable keypoints, the role of the line feature may be reduced, but it still helps to improve the alignment accuracy.





 §.§ Comparisons with local-based methods









The global version works well in preserving the content and perspective, but it is somewhat less robust when aligning images taken with large views. For high DoFs and flexible local homographies, our method that uses local homography in the pre-warping stage (called the local version) can handle the parallax issue. Therefore, in this section, we compared it with three other local-based methods: CPW <cit.>, APAP <cit.>, and SPHP+APAP <cit.>. Fig. <ref> shows the original Church, Block, and Wall images for the comparison experiments[The Church and Block images were acquired from the open dataset of <cit.>.]. Some images have little texture, which limits the extracted features. Moreover, the images' corresponding views vary greatly.

The stitching results on these three pairs of images are provided in Fig. <ref>. In terms of alignment accuracy, CPW and APAP allow higher DoFs than does global homography, but they also produce misalignments in regions that lack point correspondences (the areas partially highlighted in red boxes). In addition, CPW and APAP may cause local structure deformation in structural regions that lack keypoints. The red closeups clearly show that straight lines are bent (e.g., the stair railing in Church, the building edge in Block, and the wall edge in Wall). Using the similarity transformation, SPHP+APAP reduces the projective distortions and preserves the shape and perspective, mitigating the building distortion in the non-overlapping regions in both Church and Block. In comparison, our method not only provides accurate alignment, which benefits from the two-stage alignment scheme, but also preserves image structures and perspectives due to the line and similarity constraints.

Table <ref> shows the quantitative results of the compared methods. Our method consistently achieves better accuracy than CPW, APAP and SPHP+APAP except for Err_mg^(p) in Church result. CPW adopts feature alignment as a strong constraint; therefore, it provides a good quantitative result in Err_mg^(p). However, its results are unsatisfactory on other criteria on the Church image. Overall, our method achieves the best quantitative results.






 §.§ Stitching of multiple images
 
Figs. <ref> and <ref> show the stitching results of multiple images on the Apartments and Garden data, respectively[These images were acquired from the open dataset of <cit.>]. Some distinct errors are highlighted in boxes. As can be seen, Autostitch and ICE result in some obvious misalignments because they use only global homography for alignment, which is unsuitable for images whose views differ by factors other than pure rotation. In contrast, our method largely improves the stitching performance because of the flexible line-guided local homographies and mesh optimization. Thus, the proposed method produces satisfactory stitching results that contain few misalignments and distortions.











§ CONCLUSION



This paper proposed a line-guided local warping for image stitching by imposing similarity  constraint. Our method integrates multiple constraints, including line features and global similarity constraints, into a two-stage image stitching framework that achieves accurate alignment and mitigates distortions. The line features are employed as an effective supplement to point features for alignment. Then, the line feature constraints (line matching and line collinearity) are integrated into the mesh-based warping framework, which further improves the alignment while preserving the image structures. Additionally, the global similarity transformation is combined with the projective warping to maintain the image content and perspective. As shown by the results of performed experiments, the proposed method achieves a good image stitching result that yields the fewest alignment errors and distortions compared to other methods. The proposed method depends on line detection and matching; thus, incomplete or broken line segments may influence its structure-preserving performance. In future work, we would like to explore other complex structure constraints, such as contours <cit.>, to improve the image stitching performance, and explore the possibility of applying our warping model to other applications, such as video stabilization <cit.>.







IEEEtran

