BezierFormer: A Unified Architecture for 2D and 3D Lane Detection
Zhiwei Dong Xi Zhu Xiya Cao Ran Ding Wei Li Caifa Zhou Yongliang Wang Qiangbo Liu








BézierFormer: A Unified Architecture for 2D and 3D Lane Detection
    
Zhiwei Dong, Xi Zhu, Xiya Cao, Ran Ding, Caifa Zhou, Wei Li, Yongliang Wang, Qiangbo Liu
Riemann lab, Huawei Technologies
{dongzhiwei1, caoxiya1, dingran7, zhoucaifa, liwei.levi, wangyongliang775}@huawei.com, zhux2012@uw.edu

    April 29, 2024
======================================================================================================================================================================================================================================






Lane detection have made significant progress in recent years, but there is not a unified architecture for its two sub-tasks: 2D lane detection and 3D lane detection.
To fill this gap, we introduce BézierFormer, a unified 2D and 3D lane detection architecture based on Bézier curve lane representation. 
BézierFormer formulate queries as Bézier control points and incorporate a novel Bézier curve attention mechanism.
This attention mechanism enables comprehensive and accurate feature extraction for slender lane curves via sampling and fusing multiple reference points on each curve.

In addition, we propose a novel Chamfer IoU-based loss which is more suitable for the Bézier control points regression.
The state-of-the-art performance of BézierFormer on widely-used 2D and 3D lane detection benchmarks verifies its effectiveness and suggests the worthiness of further exploration.



2D Lane Detection, 3D Lane Detection, Autonomous Driving, Bézier Curve




§ INTRODUCTION






Lane detection based on RGB images is a critical foundational perception task in various applications such as autonomous driving, lane-level AR navigation, and HD map construction. 
This task is specifically divided into 2D lane detection and 3D lane detection.
Both of them are challenging due to the inherent difficulties, including obscured or worn lane markers, and variations in lighting and weather conditions. 
With the advent of deep learning, both 2D and 3D lane detection have made significant progress <cit.>.
However, there is not a unified architecture for these two subtasks, although they share many similarities. It usually takes a lot of efforts to adapt a state-of-the-art 2D lane detection model to 3D lane detection(or vice versa), and this hinders further development of lane detection.

In this paper, we aim to answer the question of whether it is possible to
build a unified state-of-the-art architecture for both 2D and 3D lane detection.


















First of all, we utilize the Bézier curve to represent lane curves in our method, because Bézier curve can uniformly and efficiently represent both 2D and 3D curves via a few control points.


Inspired by the DEtection TRansformers <cit.>, we introduce BézierFormer, which aims to query lane features from the input monocular image and output 2D or 3D control points of lanes.
As shown in Figure <ref>, our method formulates queries as dynamic Bézier control points, uses a Bézier curve attention mechanism to extract lane features.
Besides we propose a Chamfer IoU-based loss compatible with 2D and 3D Bézier control points regression.










Specifically, initial control point queries, learnable curve embeddings, and image features are fed into the decoder.
With the aid of Bézier curve attention, each decoder layer extracts lane features accurately and comprehensively to refine curve embeddings and control points from the previous layer. 
To implement Bézier curve attention mechanism, the front-view (FV) sampling module first samples a few sparse reference points along the curve based on control points, ensuring that the attention's receptive field fully covers the entire lane curve. 
Then we use the Bézier curve attention operation, a multi-reference-points variant of deformable attention <cit.>, to extract and fuse the features around these reference points on the curve.
Finally, extracted lane features enhance curve embeddings for lane regression and recognition. 

For lane regression, we present a loss derived from Chamfer Distance. It facilitates a more effective learning process by fitting the overall shape of the target curve.
To make our method compatible with 2D and 3D lane detection, there is an optional perspective projection operation in the FV sampling module as depicted in Figure <ref>(b).
When performing 3D lane detection, the perspective projection operation uses intrinsic and extrinsic parameters of the camera to project 3D reference points, sampled based on 3D control points, onto the corresponding 2D positions on FV features for Bézier curve attention. 
Then we can reapply the whole structure in Figure <ref>(a) and directly detect 3D lane curves from FV features.







We conduct experiments on two widely-used 2D and 3D lane detection benchmarks. The experiments show that BézierFormer is effective in both 2D and 3D lane detection, achieving state-of-the-art 90.72% F1 score on CurveLanes and 58.1% F1 score on OpenLane.







Our main contributions are summarized as follows. (1) We propose a unified 2D/3D lane detection architecture named BézierFormer. It formulates queries as dynamic Bézier control points, introduces a novel Bézier curve attention mechanism to extract lane features accurately and comprehensively, and effectively regresses Bézier control points using a novel Chamfer IoU-based loss. (2) BézierFormer achieves state-of-the-art performance on popular 2D and 3D lane detection benchmarks and suggests the worthiness of exploration in the future.




§ RELATED WORK


2D lane detection: Early methods use semantic segmentation or keypoint detection to find pixels or keypoints of lane markers, then associate them to get curve instances.
These works focus on achieving better segmentation <cit.> and designing better association method <cit.> to improve the overall lane detection performance.



These methods are intuitive but inefficient, and their detection results lack a holistic nature. 
Recently, top-down methods get more and more attention for their ability to detect lanes holistically and deal with visually challenging situations. They usually represent lanes as row-based coordinates <cit.>, polynomials <cit.> or parametric curves <cit.>, and then detect lanes in a similar way to object detection.





3D lane detection: Most methods transform image features from FV to bird-eye-view(BEV) for 3D lane detection. 
3D-LaneNet <cit.> and Gen-LaneNet <cit.> apply inverse perspective mapping (IPM) to transform features and utilize row-based lane anchors.
PersFormer <cit.> applies deformable attention for the transformation. 

Recent works have tried to skip the feature transformation through DETR-like architecture <cit.> or 3D lane anchors <cit.>.

BézierFormer also avoids feature transformation and is a unified framework for 2D and 3D lane detection.
It should be noted that BézierFormer is totally different with PersFormer <cit.>, which just integrates distinct 2D and 3D lane detection components into a single network, instead of designing a general network for these two tasks.





§ METHOD






 §.§ Lane Representation

In BézierFormer, we represent lane curves as Bézier Curves. 
Bézier curve of order N uses N+1 control points (c_1, c_2,...,c_N+1)  to represent a curve S and is defined by:

    S(t)=∑_n=0^Nb_n,N(t)· c_n+1,0≤ t≤1

Variable t ranges from 0 to 1, representing the Bézier curve being sampled from the start to the end. Coefficient b_n,N(t) is the Bernstein basis polynomial of degree n given by:

    b_n,N(t)=N!/n!(N-n)!t^n(1-t)^N-n,n=0,1,...,N

We adopt the classic cubic Bézier curve, which uses four control points (c_1, c_2, c_3, c_4) to represent a curve.
For 2D lane detection, c_1,c_2,c_3,c_4 are all 2D vectors representing the xy coordinates of control points.
For 3D lane detection, they are 3D vectors representing xyz coordinates.
The Bézier curve's ability to represent lane curves without relying on variables x, y or z allows for a unified representation of 2D and 3D lanes of any orientation. 




 §.§ Network Architecture

As shown in Figure <ref>, BézierFormer consists of a feature extractor and Bézier curve decoder. 
The feature extractor extracts multi-scale features of input images through a backbone network and a transformer encoder containing multi-scale deformable self-attention.
The generated features are denoted as X={x_l}_l=1^L, where L represents the number of feature map scales.
The decoder consists of N_layer homogeneous layers indexed starting from 1.
Decoder layeri receives features X, Bézier control point queries and corresponding curve embeddings coming from Decoder layeri-1 as inputs. 
The Bézier control point queries model the shapes and positions of lane curves and are denoted as C_i-1={ (c_1, c_2, c_3, c_4 )_i-1,j}_j=1^N_query.
N_query is the number of queries, which means the maximum number of lane curves that BézierFormer can detect. 
Curve embeddings contain lane features and are denoted as E_i-1= {e_i-1,j}_j=1^N_query.
Decoder layeri outputs more precise control points C_i and better embeddings E_i for the next layer.







For the first decoder layer, initial embeddings E_0 are randomly initialized learnable parameters. Initial control points C_0 could be initialized by C_0=MLP(X). Because C_0 is related to the input image, this way brings a good generalization ability.




 §.§ Bézier Curve Decoder

Bézier curve decoder consists of N_layer layers with the same structure.
It iteratively refines curve embeddings and control point queries. 
We employ a classification head for category recognition and use Bézier curve sampling defined by Eq.(<ref>) to produce dense lane curve points for detection results.

Taking Decoder layeri as an example, the inputs are X, C_i-1, and E_i-1.
To leverage the geometric information of lane curves in self-attention and Bézier curve attention, the FV sampling module first sample N_ref points for each query. Then, using sine positional encoding and a MLP, we fuse the positions of these N_ref points and obtain the query's positional encoding. The formula is:

    PE(C_i-1,j)=MLP(Concat {PE(S_i-1,j(t_k))}_k=1^N_ref)


After positional encoding, the relationships among lanes are modeled via  multi-head self-attention, which enhances E_i-1 to better curve embeddings E'_i-1 with global context:

    e'=e+∑_m=1^MW_m∑_j=1^N_queryA_m,jW_m'e_j

e and e' are single embeddings in E_i-1 and E'_i-1. 
m is the index of the attention head.
W_m∈ℝ^C_e× C_vand W'_m∈ℝ^C_v× C_e are learnable parameters, where C_e and C_v=C_e/M are the feature dimensions of the embedding and the key in attention respectively.
The attention weights A_m,j∝exp{e_j^TU_m^TV_me_j/√(C_v)} are normalized to ∑_j=1^N_queryA_m,j=1, and U_m,V_m∈ℝ^C_v× C_e are learnable parameters.

Subsequent Bézier curve attention employs the N_ref points produced by FV sampling module as reference points.
It collects curve features of each query from X and updates E'_i-1 to better embeddings E_i for the next layer. 
A classification head calculates the category vectors V_i from E_i.
A simple FFN predicts the offset Δ C_i, then the control points are refined as C_i=Δ C_i+C_i-1.
To achieve layer-by-layer refinement, V_i and C_i of each layer need to be supervised.



 §.§ Bézier Curve Attention Mechanism

Bézier curve attention mechanism aims to better capture features of the slender lane curves represented by Bézier curves.
We refrain from directly using vanilla deformable attention due to two primary reasons that render it suboptimal for extracting slender lane features.


Firstly, deformable attention originates from object detection and generates only one reference point per object, but one reference point is insufficient to describe a slender lane curve comprehensively. 
Secondly, the reference points of deformable attention are adaptively generated, which causes slow convergence and less accurate lane feature extraction.
To overcome these shortcomings, Bézier curve attention mechanism leverages Bézier curve sampling and Bézier curve attention operation to sample and fuse multiple reference points along the curve for accurate and comprehensive lane features.



The Bézier curve attention operation does not involve any interaction among {e'_i-1,j}_j=1^N_query, so we simplify the explanation of this operation by considering the update of a single e'_i-1,j.
Let { r_k}_k=1^N_ref be the N_ref reference points. As shown in the left-top part of Figure <ref>, the features {{ f_k,l}_k=1^N_ref}_l=1^L of the reference points at different scales of X are sampled, then two MLPs calculate f_k,l to obtain N_sample attention location offsets { o_k,l,s}_s=1^N_sample and attention weights { w_k,l,s}_s=1^N_sample for r_k at scale l. 
{ o_k,l,s}_s=1^N_sample are added to r_k to get the attention locations { p_k,l,s}_s=1^N_sample. To update e'_i-1,j, the formula is:

    e=e'+∑_m=1^MW_m∑_k=1^N_ref∑_l=1^L∑_s=1^N_sampleW_m,k,l,sW'_mx_l(p_k,l,s)

For simplicity, e' and e represent e'_i-1,j and e_i,j, respectively. 
m, W_m and W'_m have the same meanings as they have in Eq.(<ref>).
The dynamically generated attention weights W_m,k,l,s are normalized to ∑_k=1^N_ref∑_s=1^N_sampleW_m,k,l,s=1.
Bézier curve attention operation extracts lane curve features from FV features and updates curve embeddings E'_i-1 via Eq.(<ref>). Enhanced curve embeddings E_i are subsequently used for control points refinement and category recognition.


























































 §.§ Loss Function and Label Assignment

We present a lane regression loss derived from Chamfer Distance (CD) to better regress Bézier curve and use Focal Loss <cit.> for classification. 
<cit.> shows that regressing the sampled points can achieve better results than directly regressing Bézier control points. 
However, this method may sometimes be limited to optimizing the distance between local points, while ignoring the overall shape and location fit between two curves.
We give an intuitive explanation of this limitation in the Appendix.
We propose calculating CD between curves, as it considers each curve a whole entity to get the shortest distance from a point to the curve.
As CD is calculated between point sets instead of continuous curves, N_dis points { p_i^A}_i=1^N_dis and { p_i^B}_i=1^N_dis on curve A and B are sampled.
Inspired by <cit.>, we give the lane curve a width e and normalize CD to IoU, which we call Chamfer IoU, or CIoU for short. It is defined by:

    CIoU_A→ B=1/N_dis∑_i=1^N_dis2e-min{ L2 (p_i^A, p_j^B )  } _j=1^N_dis/2e+min{ L2 (p_i^A, p_j^B )  } _j=1^N_dis

The regression loss between a pair of predicted curve S_pred and ground truth curve S_gt can be formulated as:

    L_loc=1-1/2 ( CIoU_S_pred→ S_gt + CIoU_S_gt→ S_pred )


However, the computation of CIoU ignores the order among the sampled points. To ensure the correct order of the sampled points, we add two simple geometric constraints.

    L_len=L1(len(S_pred)/len(S_gt),1)


    L_endpoint=1/2 [L2(p̂_1^S_pred,p̂_1^S_gt)+L2(p̂_N_dis^S_pred,p̂_N_dis^S_gt) ]

len() means calculating the curve length. p̂_1, p̂_N_dis represent the normalized endpoints of the curve. Then, we get regression loss as L_reg=L_loc+L_len+L_endpoint
and total loss is the sum of the losses of all decoder layers. The formula is:

    L_total= ( L_reg^0+L_cls^0 ) + ∑_i=1^N_layer ( L_reg^i+L_cls^i )


To match predictions with ground truth, we adopt SimOTA <cit.> to dynamically assign topk predictions to each ground truth with the matching cost cost_i,j=L_reg_i,j+L_cls_i,j.
Specifically, we set topk=1 for an NMS-free pipeline.












§ EXPERIMENTS



Datasets: We conduct experiments on two widely-used 2D and 3D lane detection benchmarks.


CurveLanes <cit.> is a large-scale 2D lane dataset with complex lane topologies. 
It comprises 150,000 images for training, validation, and testing.
In CurveLanes, curved lanes account for over 90%, encompassing challenging cases such as dense, forked, merged, and nearly horizontal lanes.


OpenLane <cit.> is a recently released large-scale real-world 3D lane dataset, with a total of 200,000 images and over 880,000 3D annotations.




Evaluation Metric: To keep consistent with <cit.>, we use F1 score for 2D lane detection evaluation. 
For OpenLane, we follow <cit.> to report the F1 score and category accuracy.




Implementation Details: We choose ResNet18 <cit.> and Swin-Tiny <cit.> as pre-trained backbones.
Input images are resized to 800×320 on CurveLanes, and 480×360 on OpenLane.



We set N_query for CurveLanes and OpenLane to 16 and 32, respectively. 
We train 24 epochs with batch size 16, using an AdamW optimizer with a learning rate of 1e-4. 
We set t to 0, 0.25, 0.5, 0.75, and 1 to obtain N_ref=5 reference points, and N_sample=5.
To calculate L_reg, we set N_dis=200 and lane width e=10 for CurveLanes, e=0.9 for OpenLane.

More details are in the Appendix.



 §.§ Comparisons with the State-of-the-Art Methods















  §.§.§ 2D lane detection




Table <ref> shows BézierFormer achieves top-tier performance on CurveLanes.
Even based on ResNet18, BézierFormer achieves a better F1 score of 89.06% than CLRNet's 86.48% based on ResNet101.
With Swin-Tiny as the backbone, BézierFormer achieves a state-of-the-art F1 score of 91.06%, which is 4.58% higher than that of CLRNet.
Compared to the best bottom-up method RCLane, BézierFormer achieves a comparable performance but a much higher inference efficiency.
The results on CurveLanes indicate that BézierFormer is effective and effecient in 2D lane detection. 
We give more qualitative visualized results in the Appendix.








  §.§.§ 3D lane detection

Table <ref> shows that BézierFormer achieves state-of-the-art performance on OpenLane. 
With the input resolution of 480×360, BézierFormer achieves an F1 score of 56.43% and a category accuracy of 94.1%, which are 2.73% and 3.2% higher than those of the second-best Anchor3DLane, and has a higher FPS.
It is worth noting that NMS-free BézierFormer only uses 32 queries, while Anchor3Dlane needs over 4,000 3D lane anchors. 
CurveFormer is also a DETR-like method but represents lane curves as 3D point sets. 
BézierFormer outperforms CurveFormer by 5.93% F1 score and is superior in all scenarios and X/Z errors, indicating that Bézier curve is a better representation of 3D lane curves than 3D point sets. 
With a larger input resolution of 1024×576, BézierFormer⋆ achieves a higher F1 score of 58.6% and a category accuracy of 94.2%. 






Tabele <ref> demonstrates BézierFormer's effectiveness and efficiency in 3D lane detection.
More qualitative results of BézierFormer are in the Appendix.















 §.§ Ablation Studies

In this section, we conduct all the studies with ResNet18.




  §.§.§ Lane Representation


To demonstrate the superiority of Bézier curve in lane detection, we compare it with polynomial and row-based representations.
The polynomial and row-based representations are referenced to LSTR<cit.> and CurveFormer<cit.>, respectively. 
For fairness, we adapt these representations into an architecture akin to BézierFormer, detailed in the Appendix, and use advanced LineIoU loss <cit.> for better performance. 
Table <ref> reveals that Bézier curve yields much better results. 



















































  §.§.§ Attention mechanism

To validate the superiority of our Bézier curve attention mechanism in lane detection, we compare it with ordinary attention <cit.> and vanilla deformable attention.
Vanilla deformable attention only adaptively generates one reference point, so Bézier curve attention uses t=0.5 to sample one reference point for fairness.
As shown in Table <ref>, Bézier curve attention mechanism obviously brings better performance through explicitly sampling reference points.
Figure <ref> also illustrates that Bézier curve attention has more precise attention locations than the other two attention mechanisms.


































































  §.§.§ Loss Function

On CurveLanes, we compare our Chamfer IoU-based regression loss with the sampling loss <cit.> for Bézier curve fitting. 
For fairness, we normalize the sampling distance between curves' points to IoU and get sampling IoU loss. 
Table <ref> indicates that our regression loss performs better because Chamfer IoU measures overall shape and location similarity between curves better.




























§ CONCLUSION

In this work, we introduce BézierFormer, a novel unified 2D/3D lane detection architecture. 
It employs Bézier control point queries to represent lane curves and uses a novel Bézier curve attention mechanism to accurately and comprehensively extract lane features.
Besides, we propose the Chamfer IoU-based regression loss to improve the performance.
Finally, its state-of-the-art performance on widely-used 2D and 3D lane detection benchmarks attests to BézierFormer's effectiveness in both 2D and 3D lane detection and suggests that further exploration would be valuable.











§ APPENDIX



 §.§ Implementation Details

We implement BézierFormer based on MMDetection <cit.>.
For reproducibility, we give the detailed experimental settings, including hyperparameters of the training and testing process, data augmentation strategies, and network settings of BézierFormer in Table <ref>.









 §.§ Illustration of Loss Function


For a more intuitive understanding of the advantage of Chamfer IoU-based lane regression loss, Figure <ref> visualizes different losses. 
Figure <ref>(c) shows that the distances between two curves' endpoints are relatively larger than other locations, contributing to a more effective learning process compared with Figure <ref>(a) and Figure <ref>(b).










 §.§ Poly and Row-based Curve Decoder

For a fair comparison with polynomial and row-based lane representation, we employ the same feature extractor and design delicated decoders for them, similar with Bézier curve decoder as shown in Figure <ref>. 

For polynomial representation, we refer to LSTR <cit.> and use (k,f,m,n,b^',b^”,α ,β) to formulate a curve. LSTR uses vertical y as variable to describe a curve, k,f,m,n,b^',b^” is the polynomial parameters, α ,β represents the y coordinates of endpoints. The formula is:

    x=k/(y-f )^2 + m/(y-f ) + b^'× y-b^”

To sample the reference points for Poly Curve Attention, which has the same equation with Bézier Curve Attention, we selects five equally-spaced y coordinates:

    y_i=α + β - α/4· i, i=0,1,2,3,4

Thus, we can get the reference points of Poly Curve Attention according to Eq.(<ref>). Finally, the reference points are:


    ( x_i,y_i ), i=0,1,2,3,4


For row-based representation, we refer to Laneformer <cit.>. Laneformer formulates a lane curve as (x_1,x_2,...,x_72,y_start,y_end ), where (x_1,x_2,...,x_72 ) are the x coordinates for the 72 equally-spaced y coordinates, and y_start,y_end denote for the start y coordinate and end y coordinate of the curve. Row-based Curve Attention also shares the same equation with Bézier Curve Attention, but has a different reference points sampling method. To get the five reference points, we compute five equally-spaced indexes of x according to y_start and y_end, as follows.



    idx_i=⌊ y_start + y_end-y_start/4· i  ⌋, i=0,1,2,3,4


Then we can get the five reference points:

    ( x_idx_i,idx_i ), i=0,1,2,3,4







 §.§ Results on CULane

CULane <cit.> is also a famous large-scale 2D lane detection dataset including nine challenging scenarios on highways and urban roads. CULane's challenge lies in various difficult visual scenes. Although BézierFormer is not designed with this challenge in mind, it still slightly outperforms methods which are proposed to address this challenge, such as CLRNet <cit.>, on CULane. 






 §.§ Qualitative Results

To have an intuitive understanding of the performance of BézierFormer, we give the qualitative results on CULane <cit.>, CurveLanes <cit.> and OpenLane <cit.>.
As shown in Figure <ref>, BézierFormer is robust in challenging scenarios like Highlight, Night, Crowd, Arrow, Curve, Shadow, and No Line.
Figure <ref> shows the results on CurveLanes and indicates that  BézierFormer performs well in complex topologies like merged lanes, forked lanes, curves lanes, dense lanes, and nearly horizontal lanes.
Figure <ref> draws the results on OpenLane and proves the effectiveness of BézierFormer's 3D lane detection in different scenes.












IEEEbib


