DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving
end-to-end autonomous driving
Yes
No ethics review needed.
* Task parallelism is not the right way to describe this architecture, what is parallel here? 
* The benefit of this architecture needs to be motivated better, open loop results are not considerably better. Closed loop metrics are also mixed. The latency is also comparable. So, what is the benefit of using this architecture?
* What is the criterion for Dev10? 10 scenes out of 220 seem very selective, the criterion needs to be justified, otherwise the results could be prone to selection bias.
The paper proposes a new architecture where all modules interact with each other. This should be able to capture complicated spatio-temporal dependencies among modules. They have also suggested using sensor cross attention and temporal cross attention, this can, at least in theory improve the generalization performance with sensor noise and failures or modules.
This paper proposes a new architecture for end to end autonomous driving where all tasks are connected with others. Task queries and raw sensor features are connected. Also, the task queries are passed as history information for temporal cross attention. Experiments show competitive results with other SOTA architectures in the field.
* The motivation for this architecture is not clear. When all modules interact with each other, we ignore the benefits of using modular end to end architectures. This would make the system difficult to debug and maintain. This can only be justified if the performance is significantly better than the modular architectures, however this does not seem to be the case.
* The results for detection, prediction, mapping is not shown. End to end driving is primarily about planning, however, it makes it more convincing if results of related tasks are also shown to be better.
* It would have useful to see the performance of ParaDrive in closed loop setting, as this architecture is closest in concept with ParaDrive. Only open loop comparison is shown.
* This manuscript could use a grammar / spell check, there are quite a few typos.
Yes
No ethics review needed.
1. In Figure 1, could it be explained why the training stability of b) is poor? And compared to c), why is the task relation of d) okay?
2. The streaming process maintains a FIFO process, retaining and reusing historical information. The BEV temporal fusion also maintains and reuses a fixed-length bev features. If it is a recurrent model, only the previous and next two frames are needed. So the BEV temporal processing of traditional methods can also be regarded as a streaming process. What is special about the streaming process in this paper?
3. In this paper, when comparing performance, training efficiency and inference latency, the comparisons are all made with UniAD and VAD. Is it possible to make a comparison at the same sparse method level with Sparsedrive and SparseAD, also at the same task parallelism method such as ParaDrive?
4. From Table 6c and Table 8, DriveTransformer shows significantly better robustness compared to VAD. Should Table 6c be Table 7?
5. When scaling up in this paper, the largest model used is 0.6b. Would the effect be better if the model were larger?
6. It is mentioned many times in the paper that the sequential model design can cause cumulative errors and training instability. Is it possible to make some comparisons between parallel and sequential methods, especially in terms of cumulative errors and training stability?
● Originality: Proposes a new unified Transformer framework, combining sparse representation with parallelized task processing, distinct from existing methods.
● Quality: The design of each module is reasonable and effectively addresses existing problems.
● Clarity: The paper has a clear structure and it is easy to understand the core ideas.
● Significance: Provides new directions and ideas for autonomous driving research.
This paper presents DriveTransformer, a unified Transformer framework for end-to-end autonomous driving. It features three key properties: task parallelism, sparse representation, and streaming processing, and contains three unified operations: task self-attention, sensor cross-attention, and temporal cross-attention, reducing system complexity and improving training stability. The framework achieves good performance in both the simulated closed-loop Bench2Drive and the open-loop nuScenes benchmarks.
● Originality: Lacking enough original design. Because the task parallel concept is not entirely new (ParaDrive), and the sparse representation (core part following PETR & PETRv2) and streaming process lack strong originality. 
● Comparisons: Lacking comparisons with similar methods in sparse and parallel aspects.
Yes
No ethics review needed.
1. Curious about the scaling ability, the performance seems not saturate, have you tried on larger model or larger data (Bench2Drive full dataset)?
1. It is a simple and effective idea using a vanilla transformer to unify sensor tokens and task queries.  
2. Benefiting from the simple design, the model can scale up to achieve better performance.  
3. The performance on close-loop benchmark is remarkable, surpassing previous methods by a large margin.  
4. The writing and figures are easy to follow and understand.
This paper proposes a new architecture for end-to-end driving, with three key features: task parallelism, sparse representation, streaming processing. With these key features, the proposed method reduces the complexity of the system and leads to better training stability, achieving SOTA performance on both open-loop and close-loop benchmarks.
1. The open-loop experiment should exclude ego-status information to prevent short-cut learning on nuScenes.  
2. There seems no evidence to support that BEV Sequential Paradigm has bad training stability.
We develop a unified Transformer based method for end-to-end autonomous driving without complex operators and achieves SOTA performance.
@inproceedings{
anonymous2024drivetransformer,
title={DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=M42KR4W9P5},
note={under review}
}
End-to-end autonomous driving (E2E-AD) has emerged as a trend in the field of autonomous driving, promising a data-driven, scalable approach to system design. However, existing E2E-AD methods usually adopt the sequential paradigm of perception-prediction-planning, which leads to cumulative errors and training instability. The manual ordering of tasks also limits the system’s ability to leverage synergies between tasks (for example, planning-aware perception and game-theoretic interactive prediction and planning). Moreover, the dense BEV representation adopted by existing methods brings computational challenges for long-range perception and long-term temporal fusion.  To address these challenges, we present DriveTransformer, a simplified E2E-AD framework for the ease of scaling up, characterized  by three key features: Task Parallelism (All agent, map, and planning queries direct interact with each other at each block), Sparse Representation (Task queries direct interact with raw sensor features), and Streaming Processing (Task queries are stored and passed as history information). As a result, the new framework is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention, which significantly reduces the complexity of system and leads to better training stability. DriveTransformer achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS. We will open source our code and checkpoints.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
end-to-end autonomous driving
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d155420d259280f684eb00d43a538c15af6f4e26.pdf
applications to robotics, autonomy, planning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission