ESDMotion: End-to-end Motion Prediction Only with SD Maps
motion prediction
Yes
No ethics review needed.
What will the trajectory prediction performance be like if the online predicted SD-map is used as input instead of the offline SD-map?
1.This paper is the first to propose and validate the use of SD-Map for end-to-end trajectory prediction tasks. It achieves performance that is comparable to, or even surpasses, approaches utilizing online predicted HD-map.
2.The proposed strategy effectively reduces the trajectory prediction performance gap between using SD-map and HD-map.
This paper proposes an end-to-end trajectory prediction approach based on SD-map, termed ESDMotion, and validates its effectiveness across two common trajectory prediction paradigms: anchor-based and anchor-free schemes. The core contributions include an Enhanced Road Observation strategy to facilitate interaction between agent and SD-map features, as well as a Pseudo Lane Expansion approach to address the insufficient sampling of goal points in anchor-based methods by broadening the sampling range. Experiments on the nuScenes dataset shows the performance improvements in trajectory prediction using SD-map, narrowing the gap with the strategies that rely on HD maps.
1.The Pseudo Lane Expansion strategy proposed is a relatively coarse goal point sampling approach, as the generated pseudo lanes may not align with the actual road structure, potentially resulting in out-of-bound instances and other inaccuracies.
2.In the experimental section, validation is lacking for the performance of replacing SD-map with online HD-map predictions from models such as MapTRV2, combined with the proposed strategies in this study. Including this experiment would greatly enhance the validity of the paper's findings.

Limitations

Although HD-maps are not required, this study still relies on offline-generated SD-Map, which limits its applicability. Additionally, for anchor-based trajectory prediction approaches, the proposed strategy does not demonstrate a performance advantage over methods that utilize online-predicted maps, highlighting a limitation of this approach.
Yes
No ethics review needed.
N/A
1. This paper proposes an end-to-end motion prediction framework that relies solely on SD maps, addressing the scalability and cost issues associated with HD maps.
2. This paper introduces Enhanced Road Observation and Pseudo Lane Expansion to enhance feature fusion and anchor generation for SD maps, addressing the limitations of using SD maps in motion prediction.
3. The proposed ESDMotion obtains comparable or even better performance compared to methods based on HD maps or GT maps.
The paper presents ESDMotion, an end-to-end motion prediction framework that uses only standard-definition (SD) maps for autonomous driving motion prediction to address the limitations of high-definition maps. It integrates BEV features from raw sensor data into existing motion prediction models and presents Enhanced Road Observation and Pseudo Lane Expansion to handle the challenges brought by SD maps. Experiments on the nuScenes dataset show competitive performance compared to online mapping-based methods and mitigate the performance gap between HD and SD maps.
1. Although the proposed approach ESDMotion obtains improvements in Tab.1&2, the baseline methods (DenseTNT and HiVT) are too old. It will be better to adopt the latest methods.
2. The proposed ESDMotion aims for end-to-end motion prediction however it still relies on SD maps as extra inputs, which is different from previous end-to-end works[1,2].
3. The use of SDMap feature fusion as a way to enhance BEV features is already common; the authors focus on fusing BEV features and SDMap prior features around the agent, which is innovative but lacks sufficient novelty.
4. The nuScenes dataset mostly contains simple road structure scenarios (PARA-Drive, BEVPlanner), in some complex scenarios, will the ESDMotion scheme proposed by the authors perform more excellently, for example, compared with PARA-Drive?


References\
[1] Jiang et al. Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction. arXiv 2022.\
[2] Gu et al. ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries. CVPR 2023.
Yes
No ethics review needed.
1.Although SD maps are globally covered, there are small areas like parking lots (which indeed exist in nuScenes) that are not covered. How does DenseTNT handle such situations?  
2.Why is ESDMotion unable to outperform BEVPred on DenseTNT?
1.Direct and clear motivation with coherent writing.  
2.The approaches of better utilizing SDMap (Pseudo Lane Expansion and Enhanced Road Observation) are interesting.
To address the dependency of trajectory prediction models on high-cost offline high-precision maps or online mapping modules, this paper proposes ESDMotion. ESDMotion uses surround-view images and globally covered SD maps as environmental input for trajectory prediction, and experiments were conducted on nuScenes. The method proposed in this paper to use SDMap is interesting. However, I have concerns about the contributions of this paper and the fairness of some experiments.
1.The paper is misleading (including the title and contribution 1) readers. It does not only rely on SD Map for trajectory prediction but rather combines surround-view images. The surround-view images provide crucial local environmental information, which models like HiVT and DenseTNT do not utilize.  
2.The emphasis on the end-to-end architecture of ESDMotion as a contribution is questionable. With BEVPred and ViP3D's work[1-2] and open-source code as a foundation, training a model end-to-end from images to BEV features to trajectory prediction presents no real challenges or contributions. The contribution of this paper seems lacking. Although the code[3] provided by BEVPred replaces mapping module features with HD maps as input for trajectory prediction, it can easily be modified to enable end-to-end training of the BEV encoder and trajectory prediction model.  
3.From Table 2, it can be seen that for HiVT, the minADE and minFDE using HDMap and SDMap are quite similar. While ESDMotion claims to improve minADE and minFDE through an end-to-end architecture or BEV features, as I mentioned in point 2, I believe this end-to-end architecture should not be counted as a contribution of this paper. This improvement can also be observed in the comparison between BEVPred and offline HDMap in Table 1.


References:  
[1]. Gu X, Song G, Gilitschenski I, et al. Accelerating online mapping and behavior prediction via direct bev feature attention[J]. arXiv preprint arXiv:2407.06683, 2024.  
[2]. Gu J, Hu C, Zhang T, et al. Vip3d: End-to-end visual trajectory prediction via 3d agent queries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5496-5506.  
[3]. https://github.com/alfredgu001324/MapBEVPrediction
Yes
No ethics review needed.
Beyond the need for improved paper presentation and proofreading, answers to the points on `[Pseudo-lane expansion]` and `[Fair baselines]` may lead to a reconsideration of the reviewer’s recommendation.
* Introducing SD-maps, that are readily available with cheap and crowd-sourced maintenance, in trajectory prediction models is a good idea. While some previous works exist to use SD maps as priors to build HD maps, the impact on downstream application (here trajectory prediction) is an underexplored and important topic.

* The proposed modules are motivated, and their instantiation is sound.

* The method is designed for both anchor-based and anchor-free methods.

* The paper is well illustrated.

* The discussion on the contribution differences between SD / HD maps in anchor-based/anchor-free models was appreciated.
The paper presents a method to integrate Standard Definition (SD)-maps information in trajectory prediction models. The model jointly integrates sensor-based features (BEV maps) with SD-maps features with a deformable attention. Additionally, to account from the poor spatial localization of SD-maps, and missing fine-grained details, the method suggests a simple “pseudo-lane expansion” where artificial lanes are created parallelly to the main lane of the SD map. Each of these lanes serves in the deformable attention (“Enhanced Road Observation”) and provides goal candidates in anchor-based methods. Experiments are conducted on nuScenes.
`[Model presentation]` The presentation of the method (section 3) could be significantly improved. In the current form, the model presentation interleaves the general trajectory prediction framework (from previous works), contributions of the work (enhanced road observation, pseudo lane expansion), high-level insights, and technical details (e.g. instantiation of the encoders, etc…). The presentation of the instantiation of the method for either anchor-based or anchor-free models can be also improved. For instance, the title of Section 3.2 targets anchor-free models but “the anchor-based model DenseTNT” is discussed at the end.

`[Pseudo-lane expansion]` DenseTNT already “expands” the lanes by densely sampling candidate goals around the lanes. It looks like the proposed pseudo-lane expansion simply amounts to increasing the size of the sampling kernel in denseTNT. Can the authors comment on this?

`[Prior works]`
There are prior works that try to infer HD online maps from SD-maps and sensor-data [1,2,3]
The reviewer suggests discussing these approaches in the related work section, as it offers another way to integrate SD maps.

* [1] Mind the map! Accounting for existing maps when estimating online HDMaps from sensors. Sun et al. 2023
* [2] Driving with Prior Maps: Unified Vector Prior Encoding for Autonomous Vehicle Mapping, Zeng et al. 2024
* [3] Local map Construction Methods with SD map: A Novel Survey, Li et al. 2024

`[Fair baselines]` The comparison between ESDMotion and “Unc”/”BEVPred” is unfair as ESDMotion uses SD maps which the latter ones do not (lines 369-374). Related to the previous remark, it would be valuable to compare ESDMotion with “Unc” + “online HD mapping based on SD maps” and “BEVPred” + “online HD mapping based on SD maps”.

`[Impact of the Enhanced Road Observation]` The improvement brought by the Enhanced Road Observation module is somewhat limited (Table 4).

`[Map updates]` The review agrees that HD maps must be often updated (l.43). However, I believe the use of SD maps does not solve this problem as SD maps should be updated as well.

`[Multimodal future prediction]` How many modes are generated by the models? (l.362,363).

`[Title]` The use of the word “only” in the title is questionable. There are some works doing end-to-end motion prediction without any maps (SD / HD).

`[Writing clarity]` The many paper typos, syntax and grammar issues hurt the readability of the paper. A thorough proof-read is strongly recommended. A non-exhaustive list of typos are shown below.
* “ground truth” → Hyphen is needed when used as an adjective
* Multiple spaces “ “ are missing, e.g., before citations, parenthesis, after “.” etc…
* Line 220. Is it Sec 4.3?
* Line 334. The subject is missing. “We”?
* Line 413 “tab1”
* Line 296 “an”
* Please refer to Figure 1 and Figure 6 in the text where appropriate.
* Figure 3 is discussed after Figure 4.
* Some notations are introduced but never used (e.g., l.293,335) 
* Opening double quote signs are off
* Illustrations are pixelated, pdf figures are generally preferred as they also allow for text selection and search.
* Tab 3 and 4 should be at the top of the page.
* In the reviewer’s opinion, the usage of bold sentences is too excessive.
We propose a motion prediction method only requiring SD Map, achieving on-par performance with HD Map by end-to-end learning
@inproceedings{
anonymous2024esdmotion,
title={{ESDM}otion: End-to-end Motion Prediction Only with {SD} Maps},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sEJYPiVEt4},
note={under review}
}
Motion prediction is a crucial task in autonomous driving. Existing motion prediction models rely on high-definition (HD) maps to provide environmental context for agents. However, offline HD maps require extensive manual annotation, making them costly and unscalable. Online mapping-based methods still require HD map annotation to train the online mapping module, which is costly as well and may also suffer from the issue of out-of-distribution map elements.
In this work, we explore conducting motion prediction only with standard-definition (SD) maps which are more readily available and offer broader coverage. One crucial challenge is that SD maps have low resolution and poor alignment accuracy. Directly replacing HD maps with SD maps leads to a significant drop in performance. 
We introduce end-to-end learning and specially tailored modules for SD maps to solve the problems. Specifically, we propose ESDMotion, the first end-to-end motion prediction framework that uses only SD maps without any HD map supervision. We integrate BEV features obtained from raw sensor data into existing motion prediction models, with tailored designs for anchor-based and anchor-free models respectively. We find that the coarse and misaligned SD maps bring challenges to feature fusion of anchor-free model and on anchor generation of anchor-based model. Thus, we design two novel modules named Enhanced Road Observation and Pseudo Lane Expansion to address these issues. Benefiting from the end-to-end structure and new modules, ESDMotion outperforms the state-of-the-art online mapping-based motion prediction methods by 13.4% in motion prediction performance and narrows the performance gap between HD and SD maps by 73%. We will open source our code and checkpoints.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
motion prediction
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cdb09ca1cd51d94ae943f691583e4d371201ed82.pdf
applications to robotics, autonomy, planning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ESDMotion: End-to-end Motion Prediction Only with SD Maps
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission