MFPAD: Memory–Forgetting Planning for Long-Horizon End-to-End Autonomous Driving
Keywords:
End-to-End Autonomous Driving, Trajectory Prediction, Motion Planning, Long Horizon.Abstract
Recent planning-oriented end-to-end autonomous driving methods have achieved competitive performance on short-horizon (3 s) trajectory prediction. However, their accuracy and stability degrade markedly when the prediction horizon is extended to long-horizon (6 s), leading to drifting trajectories. A key reason is that most existing frameworks adopt simple feed-forward multilayer perceptron regressors in the trajectory refinement head, which lack explicit modeling of long-term temporal dependencies and planning inertia. To address this limitation, we propose the Memory–Forgetting Planning for Autonomous Driving, a plug-in refinement head that combines an LSTM-based memory network with a Transformer-based forgetting network. The memory network autoregressively rolls out a coarse long-horizon trajectory and exposes a sequence of hidden states, while the forgetting network attends over these states and surrounding-agent features with token-level dropout to suppress outdated or noisy motion cues. A lightweight gating module fuses coarse and corrected trajectories at each time step, yielding temporally consistent, interaction-aware plans over a 6 s horizon. We evaluate our method on NuScenes, Adv-NuScenes, Bench2Drive, and NAVSIM, and the results demonstrate consistent improvements. Compared with baselines, it reduces the average collision rate on nuScenes by 11.1% and the 3 s collision rate on Adv-NuScenes by 29.2%, while achieving a 6 s collision rate of 2.01% on NuScenes. In addition, the closed-loop results on Bench2Drive and NAVSIM show that the proposed refinement head also improves downstream driving performance under feedback-driven evaluation. The source code is available at https://github.com/Y1Ka1/MFPAD
Downloads
References
L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li, “End-to-end autonomous driving: Challenges and frontiers,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 46, no. 10, pp. 10 164–10 183, 2024, doi: 10.1109/TPAMI.2024.3435937.
P. S. Chib and P. Singh, “Recent advancements in end-to-end autonomous driving using deep learning: A survey,” IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp. 103–118, 2024, doi:10.1109/TIV.2023.3318070.
L. Le Mero, D. Yi, M. Dianati, and A. Mouzakitis, “A survey on imitation learning techniques for end-to-end autonomous vehicles,” IEEE
Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp.14 128–14 147, 2022, doi: 10.1109/TITS.2022.3144867.
F. Codevilla, M. Müller, A. M. López, V. Koltun, and A. Dosovitskiy, “End-to-end driving via conditional imitation learning,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 4693–4700, doi:
1109/ICRA.2018.8460487.
Z. Gao, Y. Mu, C. Chen, J. Duan, P. Luo, Y. Lu, and S. Eben Li, “Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 10, pp. 13 067–13 079, 2024, doi: 10.1109/TITS.2024.3400227.
C. Sanchez, J. Sanchez, C. Cervantes, R. Ortigoza, V. Guzman, J. Juarez, and M. Aranda, “Trajectory generation for wheeled mobile robots via bézier polynomials,” IEEE Latin America Transactions, vol. 14, no. 11, pp. 4482–4490, 2016, doi: 10.1109/TLA.2016.7795818.
M. J. Reinoso, L. I. Minchala, P. Ortiz, D. F. Astudillo, and D. Verdugo,“Trajectory tracking of a quadrotor using sliding mode control,” IEEE
Latin America Transactions, vol. 14, no. 5, pp. 2157–2166, 2016, doi:10.1109/TLA.2016.7530409.
E. Larralde-Ortiz, A. Luviano-Juárez, F. Mirelez-Delgado, and D. Mercado-Ravell, “Donkietown: a low-cost experimental testbed for research on autonomous cars,” IEEE Latin America Transactions, vol. 21, no. 6, pp. 715–722, 2023, doi: 10.1109/TLA.2023.10172136.
A. Kopacz, E. García González, C. Chira, and J. R. Villar Flecha, “Hybrid adaptive greedy algorithm addressing the multi-robot path
planning problem,” IEEE Latin America Transactions, vol. 23, no. 10, pp. 856–864, 2025, doi: 10.1109/TLA.2025.11150630.
N. S. Monteiro, V. M. Gonçalves, and C. A. Maia, “Motion planning of mobile robots in indoor topological environments using partially
observable markov decision process,” IEEE Latin America Transactions, vol. 19, no. 8, pp. 1315–1324, 2021, doi: 10.1109/TLA.2021.9475862.
M. Mendonça, R. H. C. Palácios, R. Breganon, L. Botoni de Souza, and L. Rodrigues Cintra Moura, “Analysis of the inverse kinematics and
trajectory planning applied in a classic collaborative industrial robotic manipulator,” IEEE Latin America Transactions, vol. 20, no. 3, pp. 363–
, 2022, doi: 10.1109/TLA.2022.9667133.
R. Vieira, E. Argento, and T. Revoredo, “Trajectory planning for car-like robots through curve parametrization and genetic algorithm
optimization with applications to autonomous parking,” IEEE Latin America Transactions, vol. 20, no. 2, pp. 309–316, 2022, doi:
1109/TLA.2022.9661471.
F. Ugalde Pereira, P. Medeiros de Assis Brasil, M. A. de Souza Leite Cuadros, A. R. Cukla, P. Drews Junior, and D. F. Tello Gamarra,
“Analysis of local trajectory planners for mobile robot with robot operating system,” IEEE Latin America Transactions, vol. 20, no. 1,
pp. 92–99, 2022, doi: 10.1109/TLA.2022.9662177.
W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun, “End-to-end interpretable neural motion planner,” in 2019 IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8652–8661, doi: 10.1109/CVPR.2019.00886.
S. Casas, A. Sadat, and R. Urtasun, “Mp3: A unified model to map, perceive, predict and plan,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14 398–14 407, doi: 10.1109/CVPR46437.2021.01417.
A. Sadat, S. Casas, M. Ren, X. Wu, P. Dhawan, and R. Urtasun, “Perceive, predict, and plan: Safe motion planning through interpretable
semantic representations,” in Computer Vision – ECCV 2020, ser.Lecture Notes in Computer Science. Springer International Publishing,
, vol. 12348, pp. 414–430, doi: 10.1007/978-3-030-58592-1-25.
P. Hu, A. Huang, J. Dolan, D. Held, and D. Ramanan, “Safe local motion planning with self-supervised freespace forecasting,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 12 727–12 736, doi: 10.1109/CVPR46437.2021.01254.
S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,” in Computer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature
Switzerland, 2022, pp. 533–549, doi: 10.1007/978-3-031-19839-7-31.
Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y. Qiao, and H. Li, “Planning-oriented autonomous driving,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 17 853–17 862, doi: 10.1109/CVPR52729.2023.01712.
B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang, “Vad: Vectorized scene representation for efficient autonomous driving,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8306–8316, doi:
1109/ICCV51070.2023.00766.
W. Sun, X. Lin, Y. Shi, C. Zhang, H. Wu, and S. Zheng, “Sparsedrive: End-to-end autonomous driving via sparse scene representation,” in 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 8795–8801, doi: 10.1109/ICRA55743.2025.11128800.
Z. Song, C. Jia, L. Liu, H. Pan, Y. Zhang, J. Wang, X. Zhang, S. Xu, L. Yang, and Y. Luo, “Don’t shake the wheel: Momentum-aware planning in end-to-end autonomous driving,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 22 432–22 441, doi: 10.1109/CVPR52734.2025.02089.7
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal
dataset for autonomous driving,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11 618–
628, doi: 10.1109/CVPR42600.2020.01164.
B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, and X. Wang, “Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 12 037–12 047, doi: 10.1109/CVPR52734.2025.01124.
Z. Song, L. Liu, H. Pan, B. Liao, M. Guo, L. Yang, Y. Zhang, S. Xu, C. Jia, and Y. Luo, “Breaking imitation bottlenecks: Reinforced diffusion powers diverse trajectory generation,” arXiv preprint arXiv:2507.04049,
, doi: 10.48550/arXiv.2507.04049.
L. Liu, C. Jia, G. Yu, Z. Song, J. Li, F. Jia, P. Wu, X. Hao, Y. Luo et al., “Guideflow: Constraint-guided flow matching for planning in end-to-end autonomous driving,” arXiv preprint arXiv:2511.18729, 2025, doi:
48550/arXiv.2511.18729.
B. Sun, B. Zhang, J. Lu, X. Feng, J. Shang, R. Cao, M. Zheng, C. Wang, S. Yang, Y. Cao, and Z. Song, “Focalad: Local motion planning for end-to-end autonomous driving,” arXiv preprint arXiv:2506.11419, 2025,doi: 10.48550/arXiv.2506.11419.
Z. Xing, X. Zhang, Y. Hu, B. Jiang, T. He, Q. Zhang, X. Long, and W. Yin, “Goalflow: Goal-driven flow matching for multi-modal trajectories generation in end-to-end autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025, pp. 1602–1611, doi: 10.1109/CVPR52734.2025.00157.
D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta, “Parting with misconceptions about learning-based vehicle motion planning,” arXiv preprint arXiv:2306.07962, 2023, doi: 10.48550/arXiv.2306.07962.
R. Zhao, Y. Fan, Z. Chen, F. Gao, and Z. Gao, “Diffe2e: Rethinking end-to-end driving with a hybrid action diffusion and supervised policy,” 2025, doi: 10.48550/arXiv.2505.19516.
S. Liu, W. Chen, W. Li, Z. Wang, L. Yang, J. Huang, Y. Zhang, Z. Huang, Z. Cheng, and H. Yang, “Bridgedrive: Diffusion bridge policy for closed-loop trajectory planning in autonomous driving,” in
The Fourteenth International Conference on Learning Representations (ICLR), 2026, doi: 10.48550/arXiv.2509.23589.
W. Yao, Z. Li, S. Lan, Z. Wang, X. Sun, J. M. Alvarez, and Z. Wu, “DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 14, 2026, pp. 11 910–11 918, doi: 10.1609/aaai.v40i14.38178.
Z. Li, K. Li, S. Wang, S. Lan, Z. Yu, Y. Ji, Z. Li, Z. Zhu, J. Kautz, Z. Wu, Y.-G. Jiang, and J. M. Alvarez, “Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation,” 2024, doi: 10.48550/arXiv.2406.06978.
Z. Xu, B. Li, H.-a. Gao, M. Gao, Y. Chen, M. Liu, C. Yan, H. Zhao, S. Feng, and H. Zhao, “Challenger: Affordable adversarial driving video generation,” arXiv preprint arXiv:2505.15880, 2025, doi: 10.48550/arXiv.2505.15880.
Z. Li, Z. Yu, S. Lan, J. Li, J. Kautz, T. Lu, and J. M. Alvarez, “Is ego status all you need for open-loop end-to-end autonomous driving?” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 14 864–14 873, doi:10.1109/CVPR52733.2024.01408.
X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan, “Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,” Advances in Neural Information Processing Systems, vol. 37, pp.
–844, 2024, doi: 10.52202/079017-0025.
D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone et al., “Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking,” Advances in Neural Information Processing Systems, vol. 37, pp. 28 706–28 719, 2024, doi: 10.52202/079017-0902.
W. Zheng, R. Song, X. Guo, C. Zhang, and L. Chen, “Genad: Generative end-to-end autonomous driving,” in Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, Eds. Cham: Springer Nature Switzerland, 2025, pp. 87–104, doi: 10.1007/978-3-031-73650-66.
Y. Li, L. Fan, J. He, Y. Wang, Y. Chen, Z. Zhang, and T. Tan, “Enhancing end-to-end autonomous driving with latent world model,” in The Thirteenth International Conference on Learning Representations
(ICLR), 2025, doi: 10.48550/arXiv.2406.08481.
K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger, “Transfuser: Imitation with transformer-based sensor fusion for autonomous driving,” Pattern Analysis and Machine Intelligence (PAMI), 2023, doi:
1109/TPAMI.2022.3200245.
X. Weng, B. Ivanovic, Y. Wang, Y. Wang, and M. Pavone, “PARA-Drive: Parallelized Architecture for Real-Time Autonomous Driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 15 449–15 458, doi: 10.1109/CVPR52733.2024.01463.
L. Yin, R. Ju, G. Guo, and E. Cheng, “Diffrefiner: Coarse to fine trajectory planning via diffusion refinement with semantic interaction for end to end autonomous driving,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 14, pp. 12 009–12 017, 2026, doi: 10.1609/aaai.v40i14.38189.
S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang, “Vadv2: End-to-end vectorized autonomous driving via probabilistic planning,” 2024, doi: 10.48550/arXiv.2402.13243.
D. Li, C. Li, Y. Wang, J. Ren, X. Wen, P. Li, L. Xu, K. Zhan, P. Jia, X. Lang, N. Xu, and H. Zhao, “Learning personalized driving styles via reinforcement learning from human feedback,” 2025, doi: 10.48550/arXiv.2503.10434.
