BSR-NBV: Breaking Stagnation with Random Pose Refinement in Probabilistic Grids for Latent Space Next-Best-View Prediction
Keywords:
Next-best-view, view planning, 3D object reconstruction, probabilistic gridAbstract
The Next-Best-View (NBV) problem is fundamental to autonomous 3D object reconstruction, where the goal is to generate a complete 3D model by sequentially sensing a target’s surface. This paper proposes a novel latent space-based method to address this challenge. Our architecture is trained in two distinct stages: the first stage learns to encode a probabilistic grid into a compact latent representation, while the second stage performs continuous NBV regression. Furthermore, we incorporate a pose refinement mechanism to mitigate stagnation—a phenomenon where the network repeatedly predicts views with low information gain that fail to update the probabilistic grid. Experimental evaluations on the Google Scanned Objects dataset demonstrate that the proposed method achieves superior surface
coverage compared to current state-of-the-art approaches.
Downloads
References
C. Connolly, “The determination of next best views,” in Proceedings.
IEEE international conference on robotics and automation, vol. 2.
IEEE, 1985, pp. 432–435, doi: 10.1109/ROBOT.1985.1087372.
S. Sakane, M. Ish, and M. Kakikura, “Occlusion avoidance of
visual sensors based on a hand-eye action simulator system:
Heaven,” Advanced robotics, vol. 2, no. 2, pp. 149–165, 1987, doi:
1163/156855387X00138.
T. Wang, W. Xi, Y. Cheng, H. Han, and Y. Yang, “Rl-nbv: A deep
reinforcement learning based next-best-view method for unknown object
reconstruction,” Pattern Recognition Letters, vol. 184, pp. 1–6, 2024,
doi: 10.1016/j.patrec.2024.05.014.
Y. Han, I. H. Zhan, W. Zhao, and Y.-J. Liu, “A double branch next-best-
view network and novel robot system for active object reconstruction,”
in 2022 International Conference on Robotics and Automation (ICRA).
IEEE, 2022, pp. 7306–7312, doi: 10.1109/ICRA46639.2022.9811769.
M. Breyer, L. Ott, R. Siegwart, and J. J. Chung, “Closed-loop next-best-
view planning for target-driven grasping,” in 2022 IEEE/RSJ Interna-
tional Conference on Intelligent Robots and Systems (IROS). IEEE,
, pp. 1411–1416, doi: 10.1109/IROS47612.2022.9981472.
M. Naazare, F. G. Rosas, and D. Schulz, “Online next-best-view planner
for 3d-exploration and inspection with a mobile manipulator robot,”
IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3779–3786,
, doi: 10.1109/LRA.2022.3146558.
D. Morrison, P. Corke, and J. Leitner, “Multi-view picking: Next-best-
view reaching for improved grasping in clutter,” in 2019 International
Conference on Robotics and Automation (ICRA). IEEE, 2019, pp.
–8768, doi: 10.1109/ICRA.2019.8793805.
R. Almadhoun, A. Abduldayem, T. Taha, L. Seneviratne, and Y. Zweiri,
“Guided next best view for 3d reconstruction of large complex struc-
tures,” Remote Sensing, vol. 11, no. 20, 2019, doi: 10.3390/rs11202440.
R. Zeng, W. Zhao, and Y.-J. Liu, “Pc-nbv: A point cloud based deep
network for efficient next best view planning,” in 2020 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS).
IEEE, 2020, pp. 7050–7057, doi: 10.1109/IROS45743.2020.9340916.
M. Mendoza, J. I. Vasquez-Gomez, H. Taud, L. E. Sucar, and C. Reta,
“Supervised learning of the next-best-view for 3d object reconstruc-
tion,” Pattern Recognition Letters, vol. 133, pp. 224–231, 2020, doi:
1016/j.patrec.2020.02.024.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of
data with neural networks,” science, vol. 313, no. 5786, pp. 504–507,
L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann,
T. B. McHugh, and V. Vanhoucke, “Google scanned objects: A high-
quality dataset of 3d scanned household items,” in 2022 International
Conference on Robotics and Automation (ICRA). IEEE, 2022, pp.
–2560, doi: 10.1109/ICRA46639.2022.9811809.
R. Zeng, Y. Wen, W. Zhao, and Y.-J. Liu, “View planning in robot active
vision: A survey of systems, algorithms, and applications,” Computa-
tional Visual Media, vol. 6, pp. 225–245, 2020, doi: 10.1007/s41095-
-0179-3.
A. Pop and L. Tamas, “Next best view estimation for volumetric
information gain,” IFAC-PapersOnLine, vol. 55, no. 15, pp. 160–165,
, doi: 10.1016/j.ifacol.2022.07.625.
J. I. Vasquez-Gomez, D. Troncoso, I. Becerra, E. Sucar, and R. Murrieta-
Cid, “Next-best-view regression using a 3d convolutional neural net-
work,” Machine Vision and Applications, vol. 32, no. 2, p. 42, 2021,
doi: 10.1007/s00138-020-01166-2.
S. Pan, H. Hu, H. Wei, N. Dengler, T. Zaenker, M. Dawood, and
M. Bennewitz, “Integrating one-shot view planning with a single next-
best view via long-tail multiview sampling,” IEEE Transactions on
Robotics, vol. 41, pp. 394–414, 2025.
P. Hoseini, S. K. Paul, M. Nicolescu, and M. Nicolescu, “A one-shot next
best view system for active object recognition,” Applied Intelligence,
vol. 52, no. 5, pp. 5290–5309, 2022, doi: 10.1007/s10489-021-02657-z.
C. Collander, W. J. Beksi, and M. Huber, “Learning the next best view
for 3d point clouds via topological features,” in 2021 IEEE International
Conference on Robotics and Automation (ICRA). IEEE, 2021, p. 12207,
doi: 10.1109/ICRA48506.2021.9561389.
G. Gao, M. Lauri, J. Zhang, and S. Frintrop, “Occlusion resistant object
rotation regression from point cloud segments,” in Proceedings of the
European Conference on Computer Vision (ECCV) Workshops, 2019,
doi: 10.1007/978-3-030-11009-3 44.
S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, K. Konolige, G. Bradski,
and N. Navab, “Technical demonstration on model based training, detec-
tion and pose estimation of texture-less 3d objects in heavily cluttered
scenes,” in European Conference on Computer Vision. Springer, 2012,
p. 593, doi: 10.1007/978-3-642-33885-4 60.
S. Hinterstoisser, S. Holzer, C. Cagniart, S. Ilic, K. Konolige, N. Navab,
and V. Lepetit, “Multimodal templates for real-time detection of
texture-less objects in heavily cluttered scenes,” in 2011 international
conference on computer vision. IEEE, 2011, pp. 858–865, doi:
1109/ICCV.2011.6126326.
G. Gao, M. Lauri, X. Hu, J. Zhang, and S. Frintrop, “Cloudaae:
Learning 6d object pose regression with on-line data synthe-
sis on point clouds,” in 2021 IEEE International Conference on
Robotics and Automation (ICRA). IEEE, 2021, p. 11081, doi:
1109/ICRA48506.2021.9561475.
Y. Hu, P. Fua, W. Wang, and M. Salzmann, “Single-stage 6d ob-
ject pose estimation,” in Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, 2020, p. 2927, doi:
1109/CVPR42600.2020.00300.
Z. Dong, S. Liu, T. Zhou, H. Cheng, L. Zeng, X. Yu, and H. Liu,
“Ppr-net: point-wise pose regression network for instance segmentation
and 6d pose estimation in bin-picking scenarios,” in 2019 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS).
IEEE, 2019, pp. 1773–1780, doi: 10.1109/IROS40897.2019.8967895.
Y. Mei, S. Wang, Z. Li, J. Sun, and G. Wang, “Multi-modal 6-dof object
pose tracking: integrating spatial cues with monocular rgb imagery,”
International Journal of Machine Learning and Cybernetics, vol. 16,
no. 2, pp. 1327–1340, 2025, doi: 10.1007/s13042-024-02336-8.
Y. Su, M. Saleh, T. Fetzer, J. Rambach, N. Navab, B. Busam, D. Stricker,
and F. Tombari, “Zebrapose: Coarse to fine surface encoding for 6dof
object pose estimation,” in Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, 2022, p. 6728, doi:
1109/CVPR52688.2022.00662.
C. Wang, D. Xu, Y. Zhu, R. Mart´ın-Mart´ın, C. Lu, L. Fei-Fei, and
S. Savarese, “Densefusion: 6d object pose estimation by iterative dense
fusion,” in Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, 2019, p. 3338, doi: 10.1109/CVPR.2019.00346.
M. Majcher and B. Kwolek, “Deep quaternion pose proposals for 6d
object pose tracking,” in Proceedings of the IEEE/CVF International
Conference on Computer Vision, 2021, pp. 243–251, doi: 10.1109/IC-
CVW54120.2021.00032.
C. Papaioannidis and I. Pitas, “3d object pose estimation using multi-
objective quaternion learning,” IEEE Transactions on Circuits and
Systems for Video Technology, vol. 30, no. 8, pp. 2683–2693, 2020,
doi: 10.1109/TCSVT.2019.2929600.
Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity
of rotation representations in neural networks,” in Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition,
, p. 5738, doi: 10.1109/CVPR.2019.00589.
Wolfram, MathWorld. (2025) Rodrigues’ rotation formula. [Online].
Available: https://mathworld.wolfram.com/RodriguesRotationFormula.
html
Q.-Y. Zhou, J. Park, and V. Koltun, “Open3D: A modern library for 3D
data processing,” 2018, preprint.
