ImmersiveSHAP: Immersive analytics visualization system for XAI using SHAP scatter plot
Keywords:
Immersive analytics, Explainable Artificial Intelligence, SHAP, Virtual Reality, Immersive visualizationAbstract
This study presents ImmersiveSHAP, an immersive analytics visualization system for explainable artificial intelligence that leverages a modular pipeline to render SHAP scatter dependence plots in virtual reality. The research contribution is the integration of explainable artificial intelligence (SHAP Python library) with immersive analytics in a virtual reality environment (Unity). The system integrates a Python-based preprocessing module that handles data loading, model training, and SHAP explanation computation, and a Unity-based rendering and interaction module, implemented within a client–server architecture. The WebSocket protocol establishes communication between the Python server and the Unity client. The system extends traditional 2D SHAP plots into interactive 3D visualizations, designed to support immersive analytics with post hoc model explanations. Furthermore, a technical validation used the Iris, Breast Cancer, and California Housing datasets, covering point clouds from N =150 to N =20,640, and deployed the system on a Meta Quest 3. Results identify operational constraints, showing stable performance on small-to-medium datasets (N ≤ 2,000) with an average frame rate of approximately 70 FPS, close to the device’s refresh rate target and within acceptable ranges for virtual reality. These results indicate the system’s viability as a baseline architecture for immersive visualization of SHAP-based explanations.
Downloads
References
D. Gunning, “Darpa’s explainable artificial intelligence (xai) program,” in Proceedings of the 24th International Conference on Intelligent User Interfaces, ser. IUI ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. ii, DOI: 10.1145/3301275.3308446.
S. Ali, T. Abuhmed, S. El-Sappagh, K. Muhammad, J. M. AlonsoMoral, R. Confalonieri, R. Guidotti, J. Del Ser, N. D´ıaz-Rodr´ıguez, and F. Herrera, “Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence,” Information Fusion, vol. 99, p. 101805, 2023, DOI: 10.1016/j.inffus.2023.101805.
G. Vilone and L. Longo, “Notions of explainability and evaluation approaches for explainable artificial intelligence,” Information Fusion, vol. 76, pp. 89–106, 2021, DOI: 10.1016/j.inffus.2021.05.009.
M. R. Islam, M. U. Ahmed, S. Barua, and S. Begum, “A systematic review of explainable artificial intelligence in terms of different application domains and tasks,” Applied Sciences, vol. 12, no. 3, 2022, DOI: 10.3390/app12031353.
A. Saranya and R. Subhashini, “A systematic review of Explainable Artificial Intelligence models and applications: Recent developments and future trends,” Decision Analytics Journal, vol. 7, 2023, DOI: 10.1016/j.dajour.2023.100230.
A. B. Arrieta, N. D´ıaz-Rodr´ıguez, J. D. Ser, A. Bennetot, S. Tabik,Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins,R. Chatila, and F. Herrera, “Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,” Information Fusion, vol. 58, pp. 82–115, 6 2020, DOI: 10.1016/j.inffus.2019.12.012.
G. Vilone and L. Longo, “Classification of explainable artificial intelligence methods through their output formats,” Machine Learning and Knowledge Extraction, vol. 3, pp. 615–661, 9 2021, DOI: 10.3390/make3030032.
S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Muller, and¨ W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” PLOS ONE, vol. 10, no. 7, pp. 1–46, 07 2015, DOI: 10.1371/journal.pone.0130140.
M. T. Ribeiro, S. Singh, and C. Guestrin, “”why should i trust you?”: Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY, USA: Association for Computing Machinery, 2016, pp. 1135–1144, DOI: 10.1145/2939672.2939778.
J. H. Friedman, “Greedy function approximation: A gradient boosting machine.” Annals of Statistics, vol. 29, pp. 1189–1232, 2001, DOI: 10.1214/aos/1013203451.
J. B. A. G. A. Kapelner and E. Pitkin, “Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation,” Journal of Computational and Graphical Statistics, vol. 24, pp. 44–65, 2015, DOI: 10.1080/10618600.2014.907095.
S. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” 2017, DOI: 10.48550/arXiv.1705.07874.
G. Alicioglu and B. Sun, “A survey of visual analytics for explainable artificial intelligence methods,” Computers Graphics, vol. 102, pp. 502– 520, 2022, DOI: 10.1016/j.cag.2021.09.002.
R. Yu and L. Shi, “A user-based taxonomy for deep learning visualization,” Visual Informatics, vol. 2, pp. 147–154, 2018, DOI:10.1016/j.visinf.2018.09.001.
T. Chandler, M. Cordeil, T. Czauderna, T. Dwyer, J. Glowacki, C. Goncu, M. Klapperstueck, K. Klein, K. Marriott, F. Schreiber, and E. Wilson, “Immersive analytics,” in 2015 Big Data Visual Analytics (BDVA), 2015, pp. 1–8, DOI: 10.1109/BDVA.2015.7314296.
C. Linse, H. Alshazly, and T. Martinetz, “A walk in the black-box: 3d visualization of large neural networks in virtual reality,” Neural Computing and Applications, vol. 34, pp. 21237–21252, 12 2022, DOI: 10.1007/s00521-022-07608-4.
A. Aamir, M. Tamosiunaite, and F. Worg¨ otter, “Caffe2unity: Immersive¨ visualization and interpretation of deep neural networks,” Electronics, vol. 11, no. 1, 2022, DOI: 10.3390/electronics11010083.
C. Maathuis, M. A. Cidota, D. Datcu, and L. Marin, “Integrating explainable artificial intelligence in extended reality environments: A systematic survey,” Mathematics, vol. 13, no. 2, 2025, DOI: 10.3390/math13020290.
M. Inkarbekov, R. Monahan, and B. A. Pearlmutter, “Visualization of ai systems in virtual reality: A comprehensive review,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 8, 2023, DOI: 10.14569/ijacsa.2023.0140805.
E. H. Korkut and E. Surer, “Visualization in virtual reality: a systematic review,” Virtual Real., vol. 27, no. 2, p. 1447–1480, Jan. 2023, DOI: 10.1007/s10055-023-00753-8.
J. A. Wagner Filho, M. F. Rey, C. M. D. S. Freitas, and L. Nedel, “Immersive visualization of abstract information: An evaluation on dimensionally-reduced data scatterplots,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2018, pp. 483–490, DOI: 10.1109/VR.2018.8447558.
M. Benk, R. P. Weibel, S. Feuerriegel, and A. Ferrario, “”is it my turn?”: Assessing teamwork and taskwork in collaborative immersive analytics,” Proc. ACM Hum.-Comput. Interact., vol. 6, 11 2022, DOI: 10.1145/3555580.
S. Bobek, S. K. Tadeja, Łukasz Struski, P. Stachura, T. Kipouros, J. Tabor, G. J. Nalepa, and P. O. Kristensson, “Virtual reality-based parallel coordinates plots enhanced with explainable ai and data-science analytics for decision-making processes,” Applied Sciences, vol. 12, 2022, DOI: 10.3390/app12010331.
B. Ens, B. Bach, M. Cordeil, U. Engelke, M. Serrano, W. Willett, Prouzeau, C. Anthes, W. Buschel, C. Dunne, T. Dwyer, J. Grubert,¨ J. H. Haga, N. Kirshenbaum, D. Kobayashi, T. Lin, M. Olaosebikan, F. Pointecker, D. Saffo, N. Saquib, D. Schmalstieg, D. A. Szafir, M. Whitlock, and Y. Yang, “Grand challenges in immersive analytics,” in Conference on Human Factors in Computing Systems Proceedings. Association for Computing Machinery, 5 2021, DOI: 10.1145/3411764.3446866.
R. Hackathorn and T. Margolis, “Immersive analytics: Building virtual data worlds for collaborative decision support,” in 2016 Workshop on Immersive Analytics (IA), 2016, pp. 44–47, DOI: 10.1109/IMMERSIVE.2016.7932382.
A. Fonnet and Y. Prie, “Survey of immersive analytics,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, pp. 2101–2122, 3 2021, DOI: 10.1109/TVCG.2019.2929033.
M. Kraus, J. Fuchs, B. Sommer, K. Klein, U. Engelke, D. Keim, and F. Schreiber, “Immersive analytics with abstract 3d visualizations: A survey,” Computer Graphics Forum, vol. 41, pp. 201–229, 2 2022, DOI: 10.1111/cgf.14430.
J. Montilla-Lopez and G. Ramíırez-Gonzalez, “Análisis inmersivo para´ inteligencia artificial explicable: un enfoque conceptual,” in 2025 IEEE Colombian Conference on Communications and Computing (COLCOM), 2025, pp. 1–6, DOI: 10.1109/COLCOM66267.2025.11185789.
F. A. Pedroso and P. D. P. Costa, “Immvis: Bridging data analytics and immersive visualisation,” in VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 3. SciTePress, 2021, pp. 181–187, DOI: 10.5220/0010256001810187.
S. Lundberg, “Shap: A game theoretic approach to explain the output of any machine learning model,” 2018, gitHub repository. Accessed: 2026-01-09. [Online]. Available: https://github.com/shap/shap
A. V. Ponce-Bobadilla, V. Schmitt, C. S. Maier, S. Mensing, and S. Stodtmann, “Practical guide to shap analysis: Explaining supervised machine learning model predictions in drug development,” Clinical and Translational Science, vol. 17, no. 11, p. e70056, 2024, DOI: 10.1111/cts.70056.
T. Dwyer, K. Marriott, T. Isenberg, K. Klein, N. Riche, F. Schreiber, W. Stuerzlinger, and B. H. Thomas, Immersive Analytics: An Introduction. Cham: Springer International Publishing, 2018, pp. 1–23, DOI: 10.1007/978-3-030-01388-21.
M. O. Anwer and E. Zagrouba, “A comprehensive evaluation framework for Virtual Reality applications in biological data visualization: Metrics, validation, and expert assessment,” Jul. 2025, DOI: 10.21203/rs.3.rs7132029/v1.
A. Geris, B. Cukurbasi, M. Kilinc, and O. Teke, “Balancing performance and comfort in virtual reality: A study of fps, latency, and batch values,” Software: Practice and Experience, vol. 54, no. 12, pp. 2336–2348, 2024, DOI: 10.1002/spe.3356.
M. Cordeil, A. Cunningham, B. Bach, C. Hurter, B. H. Thomas, K. Marriott, and T. Dwyer, “Iatk: An immersive analytics toolkit,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019, pp. 200–209, DOI: 10.1109/VR.2019.8797978.
R. Sicat, J. Li, J. Choi, M. Cordeil, W.-K. Jeong, B. Bach, and H. Pfister, “Dxr: A toolkit for building immersive data visualizations,” IEEE Transactions on Visualization and Computer Graphics, vol. 25, pp. 715–725, 1 2019, DOI: 10.1109/TVCG.2018.2865152.
J Montilla-Lopez, “Immersiveshap,” 2026, gitHub repository. [Online].´ Available: https://github.com/JulianaMontillaLopez/ImmersiveSHAP
R. A. FISHER, “The use of multiple measurements in taxonomic problems,” Annals of Eugenics, vol. 7, no. 2, pp. 179–188, 1936, DOI: 10.1111/j.1469-1809.1936.tb02137.x.
W. N. Street, W. H. Wolberg, and O. L. Mangasarian, “Nuclear feature extraction for breast tumor diagnosis,” in Biomedical Image Processing and Biomedical Visualization, R. S. Acharya and D. B. Goldgof, Eds., vol. 1905, International Society for Optics and Photonics. SPIE, 1993, pp. 861 – 870, DOI: 10.1117/12.148698.
R. Kelley Pace and R. Barry, “Sparse spatial autoregressions,” Statistics Probability Letters, vol. 33, no. 3, pp. 291–297, 1997, DOI: 10.1016/S0167-7152(96)00140-X.
E. Dreyer, “Nativewebsocket: A simple, fast websocket library for unity and webgl,” 2026, gitHub repository. Accessed: 2026-01-09. [Online]. Available: https://github.com/endel/NativeWebSocket
Meta Horizon Developers, “Meta horizon documentation: Unity — meta quest developer hub (mqdh),” 2026, official online documentation. Accessed: 2026-01-09. [Online]. Available: https://developers.meta. com/horizon/documentation/unity/ts-mqdh/
Unity Technologies, “Android logcat package documentation (com.unity.mobile.android-logcat@1.4),” 2026, official Unity documentation. Accessed: 2026-01-09. [Online]. Available: https://docs.unity3d. com/Packages/com.unity.mobile.android-logcat@1.4/manual/index.html
