SOVO: Usability Questionnaire for Voice-Only User Interfaces
Keywords:
Questionnaire, SOVO, Usability, Voice user interfacesAbstract
This article presents the creation and validation of the SOVO instrument, which measures perceived usability by users. To achieve this, we conducted a three-stage instrument study. Twenty out of 160 items from nine questionnaires used to assess usability in VUI (Voice User Interfaces) were selected. The researchers established factors and the type of response. Expert judges supported the application of CVC, and after two rounds, the instrument had 15 items, with eight items rated as good and seven as excellent. Next, we administered the SOVO instrument to a sample of 314 users and conducted a statistical analysis, which validated the instrument. We found that the items demonstrate high internal consistency, with an alpha of 0.96 and a lambda-6 of 0.97. Finally, we performed an exploratory factor analysis, identifying three factors: Likeability, Intelligibility, and Usability. It is important to have reliable instruments since it is very common to see VUI in a wide range of the users’ activities.
Downloads
References
B. Thormundsson, “Voice technology - statistics & facts | Statista.” https://www.statista.com (accessed May 16, 2023).
F. Laricchia, “Number of voice assistants in use worldwide 2019-2024 | Statista.” https://www.statista.com/ (accessed May 16, 2023).
F. Paz and J. A. Pow-Sang, “Usability Evaluation Methods for Software Development: A Systematic Mapping Review,” ASEA 2015, vol. 10, no. 1, pp. 1–4, 2016, doi: 10.1109/ASEA.2015.8.
M. Hassenzahl, M. Burmester, and F. Koller, “AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität,” 2003, doi: 10.1007/978-3-322-80058-9_19.
M. Hernández-Campos, J. Thomaschewski, and Y. C. Law, “Results of a Study to Improve the Spanish Version of the User Experience Questionnaire (UEQ),” International Journal of Interactive Multimedia and Artificial Intelligence, vol. InPress, no. InPress, 2022, doi: 10.9781/ijimai.2022.11.003.
M. D. Polkosky and J. R. Lewis, “Expanding the MOS: Development and psychometric evaluation of the MOS-R and MOS-X,” Int J Speech Technol, vol. 6, no. 2, 2003, doi: 10.1023/A:1022390615396.
K. Hone, “Usability measurement for speech systems: SASSI revisited,” Proceedings of CHI, no. 1, 2014.
J. F. Quesada Moreno, Z. Callejas Carrión, and D. Griol Barres, “Informe sobre sistemas conversacionales multimodales multilingues,” 2019. [Online]. Available: https://www.plantl.gob.es/
A. Mhaidli, M. K. Venkatesh, Y. Zou, F. Schaub, and M. Kandadai, “Listen Only When Spoken To: Interpersonal Communication Cues as Smart Speaker Privacy Controls,” Proceedings on Privacy Enhancing Technologies, 2020, doi: 10.2478/popets-2020-0026.
S. Ruan, J. O. Wobbrock, K. Liou, A. Ng, and J. Landay, “Speech Is 3x Faster than Typing for English and Mandarin Text Entry on Mobile Devices,” 2016.
D. Ramos, “Voice Assistants” Smartsheet.com, 2018. https://www.smartsheet.com/voice-assistants-artificial-intelligence (accessed Oct. 31, 2020).
D. A. Coates, Voice Aplications for Alexa y Google Assistant. Editorial Manning, 2019.
C. Wei and J. Finkelstein, “Comparison of Alexa Voice and Audio Video Interfaces for Home-Based Physical Telerehabilitation”, AMIA, pp.496, 2022.
J. Nielsen, “Usability inspection methods,” CHI ’94, pp. 413–414, 1994, doi: 10.1145/259963.260531.
S. L. Hura, “Usability Testing of Spoken Conversational Systems,” Journal of Usability Studies vol. 12, pp. 155–163, 2017.
Z. Wei and J. A. Landay, “Evaluating Speech-Based Smart Devices Using New Usability Heuristics,” IEEE Pervasive Computing, vol. 17, no. June, pp. 84–96, 2018, doi: 10.1109/MPRV.2018.022511249.
S. Atreja et al., “How Do People Interact in Conversational Speech-Only Search Tasks : A Preliminary Analysis,” Univers Access Inf Soc, vol. 1, no. 1, pp. 1–12, 2018, doi: 10.1145/2160601.2160619.
A. Teixeira et al., “Design and development of Medication Assistant: older adults centred design to go beyond simple medication reminders,” Univers Access Inf Soc, vol. 16, no. 3, pp. 545–560, 2017, doi: 10.1007/s10209-016-0487-7.
J. R. Lewis and M. L. Hardzinski, “Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire,” Int J Speech Technol, vol. 18, no. 3, pp. 479–487, 2015, doi: 10.1007/s10772-015-9289-1.
M. Walker, C. Kamm, and D. Litman, “Towards developing general models of usability with PARADISE,” Nat Lang Eng, vol. 6, 2000, doi: https://doi.org/10.1017/s1351324900002503.
A. L. Iniguez-Carrillo, L. S. Gaytan-Lugo, M. A. Garcia-Ruiz, and R. Maciel-Arellano, “Usability Questionnaires to Evaluate Voice User Interfaces,” IEEE Latin America Transactions, vol. 19, no. 9, pp. 1468–1477, 2021, doi: 10.1109/TLA.2021.9468439.
D. Ghosh, P. S. Foong, S. Zhang, and S. Zhao, “Assessing the utility of the system usability scale for evaluating voice-based user interfaces,” ACM International Conference Proceeding Series, pp. 11–15, 2018, doi: 10.1145/3202667.3204844.
D. S. Zwakman, D. Pal, and C. Arpnikanondt, “Usability Evaluation of Artificial Intelligence-Based Voice Assistants: The Case of Amazon Alexa,” SN Comput Sci, vol. 2, no. 1, p. 28, 2021, doi: 10.1007/s42979-020-00424-4.
L. B. Larsen, “Assessment of spoken dialogue system usability - What are we really measuring?,” EUROSPEECH 2003, pp. 1945–1948, 2003.
ISO, “Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts, ISO 9241-11:2018(en),” 2018. https://www.iso.org/
C. Murad, C. Munteanu, L. Clark, and B. R. Cowan, “Design guidelines for hands-free speech interaction,” MobileHCI 2018, pp. 269–276, 2018, doi: 10.1145/3236112.3236149.
M. Gao, P. Kortum, and F. L. Oswald, “Multi-Language Toolkit for the System Usability Scale,” Int J Hum Comput Interact, vol. 36, no. 20, pp. 1883–1901, 2020, doi: 10.1080/10447318.2020.1801173.
L. G. Juárez-Hernández and S. Tobón, “Análisis de los elementos implícitos en la validación de contenido de un instrumento de investigación,” Revista Espacios, vol. 39, no. 53, pp. 1–23, 2018.
R. Hernández-Nieto, Instrumentos de Recolección de Datos en Ciencias Sociales y Ciencias Biomédicas. Universidad de los Andes, 2011.
J. Sauro and J. R. Lewis, “When designing usability questionnaires, does it hurt to be positive?,” in CHI ’11, NY, USA, ACM Press, 2011, p. 2215. doi: 10.1145/1978942.1979266.
R Core Team, “R: A language and environment for statistical computing,” 2020. https://www.r-project.org/ (accessed May 16, 2023).
W. Revelle, “Procedures for Psychological, Psychometric, and Personality Research [R package psych version 2.3.3],” 2023, Accessed: May 16, 2023. [Online]. Available: https://CRAN.R-project.org/
F. P. Holgado–Tello, S. Chacón–Moscoso, I. Barbero–García, and E. Vila–Abad, “Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables,” Qual Quant, vol. 44, no. 1, pp.153–166, 2010, doi:10.1007/s11135-008-9190-y.
H. F. Kaiser, “A Second Generation Little Jiffy”, Psychometrika, vol. 35, pp. 401–415, 1970. doi: 10.1007/BF02291817
J. L. Horn, “A Rationale and Test for the Number of Factors in Factor Analysis,” 1965. doi: 10.1007/BF02289447.
L. Fabrigar, D. Wegener, R. MacCallum, and E. Strahan, “Evaluating the use of exploratory factor analysis in psychological research.,” Psychol Methods, vol. 4, 1999, doi: 10.1037/1082-989X.4.3.272.
B. R. Gaines, M. L. Shaw, and L. L. Chen, “Utility, Usability and Likeability: Dimensions of the Net and Web,” 1996. https://algo.informatik.uni-freiburg.de/ (accessed May 16, 2023).
M. Coppens, H. Terband, A. Snik, and B. Maassen, “Speech Characteristics and Intelligibility in Adults with Mild and Moderate Intellectual Disabilities,” Folia Phoniatrica et Logopaedica, vol. 68, no. 4, pp. 175–182, 2017, doi: 10.1159/000450548.
F. Miyara, “El ruido y la inteligibilidad de la palabra”, 2004.https://www.fceia.unr.edu.ar/acustica/biblio/inteligibilidad.pdf (accessed May 10, 2023).
M. P. F. Orlando, C. A. E. Andrea, and F. I. D. Marcela, “Tools evaluation for speech recognition based on domain ontologies over the android platform,” in COLCOM 2012, doi: 10.1109/ColComCon.2012.6233653.
N. Bevan, “Classifying and selecting UX and usability measures,” VUUM, pp. 13-18, 2008.
L. Fulfagar, A. Gupta, A. Mathur, and A. Shrivastava, “Development and Evaluation of Usability Heuristics for Voice User Interfaces,” in Design for Tomorrow, A. Chakrabarti, R. Poovaiah, P. Bokil, and V. Kant, Eds., Singapore: Springer, 2021, pp. 375–385.
W. H. Finch, “Using Fit Statistic Differences to Determine the Optimal Number of Factors to Retain in an Exploratory Factor Analysis,” Educ Psychol Meas, vol. 80, no. 2, pp. 217–241, Apr. 2020, doi: 10.1177/0013164419865769.
