Usability Questionnaires to Evaluate Voice User Interfaces

Authors

Keywords:

Questionnaires, Usability, Voice User Interfaces

Abstract

Voice user interfaces (VUI) have been increasingly used in everyday settings and they are growing in popularity. These interfaces have predominantly eyes-free and hands-free interactions. This kind of experiences continues to be an inceptive field compared to other input methods such as touch or using the keyboard/mouse. Thus, it is important to identify tools used to evaluate the usability of VUIs. This article presents a systematic review, in which we analyzed 57 articles and describes nine questionnaires used for evaluating the usability of VUIs, assessing the potential suitability of these questionnaires to measure different types of interactions and various usability dimensions. We found that these questionnaires were used to evaluate the usability of voice-only and voice-added VUIs: AttrakDiff, ICF-US, MOS-X, SUISQ-R, SUS, SASSI, UEQ, PARADISE and USE, where the SUS questionnaire is the most commonly used. However, its items do not directly assess voice quality, although it evaluates the general user interaction with a system. All the questionnaires include items related to three usability dimensions (effectiveness, efficiency, and satisfaction). The questionnaire with the most homogeneous coverage regarding the number of items in each aspect of usability is the SASSI questionnaire. It is a normal practice to use multiple questionnaires to obtain a more complete measurement of usability. We perceive the necessity to increase usability research about the differences between the voice interaction with diverse display types (voice-first, voice-only, voice-added) and the dialog types (command-based and conversational), and how usability affects the user expectations about the VUIs.

Downloads

Download data is not yet available.

Author Biographies

Adriana Lorena Iniguez-Carrillo, Universidad de Guadalajara

Adriana L. Iñiguez is an Assistant Professor at the University of Guadalajara in Mexico. She is a computer systems engineer, has a master's degree in computing and is currently pursuing a doctorate in Information Technology. She collaborates in the CUCEA Smart Cities Innovation Center of the University of Guadalajara. She has professional experience as coordinator of university programs, in the performance of quality and relevance analysis of study plans in the computer science area. Among her main research interests are Human-Computer Interaction, interactions with voice and artificial intelligence.

Laura Sanely Gaytan-Lugo, Universidad de Colima

Profesora Investigadora en la Universidad de Colima en México, en la Facultad de Ingeniería Mecánica y Eléctrica. Se enfoca en la Interacción Humano-Computadora. Sus intereses de investigación incluyen juegos serios para la educación, experiencia de usuario, computación social y accesibilidad. Tiene un doctorado en Tecnologías de Información de la Universidad de Guadalajara y es miembro del Sistema Nacional de Investigadores de México, así como de la Academia Mexicana de Computación. Se desempeña como presidenta de la Comunidad Latina de Interacción Humano-Computadora de la ACM SIGCHI. Así como parte del Comité Ejecutivo de la ACM-W Norteamérica.

Miguel Angel Garcia-Ruiz, Algoma University

Miguel A. Garcia-Ruiz graduated in Computer Systems engineering and obtained his MSc in Computer Science from the University of Colima, Mexico. Miguel received his PhD in Computer Science and Artificial Intelligence from the Centre for Research in Cognitive Science (COGS), University of Sussex, UK. Dr. Miguel Garcia-Ruiz took a virtual reality workshop at Salford University, England, and a computer graphics techniques internship at the Madrid Polytechnic University, Spain. Dr. Miguel Garcia-Ruiz has published various books and more thant 60 scientific papers in academic journals, and has directed a video documentary about an introduction to virtual reality. Dr. Miguel was a Visiting Professor at the Faculty of Business and IT of the  University of Ontario, Canada, where he participated in a research project on combating social deviations with technology, He was an Assistant Professor with the College of Telematics, University of Colima, Mexico. Currently, Miguel is an Associate Professor with the Department of Mathematics and Computer Science from Algoma University, Canada. Dr. Garcia-Ruiz was a member of Mexico's National System of Researchers (SNI) of the National Council of Science and Technology (CONACYT), and was awarded the Innovation Educator of the Year, given by the Sault Ste. Marie Innovation Centre (SSMIC) in 2015.

Rocio Maciel-Arellano, Universidad de Guadalajara

Responsable de vinculación y gestión del talento en el Centro de Innovación en Ciudades Inteligentes de la Universidad de Guadalajara. Tiene amplia experiencia en los procesos de educación, particularmente aquellos en modalidades en línea, ha impulsado dichos programas en la Universidad de Guadalajara y en la Secretaría de Educación Jalisco con más de 10,000 usuarios. Graduada del programa de doctorado y maestría en Administración y Metodología de la enseñanza del Instituto Mexicano de Estudios Pedagógicos IMEP. Tiene más de 25 años de experiencia en Tecnologías de Información, directora de Informática, coordinadora del área de Diseño Gráfico, coordinadora de la licenciatura en Sistemas de Información, consultor jurídico, asesor técnico-pedagógico. Coordinadora del Doctorado en Tecnologías de Información. Profesora de Programas de Doctorado, Maestría y Licenciatura en Tecnologías de Información.

References

V. Research, “Smart Speaker Consumer Adoption Report April 2020,” 2020.

F. Paz and J. A. Pow-Sang, “Usability Evaluation Methods for Software Development: A Systematic Mapping Review,” Proc. - 8th Int. Conf. Adv. Softw. Eng. Its Appl. ASEA 2015, vol. 10, no. 1, pp. 1–4, 2016.

A. Assila, K. Oliveira, and H. Ezzedine, “Standardized Usability Questionnaires: Features and Quality focus.,” Electron. J. Comput. Sci. Inf. Technol., vol. 6, no. 1, pp. 15–31, 2016.

S. Holmes, A. Moorhead, R. Bond, H. Zheng, V. Coates, and M. McTear, “Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?,” ECCE 2019 - Proc. 31st Eur. Conf. Cogn. Ergon’ ’Design Cogn., no. November, pp. 207–214, 2019.

G. Kouroupetroglou and D. Spiliotopoulos, “Usability methodologies for real-life voice user interfaces,” Int. J. Inf. Technol. Web Eng., vol. 4, no. 4, pp. 78–94, 2009.

C. Pearl, Designing Voice User Interfaces: Principles of Conversational Experiences. O’Reilly Media, Inc., 2016.

A. Mhaidli, M. K. Venkatesh, Y. Zou, F. Schaub, and M. Kandadai, “Listen Only When Spoken To: Interpersonal Communication Cues as Smart Speaker Privacy Controls,” Proc. Priv. Enhancing Technol. ..; .. (, vol. 2020, no. 2, pp. 1–20, 2020.

T. Uchiya, R. Nakano, D. Yamamoto, R. Nishimura, and I. Takumi, “Extension with intelligent agents for the spoken dialogue system for smartphones,” 2015 IEEE 4th Glob. Conf. Consum. Electron. GCCE 2015, pp. 281–282, 2016.

A. Pyae and T. N. Joelsson, “Investigating the usability and user experiences of voice user interface: A case of Google home smart speaker,” MobileHCI 2018 - Beyond Mob. Next 20 Years - 20th Int. Conf. Human-Computer Interact. with Mob. Devices Serv. Conf. Proc. Adjun., pp. 127–131, 2018.

M. Braun, A. Mainz, R. Chadowitz, B. Pfleging, and F. Alt, “At your service: Designing voice assistant personalities to improve automotive user interfaces a real world driving study,” Conf. Hum. Factors Comput. Syst. - Proc., pp. 1–11, 2019.

B. Axtell, C. Murad, B. R. Cowan, C. Munteanu, L. Clark, and P. Doyle, “Hey Computer, Can We Hit the Reset Button on Speech?,” Proceedings of the SIGCHI 2018 Workshop on Voice-based Conversational UX Studies and Design, p. 6, 2018.

D. A. Coates, Voice Aplications for Alexa y Google Assistant. Editorial Manning, 2019.

ISO, “Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts, ISO 9241-11:2018(en),” 2018. [Online]. Available: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en.

C. Murad, C. Munteanu, L. Clark, and B. R. Cowan, “Design guidelines for hands-free speech interaction,” Proc. 20th Int. Conf. Human-Computer Interact. with Mob. Devices Serv. Adjun. - MobileHCI ’18, pp. 269–276, 2018.

J. F. Quesada Moreno, Z. Callejas Carrión, and D. Griol Barres, “Informe sobre sistemas conversacionales multimodales multilingues,” Tecnologías y arquitecturas para el desarrollo de asistentes virtuales, sistemas de dialogo y otros interfaces conversacionales. Reporte Técnico, Plan de Impulso de las Tecnologías del Lenguaje., 2019. [Online]. Available: https://www.plantl.gob.es/tecnologias-lenguaje/actividades/Estudios%20tcnicos%20y%20de%20gobernanza/Estudio%20de%20sistemas%20conversacionales/estudio-sistemas-conversacionales.pdf. [Accessed: 28-Mar-2020].

H. R. Hartson, T. S. Andre, and R. C. Williges, “Criteria For Evaluating Usability Evaluation Methods,” Int. J. Hum. Comput. Interact., vol. 15, no. 1, pp. 145–181, 2003.

B. Weiss, I. Wechsung, C. Kühnel, and S. Möller, “Evaluating embodied conversational agents in multimodal interfaces,” Comput. Cogn. Sci., vol. 1, no. 1, p. 6, 2015.

J. R. Lewis, “Standardized Questionnaires for Voice Interaction Design,” Voice Interact. Des., vol. 1, no. 1, pp. 1–16, 2016.

B. Kitchenham, “Procedures for performing systematic reviews,” Keele, UK, Keele Univ., vol. 33, no. TR/SE-0401, p. 28, 2004.

A. M. Methley, S. Campbell, C. Chew-Graham, R. McNally, and S. Cheraghi-Sohi, “PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews,” BMC Health Serv. Res., vol. 14, no. 1, p. 579, 2014.

M. Hassenzahl, “AttrakDiff.” [Online]. Available: http://attrakdiff.de/index-en.html. [Accessed: 24-Mar-2020].

A. I. Martins, A. F. Rosa, A. Queirós, A. Silva, and N. P. Rocha, “Definition and Validation of the ICF - Usability Scale,” Procedia Comput. Sci., vol. 67, no. Dsai, pp. 132–139, 2015.

M. D. Polkosky and J. R. Lewis, “Expanding the MOS: Development and psychometric evaluation of the MOS-R and MOS-X,” Int. J. Speech Technol., vol. 6, no. 2, pp. 161–182, 2003.

A. B. Kocaballi and E. Coiera, “Measuring User Experience in Conversational Interfaces: A Comparison of Six Questionnaires,” Proc. Br. Comput. Soc. Hum. Comput. Interact. Conf. (BCS HCI ’18), no. July, pp. 1–12, 2018.

M. WALKER, C. KAMM, and D. LITMAN, “Towards developing general models of usability with PARADISE,” Nat. Lang. Eng., vol. 6, no. 3&4, pp. 363–377, 2000.

K. Hone, “Usability measurement for speech systems: SASSI revisited,” Proc. CHI, no. 1, 2014.

J. R. Lewis and M. L. Hardzinski, “Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire,” Int. J. Speech Technol., vol. 18, no. 3, pp. 479–487, 2015.

D. Ghosh, P. S. Foong, S. Zhang, and S. Zhao, “Assessing the utility of the system usability scale for evaluating voice-based user interfaces,” ACM Int. Conf. Proceeding Ser., pp. 11–15, 2018.

M. Schrepp, “User Experience Questionnaire Handbook,” 2019. [Online]. Available: https://www.ueq-online.org/Material/Handbook.pdf. [Accessed: 24-Mar-2020].

G. Cordasco et al., “Assessing Voice User Interfaces: The vassist system prototype,” 5th IEEE Int. Conf. Cogn. Infocommunications, CogInfoCom 2014 - Proc., pp. 91–96, 2014.

D. A. Robb, F. J. Chiyah Garcia, A. Laskov, X. Liu, P. Patron, and H. Hastie, “Keep me in the loop: Increasing operator situation awareness through a conversational multimodal interface,” ICMI 2018 - Proc. 2018 Int. Conf. Multimodal Interact., pp. 384–392, 2018.

L. Verde, G. De Pietro, and G. Sannino, “Vox4Health: Preliminary Results of a Pilot Study for the Evaluation of a Mobile Voice Screening Application,” vol. 476, no. ISAmI 2016, pp. 131–140, 2016.

D. Griol and Z. Callejas, “Mobile Conversational Agents for Context-Aware Care Applications,” Cognit. Comput., vol. 8, no. 2, pp. 336–356, 2016.

M. Biermann, E. Schweiger, and M. Jentsch, “Talking to Stupid ?!? Improving Voice User Interfaces,” Fischer, H. & Hess, S., Mensch und Computer 2019 - Usability Professionals. Gesellschaft für Informatik e.V. Und German UPA e.V. no. September, pp. 53–61, 2019. DOI: 10.18420/muc2019-up-0253

P. L. Mateo Navarro, S. Hillmann, S. Möller, D. Sevilla Ruiz, and G. Martínez Pérez, “Run-time model based framework for automatic evaluation of multimodal interfaces,” J. Multimodal User Interfaces, vol. 8, no. 4, pp. 399–427, 2014.

A. Teixeira et al., “Design and development of Medication Assistant: older adults centred design to go beyond simple medication reminders,” Univers. Access Inf. Soc., vol. 16, no. 3, pp. 545–560, 2017.

N. Tractinsky, “The Usability Construct: A Dead End?,” Human-Computer Interact., vol. 33, no. 2, pp. 131–177, 2018.

H. Jung, “Understanding Differences between Heavy Users and Light Users in Difficulties with Voice User Interfaces,” In Proceedings of the 2nd Conference on Conversational User Interfaces (CUI '20). Association for Computing Machinery, New York, NY, USA, Article 51, 1–4. 2020. DOI: https://doi.org/10.1145/3405755.3406170

L. B. Larsen, “Assessment of spoken dialogue system usability - What are we really measuring?,” EUROSPEECH 2003 - 8th Eur. Conf. Speech Commun. Technol., pp. 1945–1948, 2003.

J. Sauro and J. R. Lewis, “Correlations among prototypical usability metrics: evidence for the construct of usability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). Association for Computing Machinery, New York, NY, USA, 1609–1618. 2009. DOI:https://doi.org/10.1145/1518701.1518947

S. Reeves et al., “Voice-based conversational ux studies and design,” in 2018 CHI Conference on Human Factors in Computing Systems, CHI EA 2018, 2018, vol. 2018-April.

B. Cowan, J. Fischer, and L. Clark, “CUI @ CHI: Mapping Grand Challenges for the Conversational User Interface Community,” pp. 1–8, 2020.

Downloads

Published

2021-03-29

How to Cite

Iniguez-Carrillo, A. L., Gaytan-Lugo, L. S., Garcia-Ruiz, M. A., & Maciel-Arellano, R. . (2021). Usability Questionnaires to Evaluate Voice User Interfaces. IEEE Latin America Transactions, 19(9), 1468–1477. Retrieved from https://latamt.ieeer9.org/index.php/transactions/article/view/4771

Issue

Vol. 19 No. 9 (2021): Ordinary Issue

Section

Articles

Most read articles by the same author(s)