Students' Attention Monitoring System in Learning Environments based on Artificial Intelligence
Keywords:
Early warning system, Learning environment, LSTM, Monitoring, Students' attentionAbstract
The students' attention level to the explanation of a given lecture is a factor that might determine the capability of retention and subsequent application of a learned concept. For this reason, students that pay attention are generally more participatory in the learning/teaching process than those who don't, and consequently, they succeed in reaching the competencies proposed in the courses. Hence, it is important to design strategies and tools that help teachers to monitor in a non-invasive way the attention level of the students, allowing them to take actions to modify the dynamics of the lectures when needed. In this work, we introduce a fully automated system to monitor the students' attention based on computer vision algorithms. To this end, we feed a recurrent neural network with one-second sequences generated by facial landmarks. This spatiotemporal analysis of video recordings allows for identifying when a student is attending a given explanation in online educational environments. The system is tested in a database with more than 3000 sequences of students who pay or no attention to online video lectures. Obtained results show that the proposed system is suitable to monitor the students' attention to a particular explanation.
Downloads
References
E. F. Risko, N. Anderson, A. Sarwal, M. Engelhardt, and A. Kingstone, “Everyday attention: Variation in mind wandering and memory in a lecture,” Applied Cognitive Psychology, vol. 26, no. 2, pp. 234–242, 2012.
D. Dinesh, A. N. S., and K. Bijlani, “Student analytics for productive teaching/learning,” in 2016 International Conference on Information Science (ICIS), 2016, pp. 97–102.
J. B. Heppen and S. B. Therriault, “Developing early warning systems to identify potential high school dropouts. issue brief.” National High School Center, 2008. C. M´arquez-Vera, A. Cano, C. Romero, A. Y. M. Noaman, H. Mousa Fardoun, and S. Ventura, “Early dropout prediction using data mining: a case study with high school students,” Expert Systems, vol. 33, no. 1, pp. 107–124, 2016.
C.-M. Chen, J.-Y. Wang, and C.-M. Yu, “Assessing the attention levels of students by using a novel attention aware system based on brainwave signals,” British Journal of Educational Technology, vol. 48, no. 2, pp. 348–369, 2017.
S. K. D’Mello, S. D. Craig, J. Sullins, and A. C. Graesser, “Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed-initiative dialogue,” International Journal of Artificial
Intelligence in Education, vol. 16, no. 1, pp. 3–28, 2006.
J. Whitehill, Z. Serpell, Y.-C. Lin, A. Foster, and J. R. Movellan, “The faces of engagement: Automatic recognition of student engagementfrom facial expressions,” IEEE Transactions on Affective Computing, vol. 5, no. 1, pp. 86–98, 2014.
S. K. D’Mello, S. D. Craig, and A. C. Graesser, “Multimethod assessment of affective experience and expression during deep learning,” International Journal of Learning Technology, vol. 4, no. 3-4, pp. 165–
, 2009.
J. Whitehill, Z. Serpell, Y.-C. Lin, A. Foster, and J. R. Movellan, “The faces of engagement: Automatic recognition of student engagementfrom facial expressions,” IEEE Transactions on Affective Computing, vol. 5, no. 1, pp. 86–98, 2014.
S. K. D’Mello, S. D. Craig, and A. C. Graesser, “Multimethod assessment of affective experience and expression during deep learning,” International Journal of Learning Technology, vol. 4, no. 3-4, pp. 165–
, 2009.
S. K. D’Mello and A. Graesser, “Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features,” User Modeling and User-Adapted Interaction, vol. 20, no. 2,
pp. 147–187, 2010.
D. Canedo, A. Trifan, and A. J. Neves, “Monitoring students’ attention in a classroom through computer vision,” in International Conference on Practical Applications of Agents and Multi-Agent Systems. Springer, 2018, pp. 371–378.
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014, pp. 1725–1732.
H. Wang, A. Kl¨aser, C. Schmid, and C.-L. Liu, “Action recognition by dense trajectories,” in CVPR 2011, 2011, pp. 3169–3176.
X. Wang, D. Chen, T. Yang, B. Hu, and J. Zhang, “Action recognition based on object tracking and dense trajectories,” in 2016 IEEE International Conference on Automatica (ICA-ACCA). IEEE, 2016, pp. 1–5.
Z. Zhang, Z. Lv, C. Gan, and Q. Zhu, “Human action recognition using convolutional lstm and fully-connected lstm with different attentions,” Neurocomputing, vol. 410, pp. 304–316, 2020.
J. Su, W. Byeon, J. Kossaifi, F. Huang, J. Kautz, and A. Anandkumar, “Convolutional tensor-train lstm for spatio-temporal learning,” arXiv preprint arXiv:2002.09131, 2020.
L. Wei, S. Zhao, O. F. Bourahla, X. Li, F. Wu, Y. Zhuang, J. Han,
and M. Xu, “End-to-end video saliency detection via a deep contextual spatiotemporal network,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
J. Zaletelj and A. Koˇsir, “Predicting students’ attention in the classroom from kinect facial and body features,” EURASIP journal on image and video processing, vol. 2017, no. 1, pp. 1–12, 2017.
M. S. Young, S. Robinson, and P. Alberts, “Students pay attention! combating the vigilance decrement to improve learning during lectures,” Active Learning in Higher Education, vol. 10, no. 1, pp. 41–55, 2009.
A. S. Won, J. N. Bailenson, and J. H. Janssen, “Automatic detection of nonverbal behavior predicts learning in dyadic interactions,” IEEE Transactions on Affective Computing, vol. 5, no. 2, pp. 112–125, 2014.
K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
Z. Huang, W. Xu, and K. Yu, “Bidirectional lstm-crf models for sequence tagging,” arXiv preprint arXiv:1508.01991, 2015.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035.
A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks),” in Proceedings of the IEEE International Conference on
Computer Vision, 2017, pp. 1021–1030.
