Unveiling the core constructs: A statistical approach to evaluating user experience with Chatbots in higher education (A case study from a university in Ecuador)

Jomar Elizabeth Guzman Seraquive ,Patricio Álvarez-Muñoz ,Kerly Palacios-Zamora ,Dennis Alfredo Peralta Gamboa ,Marco Faytong-Haro

doi:10.24294/jipd.v8i10.6381

Home - JIPD - Volume 8 - Issue 10

Submit to JIPD

Journal Browser

Volume | Year

Issue

• Current lssue

View All

News and Announcements

View All

Unveiling the core constructs: A statistical approach to evaluating user experience with Chatbots in higher education (A case study from a university in Ecuador)

Jomar Elizabeth Guzman Seraquive

Patricio Álvarez-Muñoz

Kerly Palacios-Zamora

Dennis Alfredo Peralta Gamboa

Marco Faytong-Haro

Journal of Infrastructure Policy and Development 2024, 8(10); https://doi.org/10.24294/jipd.v8i10.6381

Submitted：14 May 2024

Accepted：13 Jun 2024

Published：30 Sept 2024

Download PDF(623.84KB)

XML

Abstract

Introduction: Chatbots are increasingly utilized in education, offering real-time, personalized communication. While research has explored technical aspects of chatbots, user experience remains under-investigated. This study examines a model for evaluating user experience and satisfaction with chatbots in higher education. Methodology: A four-factor model (information quality, system quality, chatbot experience, user satisfaction) was proposed based on prior research. An alternative two-factor model emerged through exploratory factor analysis, focusing on “Chatbot Response Quality” and “User Experience and Satisfaction with the Chatbot.” Surveys were distributed to students and faculty at a university in Ecuador to collect data. Confirmatory factor analysis validated both models. Results: The two-factor model explained a significantly greater proportion of the data’s variance (55.2%) compared to the four-factor model (46.4%). Conclusion: This study suggests that a simpler model focusing on chatbot response quality and user experience is more effective for evaluating chatbots in education. Future research can explore methods to optimize these factors and improve the learning experience for students.

Keywords

References

Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006. https://doi.org/10.1016/j.mlwa.2020.100006

Ait Baha, T., El Hajji, M., Es-Saady, Y., et al. (2023). The impact of educational chatbot on student learning experience. Education and Information Technologies, 29(8), 10153–10176. https://doi.org/10.1007/s10639-023-12166-w

Baskara, F. R. (2023). Chatbots and flipped learning: Enhancing student engagement and learning outcomes through personalised support and collaboration. International Journal of Recent Educational Research, 4(2), 223–238.

Cao, C. C., Ding, Z., Lin, J., & Hopfgartner, F. (2023). AI Chatbots as Multi-Role Pedagogical Agents: Transforming Engagement in CS Education. arXiv, arXiv:2308.03992.

Chung, M., Ko, E., Joung, H., & Kim, S. J. (2020). Chatbot e-service and customer satisfaction regarding luxury brands. Journal of Business Research, 117, 587–595.

Deng, X., & Yu, Z. (2023). A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability, 15(4), 2940.

Fox, C. (2014). Chatbot takes on Turing. The Philosophers’ Magazine, 66, 8–8.

Liu, L., Subbareddy, R., & Raghavendra, C. (2022). Ai intelligence chatbot to improve students learning in the higher education platform. Journal of Interconnection Networks, 22(Supp02), 2143032.

López-Aguado, M., & Gutiérrez-Provecho, L. (2019). How to perform and interpret an exploratory factor analysis using SPSS Statistics. Revista d’Innovació i Recerca En Educació, 12(2), 1.

Malhotra, N. K., Nunan, D., & Birks, D. F. (2020). Marketing research. Pearson UK.

Martínez, C. M., & Sepúlveda, M. A. R. (2012). Introduction to exploratory factor analysis (Spanish). Revista Colombiana de Psiquiatría, 41(1), 197–207.

Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033.

Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549–1565.

Peyton, K., & Unnikrishnan, S. (2023). A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs. Results in Engineering, 17, 100856.

Przegalinska, A., Ciechanowski, L., Stroz, A., et al. (2019). In bot we trust: A new methodology of chatbot performance measures. Business Horizons, 62(6), 785–797.

Sakulwichitsintu, S. (2023). ParichartBOT: a chatbot for automatic answering for postgraduate students of an open university. International Journal of Information Technology, 15(3), 1387–1397.

Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862.

Trivedi, J. (2019). Examining the customer experience of using banking chatbots and its impact on brand love: The moderating role of perceived risk. Journal of Internet Commerce, 18(1), 91–111.

Zarouali, B., Van den Broeck, E., Walrave, M., & Poels, K. (2018). Predicting consumer responses to a chatbot on Facebook. Cyberpsychology, Behavior, and Social Networking, 21(8), 491–497.

Zhang, G., & Preacher, K. J. (2015). Factor rotation and standard errors in exploratory factor analysis. Journal of Educational and Behavioral Statistics, 40(6), 579–603.

Previous
article in this issue

Next
article in this issue