The architecture of automatic scoring systems for non-native English spontaneous speech: A systematic literature review

Un I. Kuok

doi:10.24294/jipd10078

Home - JIPD - Volume 9 - Issue 2

Submit to JIPD

Journal Browser

Volume | Year

Issue

• Current lssue

View All

News and Announcements

View All

The architecture of automatic scoring systems for non-native English spontaneous speech: A systematic literature review

Un I. Kuok

Journal of Infrastructure Policy and Development 2025, 9(2), 10078; https://doi.org/10.24294/jipd10078

Submitted：04 Nov 2024

Accepted：04 Dec 2024

Published：07 Apr 2024

Download PDF(624.05KB)

XML

Abstract

Given the heavy workload faced by teachers, automatic speaking scoring systems provide essential support. This study aims to consolidate technological configurations of automatic scoring systems for spontaneous L2 English, drawing from literature published between 2014 and 2024. The focus will be on the architecture of the automatic speech recognition model and the scoring model, as well as on features used to evaluate phonological competence, linguistic proficiency, and task completion. By synthesizing these elements, the study seeks to identify potential research areas, as well as provide a foundation for future research and practical applications in software engineering.

Keywords

References

Aksënova, A., Chen, Z., Chiu, C.-C., et al. (2022). Accented Speech Recognition: Benchmarking, Pre-training, and Diverse Data (arXiv:2205.08014). arXiv. https://doi.org/10.48550/arXiv.2205.08014

Alharbi, S., Alrazgan, M., Alrashed, A., et al. (2021). Automatic Speech Recognition: Systematic Literature Review. IEEE Access, 9, 131858–131876. https://doi.org/10.1109/ACCESS.2021.3112535

Anderson‐Hsieh, J., Johnson, R., Koehler, K. (1992). The Relationship Between Native Speaker Judgments of Nonnative Pronunciation and Deviance in Segmentais, Prosody, and Syllable Structure. Language Learning, 42(4), 529–555. https://doi.org/10.1111/j.1467-1770.1992.tb01043.x

Arslan, L. M., Hansen, J. H. L. (1996). Language accent classification in American English. Speech Communication, 18(4), 353–367. https://doi.org/10.1016/0167-6393(96)00024-6

Benkerzaz, S., Elmir, Y., Dennai, A. (2019). A Study on Automatic Speech Recognition. Journal of Information Technology Review, 10(3). https://doi.org/10.6025/jitr/2019/10/3/77-85

Bhat, S., Yoon, S.-Y. (2015). Automatic assessment of syntactic complexity for spontaneous speech scoring. Speech Communication, 67, 42–57. https://doi.org/10.1016/j.specom.2014.09.005

Brown, P. F., Della Pietra, V. J., deSouza, P. V., Lai, J. C., & Mercer, R. L. (1992). Class-based n-gram models of natural language. Computational Linguistics, 18(4), 467–480. Retrieved from https://aclanthology.org/J92-4003/

Chambers, F. (1997). What do we mean by fluency? System, 25(4), 535–544. https://doi.org/10.1016/S0346-251X(97)00046-8

Chen, M., Zechner, K. (2011). Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-Native Speech. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies; June 2011; Portland, OR, USA. pp. 722–731

Chen, Y., Hu, J., Zhang, X. (2019). Sell-corpus: An Open Source Multiple Accented Chinese-english Speech Corpus for L2 English Learning Assessment. In: Proceedings of the ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 12–17 May 2019; Brighton, UK. 7425–7429. https://doi.org/10.1109/ICASSP.2019.8682612

Cheng, J., Chen, X., Metallinou, A. (2015). Deep neural network acoustic models for spoken assessment applications. Speech Communication, 73, 14–27. https://doi.org/10.1016/j.specom.2015.07.006

Cheng, Z., Wang, Z. (2022). Automatic Scoring of Spoken Language Based on Basic Deep Learning. Scientific Programming, 2022, 1–14. https://doi.org/10.1155/2022/6884637

Cucchiarini, C., Strik, H., Boves, L. (2000). Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. The Journal of the Acoustical Society of America, 107(2), 989–999. https://doi.org/10.1121/1.428279

Cucchiarini, C., Strik, H., Boves, L. (2002). Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech. The Journal of the Acoustical Society of America, 111(6), 2862–2873. https://doi.org/10.1121/1.1471894

Défossez, A., Mazaré, L., Orsini, M., et al. (2024). Moshi: A speech-text foundation model for real-time dialogue. arXiv. https://doi.org/10.48550/arXiv.2410.00037

Deng, Y., Li, X., Kwan, C., Raj, B., & Stern, R. (2007). Continuous feature adaptation for non-native speech recognition. International Journal of Computer and Information Engineering, 1(6), 1701–1707. https://doi.org/10.5281/zenodo.1329829

Dufour, R., Estève, Y., Deléglise, P. (2014). Characterizing and detecting spontaneous speech: Application to speaker role recognition. Speech Communication, 56, 1–18. https://doi.org/10.1016/j.specom.2013.07.007

Ellis, R. (2009). Task-based language learning and teaching (7th print). Oxford University Press.

Fendji, J. L. K. E., Tala, D. C. M., Yenke, B. O., Atemkeng, M. (2022). Automatic Speech Recognition Using Limited Vocabulary: A Survey. Applied Artificial Intelligence, 36(1), 2095039. https://doi.org/10.1080/08839514.2022.2095039

Franzke, M., Kintsch, E., Caccamise, D., et al. (2005). Summary Street®: Computer Support for Comprehension and Writing. Journal of Educational Computing Research, 33(1), 53–80. https://doi.org/10.2190/DH8F-QJWM-J457-FQVB

Fu, J., Chiba, Y., Nose, T., Ito, A. (2020). Automatic assessment of English proficiency for Japanese learners without reference sentences based on deep neural network acoustic models. Speech Communication, 116, 86–97. https://doi.org/10.1016/j.specom.2019.12.002

Gabler, P., Geiger, B. C., Schuppler, B., Kern, R. (2023). Reconsidering Read and Spontaneous Speech: Causal Perspectives on the Generation of Training Data for Automatic Speech Recognition. Information, 14(2), 137. https://doi.org/10.3390/info14020137

Ge, Z. (2015). Improved accent classification combining phonetic vowels with acoustic features. In: Proceedings of the 2015 8th International Congress on Image and Signal Processing (CISP); 14–16 October 2015; Shenyang, China. pp. 1204–1209. https://doi.org/10.1109/CISP.2015.7408064

Georgakis, C., Petridis, S., Pantic, M. (2016). Discrimination Between Native and Non-Native Speech Using Visual Features Only. IEEE Transactions on Cybernetics, 46(12), 2758–2771. https://doi.org/10.1109/TCYB.2015.2488592

Gerosa, M., Giuliani, D., & Narayanan, S. (2006). Acoustic analysis and automatic recognition of spontaneous children’s speech. Interspeech 2006, 519–522. https://doi.org/10.21437/Interspeech.2006-519

Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8(6). Retrieved June 20, 2024, from http://www.jtla.org

Hayashi, Y., Kondo, Y., Ishii, Y. (2024). Automated speech scoring of dialogue response by Japanese learners of English as a foreign language. Innovation in Language Learning and Teaching, 18(1), 32–46. https://doi.org/10.1080/17501229.2023.2217181

Heeman, P. A., & Allen, J. F. (1999). Speech repairs, intonational phrases, and discourse markers: Modeling speakers’ utterances in spoken dialogue. Computational Linguistics, 25(4), 527–572. Retrieved from https://aclanthology.org/J99-4003/

Housen, A., Kuiken, F. (2009). Complexity, Accuracy, and Fluency in Second Language Acquisition. Applied Linguistics, 30(4), 461–473. https://doi.org/10.1093/applin/amp048

Johnson, M., Charniak, E. (2004). A TAG-based noisy channel model of speech repairs. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics; 21–26 July 2004; Stroudsburg PA USA. p. 33-es. https://doi.org/10.3115/1218955.1218960

Kang, B. O., Jeon, H., Lee, Y. K. (2024). AI‐based language tutoring systems with end‐to‐end automatic speech recognition and proficiency evaluation. ETRI Journal, 46(1), 48–58. https://doi.org/10.4218/etrij.2023-0322

Kang, O., Johnson, D. (2018). The roles of suprasegmental features in predicting English oral proficiency with an automated system. Language Assessment Quarterly, 15(2), 150–168. https://doi.org/10.1080/15434303.2018.1451531

Kat L. W., Fung, P. (1999). Fast accent identification and accented speech recognition. In: Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258); 15–19 March 1999; Phoenix, AZ, USA. pp. 221–224. https://doi.org/10.1109/ICASSP.1999.758102

Knill, K. M., Gales, M. J. F., Manakul, P. P., Caines, A. P. (2019). Automatic Grammatical Error Detection of Non-native Spoken Learner English. In: Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 12–17 May 2019; Brighton, UK. pp. 8127–8131. https://doi.org/10.1109/ICASSP.2019.8683080

Kobayashi, Y., & Abe, M. (2016). Automated scoring of L2 spoken English with random forests. Journal of Pan-Pacific Association of Applied Linguistics, 20(1), 55–73. Retrieved from https://eric.ed.gov/?id=EJ1110804

Lease, M., Johnson, M., Charniak, E. (2006). Recognizing disfluencies in conversational speech. IEEE Transactions on Audio, Speech and Language Processing, 14(5), 1566–1573. https://doi.org/10.1109/TASL.2006.878269

Lee, H.-S., Chen, P.-Y., Cheng, Y.-F., et al. (2022). Speech-enhanced and Noise-aware Networks for Robust Speech Recognition. arXiv. http://arxiv.org/abs/2203.13696

Lennon, P. (1990). Investigating Fluency in EFL: A Quantitative Approach. Language Learning, 40(3), 387–417. https://doi.org/10.1111/j.1467-1770.1990.tb00669.x

Li, K.-C., Chang, M., Wu, K.-H. (2020). Developing a Task-Based Dialogue System for English Language Learning. Education Sciences, 10(11), 306. https://doi.org/10.3390/educsci10110306

Li, S., Ouyang, B., Liao, D., et al. (2021). End-To-End Multi-Accent Speech Recognition with Unsupervised Accent Modelling. In: Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 6–11 June 2021; Toronto, ON, Canada. pp. 6418–6422. https://doi.org/10.1109/ICASSP39728.2021.9414833

Liu, J., Wumaier, A., Fan, C., Guo, S. (2023). Automatic Fluency Assessment Method for Spontaneous Speech without Reference Text. Electronics, 12(8), 1775. https://doi.org/10.3390/electronics12081775

Livescu, K., Glass, J. (2000). Lexical modeling of non-native speech for automatic speech recognition. In: Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100); 5–9 June 2000; Istanbul, Turkey. 1683–1686. https://doi.org/10.1109/ICASSP.2000.862074

Ma, Z., Song, Y., Du, C., et al. (2024). Language Model Can Listen While Speaking. arXiv. http://arxiv.org/abs/2408.02622

Miao, Y., Metze, F. (2017). End-to-End Architectures for Speech Recognition. In: Watanabe, S., Delcroix, M., Metze, F., Hershey, J. R. (editors). New Era for Robust Speech Recognition, Springer International Publishing. pp. 299–323. https://doi.org/10.1007/978-3-319-64680-0_13

Molenaar, B., Tejedor-Garcia, C., Cucchiarini, C., Strik, H. (2023). Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics. INTERSPEECH, 2023, 5232–5236. https://doi.org/10.21437/Interspeech.2023-1681

Mørch, A., Engeness, I., Cheung, K.-W. (2017). EssayCritic: Writing to learn with a knowledge-based design critiquing system. Educational Technology & Society, 20(2), 213–223.

Nakamura, M., Iwano, K., Furui, S. (2008). Differences between acoustic characteristics of spontaneous and read speech and their effects on speech recognition performance. Computer Speech & Language, 22(2), 171–184. https://doi.org/10.1016/j.csl.2007.07.003

Nunes, A., Cordeiro, C., Limpo, T., Castro, S. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599–620. https://doi.org/10.1111/jcal.12635

Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, n71. https://doi.org/10.1136/bmj.n71

Qian, Y., Wang, X., Evanini, K., Suendermann-Oeft, D. (2016). Self-Adaptive DNN for Improving Spoken Language Proficiency Assessment. Interspeech, 2016, 3122–3126. https://doi.org/10.21437/Interspeech.2016-291

Rosenfeld, R. (2000). Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8), 1270–1278. https://doi.org/10.1109/5.880083

Sainath, T. N., Weiss, R. J., Wilson, K. W., et al. (2017). Raw Multichannel Processing Using Deep Neural Networks. In: Watanabe, S., Delcroix, M., Metze, F., Hershey, J. R. (Eds.). New Era for Robust Speech Recognition, Springer International Publishing. pp. 105–133. https://doi.org/10.1007/978-3-319-64680-0_5

Saon, G., Chien, J.-T. (2012). Large-Vocabulary Continuous Speech Recognition Systems: A Look at Some Recent Advances. IEEE Signal Processing Magazine, 29(6), 18–33. https://doi.org/10.1109/MSP.2012.2197156

Shriberg, E. E. (1999). Phonetic consequences of speech disfluency. Proceedings of the International Congress of Phonetic Sciences, 1(2), 619–622. Retrieved from https://www.internationalphoneticassociation.org/icphs-proceedings/ICPhS1999/papers/p14_0619.pdf

Tamazin, M., Gouda, A., Khedr, M. (2019). Enhanced Automatic Speech Recognition System Based on Enhancing Power-Normalized Cepstral Coefficients. Applied Sciences, 9(10), 2166. https://doi.org/10.3390/app9102166

Tao, J., Ghaffarzadegan, S., Chen, L., Zechner, K. (2016). Exploring deep learning architectures for automatically grading non-native spontaneous speech. In: Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 20–25 March 2016; Shanghai, China. pp. 6140–6144. https://doi.org/10.1109/ICASSP.2016.7472857

Trentin, E., Gori, M. (2001). A survey of hybrid ANN/HMM models for automatic speech recognition. Neurocomputing, 37(1–4), 91–126. https://doi.org/10.1016/S0925-2312(00)00308-8

Van Bergem, D. R. (1995). Perceptual and acoustic aspects of lexical vowel reduction, a sound change in progress. Speech Communication, 16(4), 329–358. https://doi.org/10.1016/0167-6393(95)00003-7

Wang, Y., Gales, M. J. F., Knill, K. M., et al. (2018). Towards automatic assessment of spontaneous spoken English. Speech Communication, 104, 47–56. https://doi.org/10.1016/j.specom.2018.09.002

Wang, Y.-J., Shang, H.-F., Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 234–257. https://doi.org/10.1080/09588221.2012.655300

Wills, S., Bai, Y., Tejedor-Garcia, C., et al. (2023). Automatic Speech Recognition of Non-Native Child Speech for Language Learning Applications. Slate. https://doi.org/10.4230/OASICS.SLATE.2023.11

Wilson, J., Czik, A. (2016). Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education, 100, 94–109. https://doi.org/10.1016/j.compedu.2016.05.004

Yoon, S. Y., & Bhat, S. (2012). Assessment of ESL learners' syntactic competence based on similarity measures. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 600–608). Association for Computational Linguistics. https://doi.org/10.3115/v1/D12-1055

Yoon, S.-Y., Bhat, S. (2018). A comparison of grammatical proficiency measures in the automated assessment of spontaneous speech. Speech Communication, 99, 221–230. https://doi.org/10.1016/j.specom.2018.04.003

Yu, D., Deng, L. (2015). Automatic Speech Recognition: A Deep Learning Approach. Springer London. https://doi.org/10.1007/978-1-4471-5779-3

Zechner, K., Evanini, K., Yoon, S.-Y., et al. (2014). Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language. In: Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications; June 2014; Baltimore, Maryland. pp. 134–142. https://doi.org/10.3115/v1/W14-1816

Previous
article in this issue

Next
article in this issue