Can learners benefit from chatbots instead of humans? A systematic review of human-chatbot comparison research in language education

Research has demonstrated the promising potential of chatbots in education. Moreover, technological advancements, such as ChatGPT, prompted us to reexamine distinctions between pedagogical roles that humans and chatbots assume. In this context, a systematic review of 11 experimental studies on human-chatbot comparisons in language education was performed, yielding 64 statistical findings, which were then categorized into 11 overarching variables. The analysis indicates that chatbots provide benefits comparable to those afforded by human-human interaction in some domains, such as eliciting utterances of similar sophistication, vocabulary, and grammar levels and facilitating improvements in speaking and listening proficiency. In contrast, chatbots were less effective than humans in areas that may demand socially appropriate interpersonal elements, such as sustaining interactivity, providing sufficient information in elaborations, and maintaining a positive attitude toward target language conversations over the long term. Based on the results, we suggest that chatbots be conceptualized as novel interlocutors rather than as simulations striving to perfectly mimic humans and that emphasis be placed on aspects humans should focus more on in educational scenarios where chatbots are involved. Additionally, other implications for researchers and teachers are discussed to inform future research and practice.