top of page

Regional Finalist, SARC 2025

Investigating the Influence of a Socratic Chatbot on Year 10 English Students’ Depth of Understanding and Critical Thinking about Curricular Topics in New Zealand

By Jeshna Madaan, New Zealand

Abstract:

This proposal investigates the possibility of utilising Socratic questioning using AI chatbots to improve Year 10 English skills in New Zealand. Socratic inquiry, which has typically been beneficial in one-on-one discussions, has been difficult to scale, but AI may offer a solution. This proposal uses AI chatbots to deliver individualised, thought-provoking discussion to help students improve their critical thinking and depth of understanding.

​

Introduction:

Co-requisite assessment results, required for the National Certificate of Educational Achievement (NCEA), typically attempted in Year 10 as preparation for NCEA, are poor, highlighting significant gaps in literacy skills and raising concerns about students' readiness for higher-level academic demands. According to the official New Zealand Qualifications Authority (NZQA) website, only 66% achieved the writing standard, and 70% achieved the reading standard. This indicates significant deficiencies in the literacy abilities needed for success, such as depth of understanding and critical thinking. By promoting critical thinking through tailored, thought-provoking conversations, this proposal seeks to examine how a Socratic AI chatbot may assist Year 10 students English performance.

 

Literature Review:

Socratic questioning is designed to prompt students to critically process information, use it effectively, and draw their own conclusions (Robinson, 2017). It guidesrather than leads - learners to reflect on and assess their thinking with open-ended questions. In doing so, this approach cultivates critical thinking and reflective inquiry, essential for grappling with complex ideas and varied viewpoints (Favero et al., 2024). Additionally, Socratic questioning challenges students to uncover unconscious biases and conflicting beliefs and seek deeper truths (Fakour & Imani, 2025).

​

A number of investigations have compared human tutors with Artificial Intelligence (AI) chatbots, highlighting the strengths and limitations of each. Recurring concerns that were discussed regarding the AI chatbots in the articles looked at were misinformation being presented as fact (“...ChatGPT’s tendency to hallucinate” - Zafar et al., 2025) and overreliance on the chatbot, resulting in academic dishonesty and an overall degradation in critical thinking and analytical ability (Zafar et al., 2025) (Blasco & Charisi, 2024) (Ji Eun Lee & Maeng, 2023). These investigations also uncovered the benefits of AI chatbots. One study, done on 300 secondary students in Islamabad, surveyed their perceptions of ChatGPT, an AI tool. “68% of students perceived ChatGPT as a beneficial tool for fostering critical thinking, with frequent users showing a 15% improvement in assessment scores.” (Zafar et al., 2025) AI was particularly noted for its accessibility, speed, personalised-paced learning, real-time feedback, and lack of time constraints (Hoda Fakour & Imani, 2025) (Zafar et al., 2025) (Ji Eun Lee & Maeng, 2023)

​

Most of these investigations found through their research and/or came to the conclusion of incorporating both AI and human tutors into learning due to their unique benefits (Ji Eun Lee & Maeng, 2023) (Hoda Fakour & Imani, 2025) (Zafar et al., 2025). Human tutors differ from AI chatbots in the way that they are perceived as more effective in encouraging engagement, critical thinking, and providing emotional support (Hoda Fakour & Imani, 2025). One study on the perceptions of 30 female high school students in South Korea on the use of AI Chatbots English learning recgonised that “Recently, with the rapid development of AI technology, generative AI like ChatGPT has been influencing education as a whole and is anticipated to have a significant impact on English education” (Ji Eun Lee & Maeng, 2023). The study also found that students perceived AI chatbots as very beneficial in the English subject.

​

Socrates promoted personalised teaching, emphasising one-on-one interactions to ensure direct engagement between teachers and students (Robinson, 2017). However, replicating such individualised teaching is impractical in most classroom settings, making AI chatbots a promising alternative. Early implementations of AI-driven Socratic tutors have shown the technical feasibility of scaling individualised, question-based dialogue outside the constraints of human availability (Favero et al., 2024). However, these systems were tested only in simulated conversations by students simulated by AI. This limits their insight into real-world learning dynamics. In contrast, the study conducted by Blasco and Charisi (2024) employed ChatGPT-4—an LLM that cannot be fine-tuned externally—and relied on Socratic-style prompts to simulate Socratic dialogue. While this approach offered valuable insights into how prompting alone can shape student-AI interaction, it lacked the adaptive depth of a fine-tuned model.

​

This proposal aims to fill gaps in the current research: the lack of classroom-based investigations using fine-tuned Socratic AI chatbots with real students, and the limited focus on integrating both AI and human tutors to leverage their unique strengths. Additionally, this proposal seeks to mitigate recurring concerns around AI by replacing simple answers with thought-provoking questions and, explore the potential of AI chatbots and Socratic questioning in supporting Year 10 English students in Aotearoa New Zealand, a research gap especially highlighted by concerns over falling literacy performance.

 

Methodology:​ 

The chatbot will be built using Llama2 Instruct with 7B parameters for its accessibility, and be fine-tuned by the SocratiQ dataset, which is a set of Socratic question-context pairs. The bot will be hosted by Hugging Face to ensure accessibility for all participants. Initial testing will be to ensure the chatbot responds reliably with open-ended, thought-provoking questions, maintains a supportive tone, and to check that it is easy for students to navigate. Once this internal testing is completed, the chatbot will be tested in a controlled beta/pilot phase with a small number of students to evaluate usability, effectiveness, and any required adjustments before broader rollout. Students in participating schools will be randomly grouped. 75% will have access to the chatbot as a Socratic tutor, while 25% will act as a control group without access. This balance is to maximise the number of students benefiting from the tool while still providing a valid comparison group. The investigation will run for the duration of one unit in English, which typically lasts one school term (9-11 weeks), during which students will interact with the bot for 10 minutes, twice a week, during homework time as a support tool for their English studies.

​

Students with access to the Socratic tutor will complete weekly quantitative Likert-scale questionnaires, with a qualitative survey at the end of the unit to record deeper reflections. At the end of the unit, teachers will also complete a short quantitative Likert-scale questionnaire and brief written comments, taking into account previous student performance. Chat logs will be stored and anonymised to remove any identifying information before analysis. Participating schools will designate one organising teacher each to manage ethical consent processes (from students and their parents), explain the purpose of the bot, guide students in using it correctly and responsibly, and ensure academic honesty is maintained. A pre-test/post-test approach will not be used, as content varies between schools, making consistent comparison impractical.

​

Responses received from the Likert-scale questionnaires will be analysed using descriptive statistics and SPSS to identify trends between the chatbot and control groups. Thematic analysis will be applied to qualitative responses from students and teachers to identify patterns in perceived engagement, critical thinking, and depth of understanding. Sentiment analysis and thematic frequency counts will also be applied to chatbot logs. Triangulation across student surveys, teacher feedback, and chatbot interactions will help ensure validity and depth in interpreting the Socratic chatbot’s impact.

​

Conclusion:

The purpose of this proposal is to investigate potential uses of AI chatbots equipped with Socratic dialogue to increase critical thinking and literacy in Year 10 students. While admitting limits such as student variability and the difficulties of long-term evaluation, the concept establishes a solid foundation for successful, adaptive outcomes.

​

References :

Blasco, A., & Charisi, V. (2024). The Impact of Large Language Models on Students: A Randomised Study of Socratic vs. Non-Socratic AI and the Role of Step-by-Step Reasoning. The Impact of Large Language Models on Students: A Randomised Study of Socratic vs. Non-Socratic AI and the Role of Step-By-Step Reasoning. https://doi.org/10.2139/ssrn.5040921

 

Favero, L., Pérez-Ortiz, J. A., Käser, T., & Oliver, N. (2024). Enhancing Critical Thinking in Education by means of a Socratic Chatbot. Arxiv.org. https://arxiv.org/html/2409.05511v1#S2

 

Hoda Fakour, & Imani, M. (2025). Socratic wisdom in the age of AI: a comparative study of ChatGPT and human tutors in enhancing critical thinking skills. Frontiers in Education, 10. https://doi.org/10.3389/feduc.2025.1528603

 

Ji Eun Lee, & Maeng, U. (2023). Perceptions of High School

Students on AI Chatbots Use in English Learning: Benefits, Concerns, and Ethical Consideration. Journal of Pan-Pacific Association of Applied Linguistics, 27(2), 53–72. https://eric.ed.gov/?id=EJ1414374 Results from Literacy and Numeracy assessment events :: NZQA. (2024). Govt.nz. https://www2.nzqa.govt.nz/ncea/subjects/litnum/literacy-and-numeracy-data/

 

Robinson, S. M. (2017, February 11). Socratic Questioning: A Teaching Philosophy for the Student Research Consultation – In the Library with the Lead Pipe. In the Library with the Leadpipe. https://www.inthelibrarywiththeleadpipe.org/2017/socratic-questioning/

 

Zafar, S., Ayaz, N., Bibi, H., & Abbas, S. G. (2025). The Effect of Chatgpt on the Critical Thinking Skills of Secondary School Students: A Survey-Based Study. THE EFFECT of CHATGPT on the CRITICAL THINKING SKILLS of SECONDARY SCHOOL STUDENTS: A SURVEY-BASED STUDY, Vol. 3 No.(03 (2025)), 243–259. https://www.researchgate.net/publication/390161681_The_Effect_of_Chatgpt_on_the_Critica l_Thinking_Skills_of_Secondary_School_Students_A_Survey-Based_Study

bottom of page