DOI Services Persistent Identifiers for Scholarly Research
You are viewing the preserved copy. Open the article on the journal's website →

Contemporary Journal of Social Science Review • pp. 1-24

DO AI SCORERS UNDERSTAND URDU-ENGLISH CODE-SWITCHING? FAIRNESS OF AUTOMATED WRITING EVALUATION FOR BILINGUAL LEARNERS

  • Bilal Jamil M. Phil. English Linguistics Emerson University Multan. Lecturer English Hameeda Rasheed institute of Science and Technology Multan. (Corresponding author)
  • Shahbaz Ahmad M.Phil English Literature, University of Southern Punjab English lecturer in Hameeda Rasheed Institute of Science and Technology, Multan
  • Dr. Muhammad Umar Mehmood PhD Education, Punjab School Education Department, Pakistan

Abstract

The use of Artificial Intelligence (AI) – based Automated Writing Evaluation (AWE) systems and Large Language Models (LLMs) has been growing in the field of educational assessment, with AI tools offering timely feedback to help students improve their written responses. Their fairness and reliability is still questionable, however, when evaluating bilingual learners who employ code-switching and multilingual expressions in their writing. The aim of this study is to examine whether AI-based scoring systems provide fair evaluation of Urdu-English code-switched writing among the undergraduate students of higher education institutions in Pakistan or not, when compared with the English-only writing. It also explores how the effect of code-switching features, bilingual-aware assessment instructions and human supervision impact automated assessment results. Explanatory sequential mixed methods design was used. The quantitative phase comprised 120 writing samples from students, wherein English-only responses were matched with Urdu-English code-switched responses, which were assessed using two rubrics: the content rubric and the English-proficiency rubric, using AI systems and trained human raters. The qualitative phase consisted of semi-structured interviews conducted with 14 undergraduate students and 9 faculty members, with a focus on their perceptions of assessment fairness, linguistic identity, transparency and human involvement. The results showed significant discrepancies between the scores of AI and human in writing in code. The mean difference between AI and human for English-only responses was -0.9 points, whereas it was -7.6 points for code-switched responses. In the same way, the agreement between AI and humans was lower for code-switched writing (ICC = 0.54) than for English-only writing (ICC = 0.82). The regression analysis revealed that code switching density negatively correlated with unprompted AI scores (R² = 0.38 p < .01). When the prompting was bilingual-aware, however, the reported score gap went down from 7.6 to 1.4 points. The results from the interviews emphasized three factors: linguistic alienation, differences in scoring that depend on the construct, and the necessity of human supervision. The study finds that it is inappropriate to automatically penalize bilingual expression when it is allowed according to assessment criteria. It suggests rubrics aligned to the teaching and learning of the text(s) under consideration, bilingual-aware instructions for teachers and students, systematic fairness audits, transparent feedback, teacher training, and valid and meaningful moderation by humans to support equitable and valid automated writing assessment in multilingual educational settings.

Record

Identifier
https://doi-services.com/1102/3537
DOI
10.66857/s8893g03
Published
2026-03-12
Version
1
License
https://creativecommons.org/licenses/by/4.0
Original
https://studentsconsultancyhome.com/contemporary-journal/index.php/cjssr/article/view/3537