Contemporary Journal of Social Science Review • pp. 1-24
DO AI SCORERS UNDERSTAND URDU-ENGLISH CODE-SWITCHING? FAIRNESS OF AUTOMATED WRITING EVALUATION FOR BILINGUAL LEARNERS
Abstract
The use of Artificial Intelligence (AI) – based Automated Writing Evaluation (AWE) systems and Large Language Models (LLMs) has been growing in the field of educational assessment, with AI tools offering timely feedback to help students improve their written responses. Their fairness and reliability is still questionable, however, when evaluating bilingual learners who employ code-switching and multilingual expressions in their writing. The aim of this study is to examine whether AI-based scoring systems provide fair evaluation of Urdu-English code-switched writing among the undergraduate students of higher education institutions in Pakistan or not, when compared with the English-only writing. It also explores how the effect of code-switching features, bilingual-aware assessment instructions and human supervision impact automated assessment results. Explanatory sequential mixed methods design was used. The quantitative phase comprised 120 writing samples from students, wherein English-only responses were matched with Urdu-English code-switched responses, which were assessed using two rubrics: the content rubric and the English-proficiency rubric, using AI systems and trained human raters. The qualitative phase consisted of semi-structured interviews conducted with 14 undergraduate students and 9 faculty members, with a focus on their perceptions of assessment fairness, linguistic identity, transparency and human involvement. The results showed significant discrepancies between the scores of AI and human in writing in code. The mean difference between AI and human for English-only responses was -0.9 points, whereas it was -7.6 points for code-switched responses. In the same way, the agreement between AI and humans was lower for code-switched writing (ICC = 0.54) than for English-only writing (ICC = 0.82). The regression analysis revealed that code switching density negatively correlated with unprompted AI scores (R² = 0.38 p < .01). When the prompting was bilingual-aware, however, the reported score gap went down from 7.6 to 1.4 points. The results from the interviews emphasized three factors: linguistic alienation, differences in scoring that depend on the construct, and the necessity of human supervision. The study finds that it is inappropriate to automatically penalize bilingual expression when it is allowed according to assessment criteria. It suggests rubrics aligned to the teaching and learning of the text(s) under consideration, bilingual-aware instructions for teachers and students, systematic fairness audits, transparent feedback, teacher training, and valid and meaningful moderation by humans to support equitable and valid automated writing assessment in multilingual educational settings.
Keywords
Record
- Identifier
- https://doi-services.com/1102/3537
- Published
- 2026-03-12
- Version
- 1