AL AASAR JOURNAL • pp. 214-236
HUMAN VERSUS AI ASSESSMENT OF ENGLISH ACADEMIC WRITING A COMPARATIVE ANALYSIS OF SCORING RELIABILITY AND VALIDITY
Abstract
The increasing use of generative artificial intelligence (AI) in education has created new opportunities for assessing students' academic writing. However, there are doubts about the reliability, agreement, and validity of the scores produced by AI versus human evaluation. The objective of this study was to investigate how three human and three Artificial Intelligence (AI) models: ChatGPT, Gemini, and Claude assessed the academic writing of English. The literature review focused on traditional human-based writing evaluation, automated essay scoring, generative AI, reliability, human–AI agreement and validity issues, highlighting the need for context-specific evidence from Pakistan. A quantitative comparative research design was used. The students of selected three divisions of Punjab (Lahore, Gujranwala and Multan) of university undergraduate and postgraduate level were the target population. Three human writing teachers and the three AI systems evaluated 148 academic writing samples from the English language with the same evaluation system, the results of which form the end data. Descriptive statistics and intraclass correlation coefficients (ICC), Pearson correlation, paired-samples t-tests, and trait-level correlation analysis were performed on this data. The inter-rater reliability results were good, with strong average-measures inter-rater reliability between the three human evaluators (ICC = 0.875). The mean score of all three AI systems was lower than the composite human rating. The mean difference was highest for ChatGPT (3.29 points) followed by Gemini (1.64 points) and Claude (1.19 points). Positive correlations were found between the human and AI scores, with Claude showing the strongest correlation (r = 0.835), followed by Gemini (r = 0.814) and ChatGPT (r = 0.782). Trait level results showed higher correlations of grammatical accuracy and vocabulary than of content and argumentation skills. The overall result of the study is that generative AI systems can be used as a valuable supplementary tool to support academic writing assessment in English, however, the variations in scoring and the availability of reliability and validity evidence suggests that the scores generated by generative AI cannot be substituted for human scores. The study suggests that combining AI with a human evaluation process is a recommended strategy for assessing the study and that the current study should be replicated using AI scoring and more robust agreement and validity analyses.
Keywords
Record
- Identifier
- https://doi-services.com/1101/2198
- Published
- 2026-03-19
- Version
- 1