Guardians of Academic Integrity: Multilingual Fusion Models for AI Essay Detection

TL;DR

As Large Language Models (LLMs) like ChatGPT become ubiquitous, the line between student work and machine output has blurred. This paper introduces EssayDetect, a fusion model that marries the deep semantic understanding of Pre-trained Language Models (PLMs like DeBERTa and AraBERT) with traditional stylometric analysis. By focusing on linguistic nuances and addressing class imbalance via Focal Loss, the team achieved top-tier performance in both English (0.978 F1) and Arabic (0.9429 F1) academic essay detection.

Problem & Motivation: The "False Positive" Trap

Current AI detection methods often suffer from an over-reliance on probability-based patterns, leading to high false-positive rates. This is particularly damaging in academia, where a non-native English speaker's formal writing might be unfairly flagged as "robotic." Furthermore, most detectors are optimized for English, leaving languages with complex morphology—like Arabic—under-protected. The authors recognized that while PLMs are good at context, they often ignore stylometry (the unique "DNA" of an author's writing style, such as punctuation frequency or lexical diversity).

Methodology: The Fusion Architecture

The core innovation of this work is the Fusion Model, which acts as a two-pronged detector:

1. Deep Semantic Path: The model extracts the [CLS] token from a transformer (e.g., DeBERTa for English, AraBERT for Arabic) to capture global context.
2. Stylometric Path: It calculates 21+ features including phraseology (sentence length), lexical diversity (Type-Token Ratio), and syntactic diversity (POS proportions).
3. Self-Attention & Selection: Instead of just dumping features in, the authors use LIME to select the most discriminative features and a self-attention mechanism to capture how these stylistic markers depend on one another.

Architecture Overview

The fusion happens at the final layers. Interestingly, the authors use layer-wise freezing, keeping the lower layers of the PLM intact to preserve general language understanding while fine-tuning only the top layers ($k=6$) to specialize in essay-specific styles.

Experiments & Results

The model was tested against the COLING 2025 DAIGenC Workshop benchmarks.

Key Breakthroughs:

• English (Subtask A): DeBERTa blossomed when combined with features, jumping from a baseline F1 of 0.617 to a staggering 0.978.
• Arabic (Subtask B): AraBERT, which is already optimized for Arabic morphology, reached 0.9429 with stylometric features.

Results Comparison

The confusion matrices indicate that while the model is highly accurate, Arabic essays remain slightly more difficult to classify than English ones, likely due to the "rich morphology" and less strict word order of the Arabic language which can camouflage AI patterns.

Deep Insight: Why Stylometry Matters

The LIME analysis revealed fascinating differences:

• AI Tendencies: Machine-generated essays usually show "mechanical" consistency—higher average word counts and more frequent use of periods, but lower emotional nuance in sentiment polarity.
• Human Nuance: Human writing tends to have higher lexical diversity and more varied sentence structures that "break" the predictable flow of a standard LLM.

Feature Distribution Analysis

Critical Analysis & Conclusion

The EssayDetect framework proves that "black-box" transformers are not enough. By re-integrating traditional linguistic features, we regain interpretability.

Limitations: The model's performance relies on the availability of high-quality stylometric extractors. For Arabic, features like POS (Part-of-Speech) tags are harder to extract accurately than in English.

Conclusion: This work sets a new standard for multilingual GenAI detection in education. By combining the "what" (semantics from PLMs) with the "how" (style from stylometry), we can better safeguard academic integrity in an era where AI is just a prompt away.