Automated essay scoring for Brazilian Portuguese: evidence from Cross-Prompt evaluation of ENEM essays

• Ano de defesa: 2025
• Autor(a) principal: Barbosa, André
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Biblioteca Digitais de Teses e Dissertações da USP
• Palavras-chave em Português: Aprendizado profundo; Artificial intelligence; Automated essay scoring; Avaliação automática de redações; Deep learning; Inteligência artificial; Natural language processing; Processamento de linguagem natural
• Link de acesso: https://www.teses.usp.br/teses/disponiveis/45/45134/tde-25092025-141609/

Automated essay scoring for Portuguese faces unique challenges due to limited research, fragmented datasets, and complex multi-trait evaluation frameworks like Brazil\'s ENEM. This dissertation investigates cross-prompt, trait-specific scoring formalized as f1r (x, p) = s, where models must evaluate, at test time, essays x on unseen prompts p across five competencies, each scored on a six-point ordinal scale {0, 40, 80, 120, 160, 200}. The investigation systematically compares three model classes: feature-based baselines (72 linguistic features), encoder-only transformers (109M-1.5B parameters), and decoder architectures divided into fine-tuned small language models (2.4B-14.7B parameters) and zero-shot learners (including proprietary models and the 671B-parameter DeepSeekR1). Under varying information access paradigms and rubric conditioning strategies, experiments reveal that no single configuration serves all evaluation needs: encoder models excel at mechanical competencies (fluency, cohesion) despite their inability to access prompts due to context limitations; decoder models achieve superior performance on argumentation (QWK 0.73) and style (QWK 0.60) when provided full context; and language-specific pretraining benefits only surface-level linguistic features without improving reasoning required for more complex tasks. Best-performing models achieve QWK scores of 0.60-0.73 across competencies, remaining 11-23 points below human inter-rater agreement (0.72-0.85), with the largest gaps occurring in style and persuasiveness evaluation. These findings demonstrate that different competencies require fundamentally different computational approaches, ranging from lightweight encoders for grammar assessment to massive decoder models for argumentative analysis, thus informing strategic deployment where institutions must accept substantial performance trade-offs relative to human evaluation.