DeB3RTa: Um modelo de linguagem pré-treinado para análise de textos financeiros em português usando domínios mistos

• Ano de defesa: 2025
• Autor(a) principal: PIRES, Higo Felipe Silva
• Orientador(a): PAUCAR CASAS, Vicente Leonardo
• Banca de defesa: PAUCAR CASAS, Vicente Leonardo , CARVALHO, João Paulo Baptista de , OLIVEIRA, Roberto Célio Limão de , SERUFFO, Marcos César da Rocha , FONSECA NETO, João Viana da
• Tipo de documento: Tese
• Tipo de acesso: Acesso aberto
• Idioma: por
• Instituição de defesa: Universidade Federal do Maranhão
• Programa de Pós-Graduação: PROGRAMA DE PÓS-GRADUAÇÃO EM ENGENHARIA DE ELETRICIDADE/CCET
• Departamento: DEPARTAMENTO DE ENGENHARIA DA ELETRICIDADE/CCET
• País: Brasil
• Palavras-chave em Português: Processamento de Linguagem Natural em Português; Processamento de Linguagem Financeira; Transformers; Pré-treinamento de Domínio Misto; Decaimento da Taxa de Aprendizado por Camada; Regularização de Mixout; Reinicialização de Camadas; Arquitetura DeBERTa; Classificação de Textos Financeiros
• Palavras-chave em Inglês: Portuguese Natural Language Processing; Financial Language Processing; Transformers; Mixed Domain Pre-training; Layer-wise Learning Rate Decay; Mixout Regularization; Layer Resetting; DeBERTa Architecture; Financial Text Classification
• Área do conhecimento CNPq: Arquitetura de Sistemas de Computação
• Link de acesso: https://tedebc.ufma.br/jspui/handle/tede/6179

The complex and specialized terminology of financial language in Portuguese-speaking markets create significant challenges for natural language processing (NLP) applications, which must capture nuanced linguistic and contextual information to support accurate analysis and decision-making. This paper presents DeB3RTa, a transformer-based model specifically developed through a mixed-domain pretraining strategy that combines extensive corpora from finance, politics, business management, and accounting to enable a nuanced understanding of financial language. DeB3RTa was evaluated against prominent models—including BERTimbau, XLM-RoBERTa, SEC-BERT, BusinessBERT, and GPTbased variants — and consistently achieved significant gains across key financial NLP benchmarks. To maximize adaptability and accuracy, DeB3RTa integrates advanced finetuning techniques such as layer reinitialization, Mixout regularization, stochastic weight averaging, and layer-wise learning rate decay, which together enhance its performance across varied and high-stakes NLP tasks. These findings underscore the efficacy of mixed-domain pretraining in building high-performance language models for specialized applications. With its robust performance in complex analytical and classification tasks, DeB3RTa offers a powerful tool for advancing NLP in the financial sector and supporting nuanced language processing needs in Portuguese-speaking contexts.