Isolated sign language recognition in LIBRAS

• Ano de defesa: 2025
• Autor(a) principal: Delucis, Marcelo Mussi
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Pontifícia Universidade Católica do Rio Grande do Sul; Escola Politécnica; Brasil; PUCRS; Programa de Pós-Graduação em Ciência da Computação
• Palavras-chave em Português: LIBRAS; Isolated Sign Language Recognition; Vision Transformers; Reconhecimento Isolado de Línguas de Sinais; CIENCIA DA COMPUTACAO::TEORIA DA COMPUTACAO
• Link de acesso: https://tede2.pucrs.br/tede2/handle/tede/11629

The present work focuses on the isolated recognition of sign language in Brazilian Sign Language (LIBRAS), essential for promoting digital accessibility for the Deaf community. However, data scarcity and the limited diversity of available signs and actors hinder the development of models capable of generalization and advancement in the field. Previous works, such as the MINDS dataset, are limited to reduced vocabularies, controlled environments, and low signer diversity, which tends to result, in some cases, in super-specialized models with low accuracy in scenarios different from what is seen in the training set. To address current limitations, a dataset, MALTA-LIBRAS, was developed, constructed by collecting publicly available LIBRAS videos, introducing variability in signers, environments, and recording conditions. Three architectures based on Transformers, VideoMAE, TimeSformer, and ViViT, are investigated in three experimental configurations: pre-training on action recognition datasets, application of data augmentation strategies, and exploration of possible knowledge transfer between sign languages using datasets from North American and Russian sign languages. Results on the MALTA-LIBRAS dataset indicate that models pre-trained on action recognition tasks achieve 29% accuracy, while models without pre-training achieve the equivalent of random prediction. Data augmentation techniques aid model generalization, increasing accuracy from 29% to 33.6%. Knowledge transfer between languages to LIBRAS proved limited, with gains of 2.7% in accuracy, reinforcing the need for domain-specific adaptation. It is concluded that data diversity (signers, environments) is as crucial as volume for real-world applications, and a unified framework for SLR in low-resource scenarios is proposed, combining pre-training on human actions, targeted data augmentation, and fine-tuning.