End-to-End Sign Language Translation Pipeline Using Human Keypoints and Transformers

• Ano de defesa: 2025
• Autor(a) principal: Souza, Wesley Ferreira Maia de
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Biblioteca Digitais de Teses e Dissertações da USP
• Palavras-chave em Português: Accessibility; Acessibilidade; Arquiteturas transformers; Computer vision; MediaPipe; Sign language translation; Tradução de língua de sinais; Transformer architectures; Visão computacional
• Link de acesso: https://www.teses.usp.br/teses/disponiveis/55/55137/tde-03112025-141248/

Communication between deaf and hearing individuals continues to face significant barriers, and automatic sign language translation into text is a critical tool to promote accessibility and social inclusion. This study aims to develop sign language translation models that can improve the accuracy and fluency of automatic translations. We evaluated the performance of three translation approaches using two widely recognized datasets, Phoenix-2014T and How2Sign: translation from sign language to glosses (Sign2Gloss), from glosses to text (Gloss2Text), and an integrated process combining these two phases (Sign2Gloss2Text). The developed methods demonstrated competitive results. The Sign2Gloss model achieved a word error rate (WER) of 24.69, a metric where lower scores indicate higher accuracy, while the Gloss2Text model obtained a BLEU-4 score of 24.67, a metric for evaluating the quality of machine-translated text. The integrated Sign2Gloss2Text model achieved a BLEU-4 score of 21.30 on the Phoenix-2014T dataset, showing performance similar to previous methods that used word embeddings. Although these results are promising, direct sign language to text translation remains challenging, as evidenced by lower BLEU-4 scores on the How2Sign dataset. These findings highlight the importance of incorporating an intermediate gloss phase to improve translation quality. However, limitations such as the restricted domain of the datasets and the unidirectional approach suggest the need for further investigation.