Facial expression synthesis in videos based on generative adversarial networks and inter-frame coherence

• Ano de defesa: 2024
• Autor(a) principal: Testa, Rafael Luiz
• Tipo de documento: Tese
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Biblioteca Digitais de Teses e Dissertações da USP
• Palavras-chave em Português: Coerência entre frames; Coerência Temporal; Edição de Atributos Faciais; Facial Attribute Editing; Facial Expression Generation; Facial expression manipulation; Facial Expression Reenactment; Facial Expression Synthesis; Generative Adversarial Networks; Geração de Expressão Facial; Interframe Coherence; Manipulação de expressões faciais; Mapeamento de Expressão Facial; Mapping Facial Expression; Redes Adversariais Gerativas; Reencenação da Expressão Facial; Síntese da Expressão Facial; Síntese de Vídeo; Temporal Coherence; Video Synthesis
• Link de acesso: https://www.teses.usp.br/teses/disponiveis/100/100131/tde-18122024-231426/

The ability to synthesize realistic facial expressions has the potential to improve social skills training, particularly for individuals with autism spectrum disorder, for whom recognizing and interpreting facial expressions can be challenging. The primary goal of this research is to develop an innovative approach for synthesizing realistic videos that depict facial expressions of emotions, to maintain individual identity and generate smoother transitions between frames. The research builds upon recent advancements in deep learning, particularly generative adversarial networks, which have shown promise in convincingly reenacting emotions from a reference video. The proposed approach distinguishes itself from other similar approaches by incorporating not only pixel-level features but also shape-based features, derived from facial landmarks, into the loss function to maintain inter-frame coherence. The facial expression synthesis pipeline involves reference video selection, data preparation (including image warping and displacement map generation), and GAN-based frame generation and refinement. The novel contributions include the incorporation of a previous frame into the generator architecture, the addition of two new loss functions that consider the consistency between adjacent frames, and a post-processing step of blending frames. The objective assessment demonstrated superior performance on video-specific measures compared to other image-to-video approaches and the baseline. The subjective evaluation confirmed greater realism, expressiveness, and identity preservation aspects when compared to the baseline and also outperformed real videos in recognizing certain emotions by participants. The proposed approach effectively enhances temporal coherence in facial video synthesis, offering promising implications for developing a wide range of applications, including diagnostic and therapeutic tools as well as potential uses in entertainment, education, and communication technology.