Facial expression synthesis in videos based on generative adversarial networks and inter-frame coherence - Testa, Rafael Luiz (2024)
The ability to synthesize realistic facial expressions has the potential to improve social skills training, particularly for individuals with autism spectrum disorder, for whom recognizing and interpreting facial expressions can be challenging. The primary goal of this research is to develop an innovative approach for synthesizing realistic videos that depict facial expressions of emotions, to maintain individual identity and generate smoother transitions between frames. The research builds upon recent advancements in deep learning, particularly generative adversarial networks, which have shown promise in convincingly reenacting emotions from a reference video. The proposed approach distinguishes itself from other similar approaches by incorporating not only pixel-level features but also shape-based features, derived from facial landmarks, into the loss function to maintain inter-frame coherence. The facial expression synthesis pipeline involves reference video selection, data preparation (including image warping and displacement map generation), and GAN-based frame generation and refinement. The novel contributions include the incorporation of a previous frame into the generator architecture, the addition of two new loss functions that consider the consistency between adjacent frames, and a post-processing step of blending frames. The objective assessment demonstrated superior performance on video-specific measures compared to other image-to-video approaches and the baseline. The subjective evaluation confirmed greater realism, expressiveness, and identity preservation aspects when compared to the baseline and also outperformed real videos in recognizing certain emotions by participants. The proposed approach effectively enhances temporal coherence in facial video synthesis, offering promising implications for developing a wide range of applications, including diagnostic and therapeutic tools as well as potential uses in entertainment, education, and communication technology.