Face Tells Detailed Εxpression: Generating Comprehensive Facial Εxpression Sentence through Facial Action Units
Authors: Joanna Hong, Hong Joo Lee, Yelin Kim, Yong Man Ro
 
Human facial ｅxpression plays the key role in the understanding of the social behavior. Many deep learning approaches present facial emotion recognition and automatic image captioning considering human sentiments. However, most current deep learning models for facial ｅxpression analysis do not contain comprehensive, detailed information of a single face. In this paper, we newly introduce a text-based facial ｅxpression description using several essential components describing comprehensive facial ｅxpression: gender, facial action units, and corresponding intensities. Then, we propose comprehensive facial ｅxpression sentence generating model along with facial ｅxpression recognition model for a single facial image to verify the effectiveness of our text-based dataset. Experimental results show that the proposed two models are supporting each other improving their performances: the text-based facial ｅxpression description provides comprehensive semantic information to the facial emotion recognition model. Also, the visual information from the emotion recognition model guides the facial ｅxpression sentence generation to produce a proper sentence describing comprehensive description.