Authors: Dae Hoe Kim, Wissam J. Baddar, Jinhyeok Jang and Yong Man Ro

Abstract: Facial __expression__ recognition (FER) is increasingly gaining importance in various emerging affective computing applications. In practice, achieving accurate FER is challenging due to the large amount of inter-personal variations such as __expression__ intensity variations. In this paper, we propose a new spatio-temporal feature representation learning for FER that is robust to __expression__ intensity variations. The proposed method utilizes representative __expression__-states (e.g., onset, apex and offset of __expression__s) which can be specified in facial sequences regardless of the __expression__ intensity. The characteristics of facial __expression__s are encoded in two parts in this paper. As the first part, spatial image characteristics of the representative __expression__-state frames are learned via a convolutional neural network. Five objective terms are proposed to improve the __expression__ class separability of the spatial feature representation. In the second part, temporal characteristics of the spatial feature representation in the first part are learned with a long short-term memory of the facial __expression__. Comprehensive experiments have been conducted on a deliberate __expression__ dataset (MMI) and a spontaneous micro-__expression__ dataset (CASME II). Experimental results showed that the proposed method achieved higher recognition rates in both datasets compared to the state-of-the-art methods.