Adaptive Weighting of Handcrafted Feature Losses for Facial Expression Recognition

Weicheng Xie Linlin Shen Jinming Duan

Computer Vision Institute, School of Computer Science and Software Engineering, Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen 518060, China
Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen 518060, China
the School of Computer Science, University of Birmingham, Birmingham B15 2TT, U.K.

Abstract

Due to the importance of facial expressions in human–machine interaction, a number of handcrafted features and deep neural networks have been developed for facial expression recognition. While a few studies have shown the similarity between the handcrafted features and the features learned by deep network, a new feature loss is proposed to use feature bias constraint of handcrafted and deep features to guide the deep feature learning during the early training of network. The feature maps learned with and without the proposed feature loss for a toy network suggest that our approach can fully explore the complementarity between handcrafted features and deep features. Based on the feature loss, a general framework for embedding the traditional feature information into deep network training was developed and tested using the FER2013, CK+, Oulu-CASIA, and MMI datasets. Moreover, adaptive loss weighting strategies are proposed to balance the influence of different losses for different expression databases. The experimental results show that the proposed feature loss with adaptive weighting achieves much better accuracy than the original handcrafted feature and the network trained without using our feature loss. Meanwhile, the feature loss with adaptive weighting can provide complementary information to compensate for the deficiency of a single feature.

Visualization

Evolution of the searching space as the algorithm proceeds

Data preprocessing

Proposed network

Structures

Loss weighting processes

Network structure of ResNet

Example images

Comparison

Example faces that are correctly recognized using the feature trained with the proposed feature loss

Heat maps of the average feature maps of the deep networks trained with and without the proposed feature loss

Means and variances of the recognition rates of ten runs with the SoftMax and proposed feature losses

Weight evolution of the alternative feature loss for ten-fold training on the Oulu-CASIA database

Confusion matrix


@ARTICLE{8786929,
  author={Xie, Weicheng and Shen, Linlin and Duan, Jinming},
  journal={IEEE Transactions on Cybernetics}, 
  title={Adaptive Weighting of Handcrafted Feature Losses for Facial Expression Recognition}, 
  year={2021},
  volume={51},
  number={5},
  pages={2787-2800},
  doi={10.1109/TCYB.2019.2925095}}