Triplet Loss With Multistage Outlier Suppression and Class-Pair Margins for Facial Expression Recognition

Weicheng Xie Haoqian Wu Yi Tian Mengchao Bai Linlin Shen

Computer Vision Institute, School of Computer Science and Software Engineering, Shenzhe University, Shenzhen 518060, China
Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen University, Shenzhen 518060, China
Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen 518060, China

Abstract

Deep metric based triplet loss has been widely used to enhance inter-class separability and intra-class compactness of network features. However, the margin parameters in the triplet loss for current approaches are usually fixed and not adaptive to the variations among different expression pairs. Meanwhile, outlier samples like faces with confusing expressions, occlusion and large head poses may be introduced during the selection of the hard triplets, which may deteriorate the generalization performance of the learned features for normal testing samples. In this work, a new triplet loss based on class-pair margins and multistage outlier suppression is proposed for facial expression recognition (FER). In this approach, each expression pair is assigned with an order-insensitive or two order-aware adaptive margin parameters. While expression samples with large head poses or occlusion are firstly detected and excluded, abnormal hard triplets are discarded if their feature distances do not fit the model of normal feature distance distribution. Extensive experiments on seven public benchmark expression databases show that the network using the proposed loss achieves much better accuracy than that using the original triplet loss and the network without using the proposed strategies, and the most balanced performances among state-of-the-art algorithms in the literature

Visualization

The bias defined in equation (1) for all the expression pairs, the network used and the experimental setting are same with Section III-A and the experiment in Table II

The abnormal ‘happy’ expressions

The order-insensitive class-pair margins

Example faces with different pan and tilt angles in the posed face database

Random distance and the feature distance distribution and the rejection regions

The network structure of ResNet

Example images of the seven public datasets

The confusion matrices without (a) and with (b) the proposed class-pair margins, the values of the adjusted pairwise margins (c) for the FER2013 database and their evolution curves (d)

The confusion matrices

The posed (the 1st row) and occluded (the 2nd row) faces detected with face-pose regressor and multi-task CNN for the FER2013 database

Largely posed and occluded faces and confusing outliers

The average of the feature maps generated by the baseline and the proposed algorithm

The 25th feature map generated by the baseline and the proposed algorithm


@ARTICLE{9367174,
  author={Xie, Weicheng and Wu, Haoqian and Tian, Yi and Bai, Mengchao and Shen, Linlin},
  journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
  title={Triplet Loss With Multistage Outlier Suppression and Class-Pair Margins for Facial Expression Recognition}, 
  year={2022},
  volume={32},
  number={2},
  pages={690-703},
  doi={10.1109/TCSVT.2021.3063052}}