Unsupervised Orthogonal Facial Representation Extraction via image reconstruction with correlation minimization

Abstract The identity and expression are two orthogonal facial properties which are hard to disentangle from facial images. The identity representation and emotional representation can be used for face verification and facial expression recognition. From the view of Multi-Task Learning, jointly learning both representations will generally benefit the performance. However, it is rarely considered in recent studies. In this paper, the orthogonal facial properties are modeled in a unified framework. A deep Convolutional–Deconvolutional neural network (Conv–Deconv) is proposed for extracting identity representations and emotional representations from aligned faces. A reconstruction loss, a classification loss, and a correlation loss are employed to train this network. The classification loss is used to learn emotional representations. The correlation loss ensures the independence of the identity representation and the emotional representation. The reconstruction loss confirms the information completeness of the combination of the identity representation and the emotional representation. The proof of the correctness of the proposed losses is also given in detail. The network learns both identity representation and emotional representations with only emotional labels. Compared to the existing method, the need of identity labels is eliminated, and the applicable scope is extended. Thus, we name it Unsupervised Orthogonal Facial Representation Extraction. Experiments are carried out on the synthesized LSFED dataset and the constrained RaFD dataset. The 2D t-SNE visualization and a deviation based score are used for validating the orthogonality of the representation. Furthermore, to quantitatively evaluate the learned representation, the Euclidean distance between the identity representations is used for face verification. The performance of our unsupervised face verification is comparable to the existing supervised methods.