Auxiliary task guided mean and covariance alignment network for adversarial domain adaptation

Abstract Adversarial domain adaptation (ADA) learns representations with strong transferability by eliminating the discrepancy between the probability distributions of the source and the target domains. Conventional ADA methods usually employ the Wasserstein distance as a discrepancy measure and train the classifier only from the source domain data. We show that such methods actually only consider first-order statistics in the latent feature space and the discriminability of the learned representations is not fully explored. In this paper, we propose a novel method called auxiliary task guided mean and covariance alignment network (AT-MCAN) for unsupervised domain adaptation. To take the second-order statistics differences into consideration, AT-MCAN introduces a covariance-aware divergence metric to align the distributions of two domains. To enhance the discriminability of the features, AT-MCAN introduces an auxiliary clustering task to the target domain so that the classifier can employ the data from both domains. We provide both theoretical analysis on the generalization bound and empirical evaluations on standard benchmarks to show the effectiveness of our proposed AT-MCAN.