Zero-Shot Learning: An Energy Based Approach

Zero-shot learning deals with the problem when the training domain and the test domain have different class sets of image instances. To tackle the problem of some classes in the test data never appeared in the training set, a most popular approach is to map both images and classes in a common space under the embedding based framework. Nevertheless, most embedding based models suffered from the semantic loss problem. Furthermore, the expressive power is limited by representing classes and images as mere points. To tackle these problems, in this paper, we propose an Energy-Based Zero-shot Learning model (EBZL) to encode the association between class attributes and input images for zero-shot learning. EBZL is composed of two parts. The first part is a variational autoencoder that reduces the input dimension of images with representative hidden representations. By feeding the hidden representations as the input of the second part, the second part works as the energy function part based on the deep Boltzmann machine. Specifically, we adapt tradition deep Boltzmann machine to a supervised setting without changing its property as an undirected probabilistic graphic model, which helps to preserve semantic integrity and circumvents semantic loss problem. We further utilize variational inference techniques and mean-field approximation to reduce time complexity in model training process. Finally, extensive experimental results on several real-world datasets clearly show the effectiveness of our proposed method.