Hierarchical multi-view context modelling for 3D object classification and retrieval

Abstract Recent advances in 3D sensors and 3D modelling software have led to big 3D data. 3D object classification and retrieval are becoming important but challenging tasks. One critical problem for them is how to learn the discriminative multi-view visual characteristics. To address it, we proposes a hierarchical multi-view context modelling method (HMVCM). It consists of four key modules. First, the module of view-level context learning is designed to learn visual context features with respect to individual views and their neighbours. This module can imitate the human need to look back and forth to identify and compare the discriminative parts of individual 3D objects based on a joint convolutional neural network (CNN) and bidirectional long short-term memory (Bi-LSTM) network. Then, a multi-view grouping module is introduced to split views into several groups based on their visual appearance. A raw group-level representation can be obtained by the weighted sum of the view-level descriptors. Furthermore, we employ the Bi-LSTM to exploit the context among adjacent groups to generate group-wise context features. Finally, all group-wise context features are fused into a compact 3D object descriptor according to their significance. Extensive experiments on ModelNet10, ModelNet40 and ShapeNetCore55 demonstrate the superiority of the proposed method.