A unified deep sparse graph attention network for scene graph generation

Abstract Scene graph generation (SGG) plays an important role in deep understanding of the visual scene. Despite the empirical success of traditional methods in many applications, they still have several challenges in the high computational complexity of dense graph and the inaccurate pruning of sparse graph. To tackle these problems, we propose a novel deep sparse graph attention network to mine the rich contextual clues and simultaneously preserve the statistical co-occurrence knowledge of SGG. Specifically, our Relationship Measurement Network (RelMN) is adapted to first classify all object pairs in dense graph as the foreground and background categories to filter the false relationships and then construct a sparse graph efficiently. Meanwhile, we design a novel feature aggregation and update method via graphical message passing to jointly learn the node and edge features for object recognition and relationship classification in the graph attention network. Extensive experimental results on the large scale VG and VRD datasets demonstrate our proposed method outperforms several state-of-the-art approaches.