History-based attention in Seq2Seq model for multi-label text classification

Abstract Multi-label text classification is an important yet challenging task in natural language processing. It is more complex than single-label text classification in that the labels tend to be correlated. To capture this complex correlations, sequence to sequence model has been widely applied, and achieved impressing performance for multi-label text classification. It encodes each document as contextual representations, and then decodes them to generate labels one by one. At each time step, the decoder usually adopts the attention mechanism to highlight important contextual representations to predict a related label, which has been proved to be effective. Nevertheless, the traditional attention approaches only utilize a hidden state to explore such contextual representations, which may result in prediction errors, or omit several trivial labels. To tackle this problem, in this paper, we propose “history-based attention”, which takes history information into consideration, to effectively explore informative representations for labels’ predictions in multi-label text classification. Our approach consists of two parts: history-based context attention and history-based label attention. History-based context attention considers historical weight trends to highlight important context words, which is helpful to predict trivial labels. History-based label attention explores historical labels to alleviate the error propagation problem. We conduct experiments on two popular text datasets (i.e., Arxiv Academic Paper Dataset and Reuters Corpus Volume I), it is demonstrated that the history-based attention mechanism could boost the performance to a certain extent, and the proposed method consistently outperforms highly competitive approaches.