Enabling Language Representation with Knowledge Graph and Structured Semantic Information

Pre-trained language models have been widely recognized and applied. While common pre-training language representation models(PLMs) usually focus on grasping the co-occurrence of words or sentences in simple tasks, more and more researchers realize that external information, i.e., knowledge graph (KG) and clear structured semantics, can be vital in natural language understanding tasks. Therefore, using external information to enhance PLMs (such as BERT) has gradually become a popular direction. However, the existing improvement methods often only use a certain type of external information, and it is difficult to solve the problems of common PLMs that lack common sense and semantic incompleteness in one fell swoop. Suppose the model wants to integrate multiple external information. In that case, it not only requires the model to deal with the noise problem that external information may bring but also requires the model to ensure that different information can work together effectively. In this paper, we propose Sem-K-BERT, which integrates the information of KG and semantic role labeling(SRL) before and after the BERT encoding layer, and introduces a context-aware knowledge screening mechanism based on semantic correlation calculation and a text-semantic alignment mechanism to effectively integrate the two external information and reduce the impact of noise. Experiments and analysis on 8 different Chinese natural language processing tasks show that Sem-K-BERT has better performance than BERT and the BERT model that only incorporates KG. This indicates that the simultaneous use of knowledge graph and SRL information offers a promising solution to improve the performance of PLMs.