Cluster-aware Semantic Vector Learning using BERT in Natural Language Understanding

Natural language understanding (NLU) is a core technology for implementing natural interfaces. Recently, embedding sentences and correspondence between texts as extracted semantic knowledge, called semantic frame, has shown that semantic vector representation is key for implementing or supporting robust NLU systems. However, existing studies pertain to only the relations between sentences or only the correspondence between sentences and semantic frames, and do not consider the many-to-l relationship of text-to-semantic frames and semantic clusters. Herein, we propose a novel framework that learns semantic cluster-aware vector representations using bidirectional encoder representations from transformers(BERT). A key technique is cohesion modeling for pulling paraphrase texts to semantic centroids. Another technique is separation modeling for pushing different clusters away by employing a triplet margin loss. Additionally, we propose a novel semantic frame-encoding method using bidirectional encoder representations from trans-formers(BERT). Using the proposed framework, we demonstrate that the proposed model can learn meaningful semantic vector representations.