Bag of meta-words: A novel method to represent document for the sentiment classification

Abstract It is crucial to represent the semantic information of a document in sentiment classification. Various semantic information representation models have been proposed, however existing approaches have their setbacks. Notable weaknesses among these are: (1) tradition VSM methods, completely ignore the semantic information; (2) averaging word embedding methods, cannot depict the synthetical semantic meaning of the given document; (3) neural network methods, require complex structure and are notoriously difficult to be trained. To overcome these limitations, we introduce a simple but novel method which we call bag of meta-words (BoMW). In our method, the semantic information of the document is indicated by a meta-words vector in which every single meta-word element denotes particular semantic information. Especially, these meta-words are extracted from pre-trained word embeddings through two different but complemental models, naive interval meta-words (NIM) and feature combination meta-words (FCM). In general, our new model BoMW is as simple as traditional VSM model but it can capture the synthetical semantic meanings of the document. Numerous experiments on two benchmarks (IMDB dataset and Pang’s dataset) are carried out to verify the effectiveness of the proposed method, and the results show that the performance of our method can exceed the traditional VSM methods and methods using pre-trained word embedding.