Relevance Prediction in Information Extraction using Discourse and Lexical Features

We present on-going work on estimating the relevance of the results of an Information Extraction (IE) system. Our aim is to build a user-oriented measure of utility of the extracted factual information. We describe experiments using discourse-level features, with classifiers that learn from users’ ratings of relevance of the results. Traditional criteria for evaluating the performance of IE focus on correctness of the extracted information, e.g., in terms of recall, precision and F-measure. We introduce subjective criteria for evaluating the quality of the extracted information: utility of results to the end-user. To measure utility, we use methods from text mining and linguistic analysis to identify features that are good predictors of the relevance of an event or a document to a user. We report on experiments in two real-world news domains: business activity and epidemics of infectious disease.