MATRICS: A System for Human-Machine Hybrid Forecasting of Geopolitical Events

In this paper, we present MATRICS, a human-machine hybrid system that accurately performs geopolitical forecasting by combining crowdsourcing with ensemble machine learning on online data. The system employs a pair of parallel, but highly-interconnected processing pipelines to perform “machine-aided human forecasting” and “human-aided machine forecasting”. This configuration allows the machine to provide information to the human population, saving research time and reducing fatigue, while simultaneously allowing the human population to provide feedback to the machine learning components, allowing them to filter data sources and quickly adapt to a task via online machine learning. The final forecast for each question was computed as an aggregate of the human and machine responses. The system was evaluated using data collected during the IARPA Hybrid Forecasting Competition, in which it answered 187 forecasting questions with a mean Brier score of 0.27 using volunteers and participants that were recruited via Amazon Mechanical Turk and open-source “big” data scraped from online sources such as social media, search engine results, and online historical data.