Knowing When to Stop: Joint Heterogeneous Feature Selection and Classification

We consider the problem of joint heterogeneous feature selection and classification when multiple feature sets are present. Specifically, we want to identify which feature sets and features per set to review, and perform classification using this information. To this end, we formulate an optimization problem that considers the cost of reviewing individual features, the switching cost between feature sets, and the associated classification decision cost. The objective is to minimize the expected total cost of reviewing feature sets and associated features and the misclassification cost. We derive the optimum classification decision rule, and show that it minimizes the average misclassification cost. Additionally, we derive the optimum feature review rule, which determines both the feature sets and features per set to be reviewed. We illustrate the performance of the proposed methodology on the application of the automatic classification of civil issues reported on crowdsourcing platforms. We observe that an accurate classification decision can be reached by examining ~2.6 features on average.