Introduction to Binary Coordinate Ascent: New Insights into Efficient Feature Subset Selection for Machine Learning

Feature selection has been an active area of research in machine learning area, and a number of techniques have been developed for selecting an optimal or sub-optimal subset of features, because it is a major factor to determine the performance of a machine-learning technique. In this study, we propose and develop a novel optimization technique, namely, a binary coordinate ascent (BCA) algorithm inspired by the coordinate descent algorithm. The BCA algorithm is an iterative deterministic local optimization approach that can be coupled with wrapper, filter, or hybrid feature selection (FS) techniques. The algorithm searches throughout the space of binary coded input variables by iteratively optimizing the objective function in each dimension at a time. We investigated our BCA approach in a wrapper-based FS framework for the task of classification. In this framework, area under the receiver-operating-characteristic (ROC) curve (AUC) is used as the criterion to find the best subset of features. We evaluated our BCA-based FS in optimization of features for support vector machine, multilayer perceptron, and Naive Bayes classifiers with five publicly available datasets. Our experimental datasets are distinct in terms of the number of attributes (ranging from 18 to 60), and the number of classes (binary or multi-class classification). The efficiency in terms of the number of subset evaluations was improved substantially (by factors of 5–40) compared with two popular FS meta-heuristics, i.e., sequential forward selection (SFS) and sequential floating forward selection (SFFS), while the classification performance for unseen data was maintained.