Evaluating recommender systems in feature model configuration

Configurators can be evaluated in various ways such as efficiency and completeness of solution search, optimality of the proposed solutions, usability of configurator user interfaces, and configuration consistency. Due to the increasing size and complexity of feature models, the integration of recommendation algorithms with feature model configurators becomes relevant. In this paper, we show how the output of a recommender system can be evaluated within the scope of feature model configuration scenarios. Overall, we argue that the discussed ways of measuring recommendation quality help developers to gain a broader view on evaluation techniques in constraint-based recommendation domains.