Learning from group supervision: the impact of supervision deficiency on multi-label learning

Multi-label learning studies the problem where one instance is associated with multiple labels. Weakly supervised multi-label learning has attracted considerable research attention because of the annotation difficulty. Majority of the studies on weakly supervised multi-label learning assume that one group of weak annotations is available for each instance; however, none of these studies considers multiple groups of weak annotations that can be easily acquired through crowdsourcing. Recent studies on crowdsourced multi-label learning observed that the current query strategies do not agree well with human habits and that data cannot be collected as expected. Therefore, this study aims to design a new query strategy in accordance with human behavior patterns to obtain multiple groups of weak annotations. Further, a learning algorithm is proposed based on neural networks for such type of data. In addition, this study qualitatively and empirically analyzes factors in the proposed query strategy that may impact further learning and provides insights to obtain better query strategy with respect to future crowdsourcing in case of multi-label data.