Active Advice Seeking for Inverse Reinforcement Learning

Intelligent systems that interact with humans typically require input in the form of demonstrations and/or advice for optimal decision making. In more traditional systems, such interactions require detailed and tedious effort on the part of the human expert. Alternatively, active learning systems allow for incremental acquisition of the demonstrations from the human expert where the learning system generates the queries. However, active learning allows for only labeled examples as input, significantly restricting the interaction between expert and learning algorithm. Advice-based learning systems increase the expressiveness of the interaction, but typically require all the advice about the domain in advance. By combining active learning and advice-based learning, we consider the problem of actively soliciting human advice. We present the algorithm in an inverse reinforcement learning setting where the utilities are learned from demonstrations. We show empirically the contribution of a more expressive advice over traditional active learning approaches.