Learning to execute instructions in a Minecraft dialogue

The Minecraft Collaborative Building Task is a two-player game in which an Architect (A) instructs a Builder (B) to construct a target structure in a simulated Blocks World Environment. We define the subtask of predicting correct action sequences (block placements and removals) in a given game context, and show that capturing Bs past actions as well as Bs perspective leads to a significant improvement in performance on this challenging language understanding problem.