Towards Text Generation With Adversarially Learned Neural Outlines

Authors:
Sandeep Subramanian University of Montreal
Sai Rajeswar Mudumba University of Montreal
Alessandro Sordoni Microsoft Research Montreal
Adam Trischler Microsoft
Aaron Courville U. Montreal
Chris Pal MILA, Polytechnique Montréal, Element AI

Introduction:

Recent progress in deep generative models has been fueled by two paradigms -- autoregressive and adversarial models.The authors propose a combination of both approaches with the goal of learning generative models of text.

Abstract:

Recent progress in deep generative models has been fueled by two paradigms -- autoregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs.We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative conditioning for the autoregressive stage. Our quantitative evaluations suggests that conditioning information from generated outlines is able to guide the autoregressive model to produce realistic samples, comparable to maximum-likelihood trained language models, even at high temperatures with multinomial sampling. Qualitative results also demonstrate that this generative procedure yields natural-looking sentences and interpolations.

You may want to know: