ServiceNow Research

Towards Text Generation with Adversarially Learned Neural Outlines


Recent progress in deep generative models has been fueled by two paradigms – au- toregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs. We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative condi- tioning for the autoregressive stage. Our quantitative evaluations suggests that conditioning information from generated outlines is able to guide the autoregressive model to produce realistic samples, comparable to maximum-likelihood trained language models, even at high temperatures with multinomial sampling. Qualita- tive results also demonstrate that this generative procedure yields natural-looking sentences and interpolations.

Conference on Neural Information Processing Systems (NeurIPS)
Sai Rajeswar Mudumba
Sai Rajeswar Mudumba
Research Scientist

Research Scientist at Human Decision Support located at Montreal, QC, Canada.

Christopher Pal
Christopher Pal
Distinguished Scientist

Distinguished Scientist at Low Data Learning located at Montreal, QC, Canada.