An Interactive Scene Generation Using Natural Language

Scene generation is an important step of robotic drawing. Recent works have shown success in scene generation conditioned on text using a variety of approaches, with which the generated scenes cannot be revised after its generation. To allow modification on generated scenes, we model the scene generation process as a discrete event system. Instead of training text-to-pixel mappings using large datasets, the proposed approach uses object instances retrieved from the Internet to synthesize scenes. Evaluated on 128 experiments using MSCOCO evaluation dataset, the result shows the scene generation performance has been increased by 197%, 22.3%, and 55.7% compared with the state of the art approach on three standard metrics (CIDEr, ROUGH-L, METEOR), respectively. Human evaluation conducted on Amazon Mechanical Turk shows over 80% of generated scenes are considered to have higher recognizability and better alignment with natural language descriptions than baseline works.