A language modeling-like approach to sketching

Abstract Sketching is a universal communication tool that, despite its simplicity, is able to efficiently express a large variety of concepts and, in some limited contexts, it can be even more immediate and effective that natural language. In this paper we explore the feasibility of using neural networks to approach sketching in the same way they are commonly used in Language Modeling. We propose a novel approach to what we refer to as “Sketch Modeling”, in which a neural network is exploited to learn a probabilistic model that estimates the probability of sketches. We focus on simple sketches and, in particular, on the case in which sketches are represented as sequences of segments. Segments and sequences can be either given – when the sketches are originally drawn in this format – or automatically generated from the input drawing by means of a procedure that we designed to create short sequences, loosely inspired by the human behaviour. A Recurrent Neural Network is used to learn the sketch model and, afterwards, the network is seeded with an incomplete sketch that it is asked to complete, generating one segment at each time step. We propose a set of measures to evaluate the outcome of a Beam Search-based generation procedure, showing how they can be used to identify the most promising generations. Our experimental analysis assesses the feasibility of this way of modeling sketches, also in the case in which several different categories of sketches are considered.