Deep 3D Caricature Face Generation with Identity and Structure Consistency

Abstract This paper proposed a novel approach to generate face caricatures automatically from a single portrait image. We decompose the process of 3D face caricatures generation into two independent subtasks: appearance transfer of texture and the geometry transfer of mesh. For the appearance transfer, we design a GAN-based network named CariFaceGAN to learn the style mapping from portrait to caricature, in which facial features are leveraged to preserve identity consistency. For geometry transfer, we first learn the transformation of the landmarks between portraits and caricatures in an embedded space obtained with Locally Linear Embedding method, and then Kriging interpolation is used to manipulate the portrait mesh constructed from a single image. The experimental results show that our proposed CariFaceGAN outperforms the state-of-the-art methods in terms of maintaining identity consistency and providing satisfactory visual effects.