Normalization of face illumination with photorealistic texture via deep image prior synthesis

Abstract Illumination normalization for face recognition is of great importance when the face image is taken under a harsh lighting condition and only one single image is available. In this paper, an illumination normalization method that can preserve face identity and make the processed image have photorealistic face texture is proposed. First, an illumination regression filter is proposed to remove a large number of illumination components of the input image. Then, the accelerated proximal gradient algorithm is used for low-rank decomposition of the filtered image, which further reduces the residual illumination and noise of the face image. However, after these two operations, the face image has a peeled-off appearance. To avoid the unrealistic appearance, a deep image prior synthesis process is developed to synthesize the photorealistic face texture. To accomplish this task, the filtered image, the low-rank decomposed image, the original image and any a uniformly illuminated frontal face image as reference are used as synthetic ingredients. These ingredients, as prior knowledge, synthesize a face image that has the texture features of the reference image and retains the identity of the input image. We try to bridge the appearance gap between the synthetic ingredients and the final synthetic image by a deep neural network. Instead of training the network on a large dataset, the final synthetic image is generated by iterating the parameters of the generator-structured network according to a certain proportion of the synthetic ingredients. Systematic evaluations conducted on public databases demonstrate that the proposed method is robust to illumination with better performance than state-of-the-art methods.