Convolutional Neural Network-Based Block Up-Sampling for HEVC

Recently, convolutional neural network (CNN) based methods have achieved remarkable progress in image and video super-resolution, which inspires the researches of down/upsampling- based image and video coding using CNN. Instead of hand-crafted filters for up-sampling, trained CNN models are believed to be more capable in improving image quality, and thus lead to coding gain. However, previous studies either concentrated on intra-frame coding, or performed down- and upsampling of entire frame. In this paper, we introduce block-level down- and up-sampling into inter-frame coding with the help of CNN. Specifically, each block in P or B frames can either be compressed at the original resolution, or be down-sampled and compressed at low resolution and then up-sampled by trained CNN models. Such block-level adaptivity is flexible to cope with the spatially variant texture and motion characteristics. We further investigate how to enhance the capability of CNN-based upsampling by utilizing reference frames, and study how to train the CNN models by using encoded video sequences. We implement the proposed scheme onto the High Efficiency Video Coding (HEVC) reference software, and perform a comprehensive set of experiments to evaluate our methods. Experimental results show that our scheme achieves superior performance than the HEVC anchor especially at low bit rates, leading to on average 3.8%, 2.6%, and 3.5% BD-rate reduction on the HEVC common test sequences under Random-Access, Low-Delay B, and Low-Delay P configurations, respectively. When tested on high-definition and ultra high-definition sequences, the average BD-rate exceeds 5%.