Manifold Learning Based Super Resolution for Mixed-Resolution Multi-view Video in Visual Internet of Things

In a Visual Internet of Things (VIoT), the video sequences of different viewpoints are captured by different visual sensors and transmitted simultaneously, which puts a huge burden on storage and bandwidth resources. Mixed-resolution multi-view video format can alleviate the burden on the limited storage and bandwidth resources. However, the low resolution view need to be up-sampled to provide high quality visual experiences to the users. Therefore, a super resolution (SR) algorithm to reconstruct the low resolution view is highly desirable. In this paper, we propose a new two-stage super resolution method. In the first depth-assisted high frequency synthesis stage, depth image based rendering (DIBR) is used to project a high resolution view to a low resolution view to estimate the super resolution result. Then in the second high frequency compensation stage, the local block matching model based on manifold learning is used to enhance the super resolution result. The experimental results demonstrate that our method is capable to achieving a PSNR gain up to 4.76 dB over bicubic baseline and recover details in edge regions, without sacrificing the quality of smooth areas.