Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared

Frame interpolation is the process of synthesizing a new frame in between the existing frames in an image sequence. It has emerged as a key algorithmic module in motion picture effects since its large scale use in the making of the movie “The Matrix.” This paper presents a review and a new unified view of the classical algorithms used to create in-between frames, representing most of the past 20 years of their evolution. This is used to benchmark the recent deep learning algorithms against two of the best industrial retimers available. A significantly expanded data set of 140,000 frames is used for testing. In the context of highresolution material, we find that techniques relying principally on deep neural networks (DNNs) do not clearly outperform the classical ideas. It is only with the emergence of hybrid approaches in 2019 that we see DNNs adding significantly to the performance in this space. Despite the hype surrounding DNNs, we find that there is still more work to be done .