Transcribing Content From Structural Images With Spotlight Mechanism

Authors:
Yu Yin University of Science and Technology of China
Zhenya Huang University of Science and Technology of China
Enhong Chen University of Science and Technology of China
Qi Liu University of Science and Technology of China
Fuzheng Zhang Microsoft
Xing Xie Microsoft
Guoping Hu iFLYTEK Research

Introduction:

This paper studies the problem of Transcribing content from structural images. The authors propose a hierarchical Spotlight Transcribing Network (STN) framework followed by a two-stage “where-to-what’’ solution.The authors also design a reinforcement method to refine the paper's STN framework by self-improving the spotlight mechanism.

Abstract:

Transcribing content from structural images, e.g., writing notes from music scores, is a challenging task as not only the content objects should be recognized, but the internal structure should also be preserved. Existing image recognition methods mainly work on images with simple content (e.g., text lines with characters), but are not capable to identify ones with more complex content (e.g., structured code), which often follow a fine-grained grammar. To this end, in this paper, we propose a hierarchical Spotlight Transcribing Network (STN) framework followed by a two-stage “where-to-what’’ solution. Specifically, we first decide “where-to-look’’ through a novel spotlight mechanism to focus on different areas of the original image following its structure. Then, we decide “what-to-write’’ by developing a GRU based network with the spotlight areas for transcribing the content accordingly. Moreover, we propose two implementations on the basis of STN, i.e., STNM and STNR, where the spotlight movement follows the Markov property and Recurrent modeling, respectively. We also design a reinforcement method to refine our STN framework by self-improving the spotlight mechanism. We conduct extensive experiments on many structural image datasets, where the results clearly demonstrate the effectiveness of STN framework.

You may want to know: