How to extract more information with less burden: Fundus image classification and retinal disease localization with ophthalmologist intervention

Image classification using convolutional neural networks (CNNs) outperforms other state-of-the-art methods. Moreover, attention can be visualized as a heatmap to improve the explainability of results of a CNN. We designed a framework that can generate heatmaps reflecting lesion regions precisely. We generated initial heatmaps by using a gradient-based classification activation map (Grad-CAM). We assume that these Grad-CAM heatmaps correctly reveal the lesion regions; then we apply the attention mining technique to these heatmaps to obtain integrated heatmaps. Moreover, we assume that these Grad-CAM heatmaps incorrectly reveal the lesion regions and design a dissimilarity loss to increase their discrepancy with the Grad-CAM heatmaps. In this study, we found that having professional ophthalmologists select 30% of the heatmaps covering the lesion regions led to better results, because this step integrates (prior) clinical knowledge into the system. Furthermore, we design a knowledge preservation loss that minimizes the discrepancy between heatmaps generated from the updated CNN model and the selected heatmaps. Experiments using fundus images revealed that our method improved classification accuracy and generated attention regions closer to the ground truth lesion regions in comparison with existing methods.