Stacked sparse autoencoder and case-based postprocessing method for nucleus detection

Abstract Accurate nucleus detection is of great importance in pathological image analyses and diagnoses, which is a critical prerequisite for tasks such as automated grading hepatocellular carcinoma (HCC) nuclei. This paper proposes an automated nucleus detection framework based on a stacked sparse autoencoder (SSAE) and a case-based postprocessing method (CPM) in a coarse-to-fine manner. SSAE, an unsupervised learning model, is first trained using image patches of breast cancer. Then, the transfer learning and sliding window techniques are applied to other cancers’ pathological images (HCC and colon cancer) to extract the high-level features of image patches via the trained SSAE. Subsequently, these high-level features are fed to a logistic regression classifier (LRC) to classify whether each image patch contains a complete nucleus in a coarse detection process. Finally, CPM is developed for refining the coarse detection results which removes false positive nuclei and locates adhesive or overlapped nuclei effectively. SSAE-CPM achieves an average nucleus detection accuracy of 0.8748 on HCC pathological images, which can accurately locate almost all nuclei on the pathological images with serious differentiation. In addition, our proposed detection framework is also evaluated on a public dataset of colon cancer, with a mean F1 score of 0.8355. Experimental results demonstrate the performance advantages of our proposed SSAE-CPM detection framework as compared with related work. While our detection framework is trained on the pathological images of breast cancer, it can be easily and effectively applied to nucleus detection tasks on other cancers without re-training.