Segmentation of transitional cell carcinoma nuclei by nonsupervised thresholding in different color spaces.

OBJECTIVE: To develop a method for nonsupervised thresholding of transitional cell carcinoma nuclei of hematoxylin-eosin-stained histologic sections. STUDY DESIGN: For grayscale, RGB, HSL and L*a*b* thresholding we used an extension of a clustering method, based on a between-class/within-class criterion, applying optimal gray-level thresholding to distributions of R, G, B, or H and S or L* color domains. Algorithms were tested on 20 hematoxylin-eosin-stained sections of bladder carcinomas. RESULTS: Results were compared with corresponding results of manually selected nuclear areas. Images were compared pixel to pixel with matching reference images. Grayscale automatic thresholding presented unacceptably low pixel specificity, which complicated further nuclear segmentation. Nonsupervised thresholding in RGB or HSL, as well as semimanual thresholding in L*a*b* color space demonstrated significantly better accuracy and high values of pixel specificity and sensitivity, which permitted errors of only 4.27-5.83% in the subsequent mean area estimation of the transitional cell carcinoma nuclei. CONCLUSION: Nonsupervised multispectral thresholding in RGB or HSL color space extends single graylevel thresholding techniques to multilevel thresholding. This seems to be an effective, relatively simple and fast alternative to the widely used automatic grayscale or manual color thresholding for segmentation of nuclei in routine histologic sections.