The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans

Rationale and Objectives The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the “truth” against which nodule-based studies are measured. Materials and Methods Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial “blinded read” phase, radiologists independently marked lesions they identified as “nodule ≥3 mm (diameter),” “nodule Results After the initial blinded read phase, 71 lesions received “nodule ≥3 mm” marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as “nodule ≥3 mm” by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist. Conclusion The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules ≥3 mm. Nevertheless, substantial variabilty remains across radiologists in the task of lung nodule identification.