Interpretation Accuracy and Interrater Agreement of Pediatric Skin and Soft Tissue Point-of-Care Ultrasound Images among Residents and Faculty

Abstract Background Point-of-care ultrasound (POCUS) is commonly used to facilitate care in the emergency department. Acquired images are often reviewed by local experts for educational and quality assurance purposes. However, no published study has examined the accuracy and reliability of POCUS image interpretation by multiple reviewers. Objectives We studied the accuracy and interrater agreement among expert and trainee reviewers of prerecorded pediatric skin and soft tissue (SST) POCUS images. Methods POCUS faculty and emergency medicine (EM) residents blindly reviewed deidentified pediatric SST POCUS images and indicated whether a drainable fluid collection was present, absent, or indeterminate. This was then compared with the gold standard based on discharge diagnoses and telephone follow-up. Images rated as indeterminate were excluded from the initial analysis. Sensitivity analysis assuming indeterminate answers were inaccurate was subsequently conducted. Results In phase 1, 6 pediatric EM POCUS directors reviewed 168 images. The overall accuracy was 79.7% (range 66.1–86.0%). The mean Cohen's kappa was 0.58 (range 0.24–0.84). Sensitivity analysis yielded an overall accuracy of 71.3% (range 56.5–76.9%) and a Cohen's kappa of 0.43 (range 0.20–0.59). In phase 2, 6 general EM POCUS faculty and 20 EM residents reviewed 120 images. The overall accuracy among residents was 72.2% (range 51.4–84.7%) and among faculty was 83.6% (range 77.9–88.8%). Sensitivity analysis yielded an overall resident accuracy of 63.0% (range 49.5–80.7) and an overall faculty accuracy of 73.9% (range 67.0–79.8%). Fleiss' kappa was 0.322 for residents and 0.461 for faculty. Conclusions We found moderate accuracy and fair to good interrater agreement among POCUS faculty and EM residents reviewing pediatric SST POCUS images.