Novel Natural Language Processing and Machine Learning Methods to Characterize Unstructured Patient-Reported Outcomes

BACKGROUND: We aimed to evaluate validity of natural language processing (NLP) and machine learning (ML) algorithms in identifying features of unstructured patient-reported outcomes (PROs) for child/adolescent cancer survivors vs. expert judgment as the gold standard.  METHODS: A cross-section study to collect unstructured PROs from 52 survivors aged 8-17.9 years and 35 caregivers who visited a Comprehensive Cancer Center in 2016. PROs including pain interference and fatigue symptoms were reported through in-depth interviews. After verbatim transcription, analyzable sentences (i.e., meaning units) were semantically labeled by 2 content experts based on problems with physical, cognitive, or social attributes. Two NLP/ML methods were used to extract and validate the semantic features: 1) Bidirectional Encoder Representations from Transformers (BERT) and 2) Word2vec plus one of the ML methods, the Support Vector Machine (SVM) and Extreme Gradient Boosting (XGBoost). Receiver operating characteristic (ROC) and precision-recall (PR) curves evaluated the accuracy and validity of NLP/ML methods, respectively. FINDINGS: In total, 391 meaning units in pain interference and 423 in fatigue domains were labeled and analyzed. Compared to Word2vec/SVM and Word2vec/XGBoost, BERT demonstrated higher accuracy on both symptom domains, including 0.931 (95%CI=0.905, 0.957) and 0.916 (95% CI=0.887, 0.941) for problems with cognitive and social attributes on pain interference, and 0.929 (95% CI=0.903, 0.953) and 0.917 (95% CI=0.891, 0.943) for problems with cognitive and social attributes on fatigue. In addition, BERT yielded superior areas under the ROC curve for cognitive attribute on pain interference and fatigue domains (0.923 [95% CI=0.879, 0.997]; 0.948 [95% CI=0.922, 0.979]), and superior areas under the PR curve for cognitive attribute on pain interference and fatigue domains (0.818 [95% CI=0.735, 0.917]; 0.855 [95% CI=0.791, 0.930). INTERPRETATIONS: As an alternative to standard PRO surveys, collecting unstructured PROs via interviews or conversations during clinical encounters and applying NLP/ML methods can facilitate PRO assessment in cancer survivors. FUNDING STATEMENT: The research reported in this manuscript was supported by the National Cancer Institute under award numbers U01CA195547 and P30CA021765-33, and the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under award number U19AR069525. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. DECLARATION OF INTERESTS: All co-authors declare no conflict of interest. ETHICS APPROVAL STATEMENT: The research protocol was approved by St. Jude’s Institutional Review Board. Assent from survivors and consent from caregivers were obtained.