Multicenter Colonoscopy Quality Measurement Utilizing Natural Language Processing

Abstract

Introduction: An accurate system for tracking of both colonoscopy quality and surveillance intervals could improve the effectiveness and cost-effectiveness of colorectal cancer (CRC) screening and surveillance. The purpose of this study was to create and test such a system that uses natural language processing (NLP). Methods: From 42,569 colonoscopies with pathology records from 13 centers, we randomly sampled/identified 750 paired reports. We trained (n=250) and tested (n=500) an NLP-based program on 19 measurements that encompass colonoscopy quality measures and surveillance interval determination. Blinded, paired, annotated expert manual review was used as the reference standard. The remaining 41,819 non-annotated documents were processed through the NLP system without manual review to assess performance consistency. The primary outcome was system accuracy across the 19 measures. Results: The overall error rate was 3.5% in the NLP system vs. 1.9% for the paired annotators (p=0.041). When 8 vaguely documented reports were removed and annotator errors were corrected to correspond to established pathologic definitions, 25.4% were incorrect by NLP and 21.1% by the initial annotator (p=0.07). Rates of pathologic findings calculated based on the NLP system were similar to those calculated by the gold standard annotation for the majority of the measurements (see Table for partial listing). For the testing set, accuracy of CRC detection was 99.6%, advanced adenoma 95.0%, non-advanced adenoma 94.6%, advanced sessile serrated polyp 99.8%, non-advanced sessile serrated polyp 99.2%, ≥10 mm hyperplastic polyp 96.8%, and <10 mm hyperplastic polyp 96.0%. The lesion location showed high accuracy (87.0-99.8%). The number of adenomas had an accuracy of 90.2%. Adenoma detection rate across all documents and centers based on NLP was 29.1% (range 19.3-38.0%). Conclusion: NLP can accurately report adenoma detection rate and the components for determining guideline-adherent colonoscopy surveillance intervals across a wide variety of sites that utilize different methods for reporting colonoscopy findings. Disclosure - Dr. Imler - patent holder: TRAQ-ME. Dr. Morea - patent holder: TRAQ-ME.Table 1: Results Across Non-annotated Data Set Using NLP From Training and Testing Sets (Partial Listing)