Article Text
PDF
Abstract
Background We previously utilized real-world Electronic Health Records (EHR) data from over 2,200 patients treated with immune checkpoint inhibitors (ICIs) to predict hepatitis, colitis, and pneumonitis, as well as 1-year overall survival.1 These predictive models employed a 1-year prediction time window and aggregation window of 60, 120, or 365 days, depending on type of data aggregated to generate predictive features. While our models are based on routinely collected data, the lengths of the aggregation windows may impede clinical deployment. For instance, predictions for newly diagnosed patients with less than 365 days of EHR data may be less reliable. Additionally, large aggregation windows increase the data processing burden, hindering real-time clinical implementation. In this study, we investigate the feasibility of reducing the aggregation window size while maintaining comparable predictive performance to the original models.
Methods We conducted a systematic analysis to evaluate the impact of progressively narrowing the aggregation window on the predictive performance (figure 1). We did this for all endpoints, except hepatitis, since it already uses a short 60-day aggregation window. We retrained the models using the same features and hyperparameters utilized in the original models.1 We then calculated performance metrics, including area under the receiver operator curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and their 1000-fold bootstrapped 95% confidence intervals. These metrics were plotted to evaluate trends as the window narrowed.
Results The original overall survival model had an AUC of 0.75. Our experiments with narrowing windows show that the AUC does not decrease, indicating a 100% retention of original performance in the worst-case scenario.
Regarding toxicities, the original AUC of Pneumonitis was 0.74. With reduced aggregation windows, the AUC ranges from 0.72 to 0.74, maintaining 97% of the original performance. A similar trend has been observed for colitis with original AUC of 0.76. After aggregation window reduction, its AUC ranged from 0.76 to 0.77, keeping 100% of its original performance (figure 2).
Conclusions We performed a systematic analysis of the impact of narrowing aggregation windows on model performance, demonstrating that all evaluated endpoints are reasonably robust and can maintain a comparable predictive performance using narrow aggregation windows. Our findings demonstrate that our models are reliable for newly diagnosed patients with less EHR data for model input and that narrower aggregation windows can be used to reduce data processing burden, making these models more accessible and scalable.
Reference
Lippenszky L, Mittendorf KF, Kiss Z, et al. Prediction of effectiveness and toxicities of immune checkpoint inhibitors using real-world patient data. JCO Clin Cancer Inform. 2024;8:e2300207.
Experimental setup for pneumonitis, colitis and overall survival
Results
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See http://creativecommons.org/licenses/by-nc/4.0/.