Article Text
PDF
Abstract
Background Despite the clinical success of immune checkpoint blockade therapies, many patients do not respond to treatments or become resistant. Previous attempts to predict treatment efficacy suffered from limited accuracy and deficiency to uncover determinants of response. Here, we introduce an AI framework which addresses these issues.
Methods We present a new explainable deep learning framework based on transformer architecture1, 2 which combines data with different feature sets (including sparse date) and clinical endpoints for survival prediction or classification. This framework includes: (1) A new loss function based on a sigmoid approximation of Harrell’s concordance-index.3 (2) Explainability module providing feature importance and similarity score between features based on mutual contribution to predictions. (3) Transfer learning strategy to enable leveraging diverse clinical datasets in the public or private domain.
Results We utilized seven data sets comprising of more than 140,000 patients from IO, targeted and Chemotherapy treatments to benchmark our prediction models (table 1) in addition to 10 train/test splits performance evaluations.
Consistently, our framework outperformed other methods previously described in the literature, including CoxPH 4 and random survival forest.5, 6 For example, using the concordance index, our framework achieved 0.66 (0.04) vs. 0.60 (0.04) of the second-best method (Random survival forest in all cases) on MYSTIC IO arms clinical data. This improvement was a result of including transfer learning in the training process (table 2) which also achieved better performance in less training steps (figure 1).
Utilizing our Explainability module, we identified key features driving response prediction consistent with previous publications. For example, in Chowell et al. dataset, we identified Albumin, NLR, Chemo-before-IO-treatment and TMB as the most important features (figure 2). We also identified in sparse mutation calls of 469 genes from Samstein et al. dataset, functional modules of several genes only, each with a strong predictive power. For example, the functional module comprising of the genes: AKT2, BTK, CDC73, HLA-B, IKBKE, INPPL1, RFWD2, TRAF2 and WHSC1, related to adaptive immunity, stratified patients to two groups with a Hazard Ratio of 0.58 for the Samstein dataset and 0.42 for the validation dataset (figure 3).
Conclusions We propose a new framework with state-of-the-art performance in survival prediction and potential to uncover biological and clinical insights related to patient response and resistance. Importantly, our framework simplifies the process of translating complex AI models to clinical practice and may accelerate the benefit immunotherapy can bring to patients.
References
Vaswani Ashish, et al. Attention is all you need. Advances in Neural Information Processing Systems. 2017;30.
Devlin Jacob, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
Schmid Matthias, Marvin N Wright, and Andreas Ziegler. On the use of Harrell’s C for clinical risk prediction via random survival forests. Expert Systems with Applications 2016;63:450–459.
Cox, David R. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological). 1972;34(2):187–202.
Ishwaran Hemant, et al. Random survival forests. The Annals of Applied Statistics. 2008;2(3):841-860.
Chowell Diego, et al. Improved prediction of immune checkpoint blockade efficacy across multiple cancer types. Nature Biotechnology. 2022;40(4):499–506.
Rizvi Naiyer A, et al. Durvalumab with or without tremelimumab vs standard chemotherapy in first-line treatment of metastatic non–small cell lung cancer: the MYSTIC phase 3 randomized clinical trial. JAMA oncology. 2020;6(5):661–674.
Rittmeyer Achim, et al. Atezolizumab versus docetaxel in patients with previously treated non-small-cell lung cancer (OAK): a phase 3, open-label, multicentre randomised controlled trial. The Lancet. 2017;389(10066):255–265.
Samstein Robert M, et al. Tumor mutational load predicts survival after immunotherapy across multiple cancer types. Nature Genetics. 2019;51(2):202–206.
Thorsson Vésteinn, et al. The immune landscape of cancer. Immunity. 2018;48(4):812–830.
AACR Project Genie Consortium, et al. AACR Project GENIE: powering precision medicine through an international consortium. Cancer Discovery. 2017;7(8):818–831.
Miao Diana, et al. Genomic correlates of response to immune checkpoint blockade in microsatellite-stable solid tumors. Nature Genetics. 2018;50(9):1271–1281.
Description of the datasets used to train and evaluate the clinical transformer
Clinical transformer improves patient survival prediction across multiple datasets. Model performance is evaluated using the concordance index across 10 random training/testing splits for consistency. ++Transfer learning from GENIE dataset and applied to Samstein et al. +Transfer learning from Chowell et al., dataset and applied to Mystic trial. High Collinearity: Multivariable CoxPH model did not run because of high collinearity among the input features.
Transfer learning positively impact patient response to IO in a pan-cancer and cancer specific settings. A) Prediction of patient response to immunotherapy in a pan-cancer dataset (TMB and Immunotherapy Samstein et., al). Pre-training modeling is performed using self-supervision over the GENIE dataset. Then, to predict patient response to immunotherapy, weights from GENIE pretrained model are transferred to a survival prediction model. Each learning curve describes how the model learnt across different epochs and each line indicates the baseline and transfer learned model. B) Transfer learning impact for predicting response to Immunotherapy in Non-Small Cell Lung Cancer (NSCLC). Learning curves describe the impact of different pre-training iterations over the survival outcome prediction.
Model feature importance across 10 random training/testing splits shows consistency on relevant features associated to response to IO. Feature importance is computed by perturbing each feature and computing its effect on the model performance across 10 training/testing splits.
Top 16 functional groups associated with response/resistance to treatment discovered using the Samstein et al., dataset with 10 training/testing splits. The forest plot shows the effect on patient survival of mutated/non-mutated functional groups. Pathway analysis indicates that the C8 functional group is associated with adaptive immune response (p-value=7.47e-9) and it has a positive effect on the prediction of response to IO in the training/testing dataset (Left Kaplan Meier plot) and in an independent validation dataset (Miao et., al, Middle Kaplan Meier plot). Interestingly, this pattern is not observed in TCGA pan cancer dataset using the same cancer types (Right Kaplan Meier Plot).