Background Modern machine learning (ML) models based on highly multivariate attribute sets (e.g. unbiased -omics data) can be very successful at generating clinically useful predictions, but at the price of less transparency in how individual attributes are used to make those predictions. In short, ML test algorithms tend to be ”black boxes”. Shapley values (SVs)1 describe the relative importance of the attributes used within a multivariate test to the generation of the test result for an individual patient.2 While typically the calculation of SVs is computationally prohibitive, our ML architecture permits the generation of SVs for large patient cohorts. In this study, we evaluate SVs for the Anti-PD-L1 Response Test (ART), that was shown in independent validation to predict outcomes for patients treated with atezolizumab,3 for the POPLAR Ph2 and OAK Ph3 studies of non-small cell lung cancer (NSCLC) patients .4 5
Methods ART results, Good or Poor had been produced for 262 patients in POPLAR (NCT01903993) and 786 patients in OAK (NCT02008227). Exact SVs were generated for each pretreatment serum sample for each of the 93 attributes (proteomic features) used in the test. The distribution of SVs across the cohort was investigated to assess the relative importance of each feature to test classification. Subgroups of patients with similar patterns of SVs were identified using t-sne plots and ML methods in the POPLAR cohort and validated in the OAK cohort.
Results The SV distributions showed that the features influencing ART classification most were similar in both POPLAR and OAK. The relative importance of features to test classification differed between patients (figure 1), but subgroups of patients within test classification groups showed similar patterns of SVs (figures 2 and 3). Such patient subgroups, identified within POPLAR, were also found in the OAK cohort and were associated with differences in outcome and/or differences in patient characteristics.
Conclusions SVs can explain how complex ML-based tests combine molecular attributes to produce individual patient results. Exact SVs can be obtained for certain ML architectures used in molecular test development, revealing the overall relative importance of attributes used in such molecular tests. Subgrouping of patients with the same test classification by different patterns of SVs is possible. This may reveal different biologies contributing to a Good or Poor phenotype and inform translational studies.
Shapley L. A value for n-person games. Contributions to the Theory of Games. 1953;2.28:307–317.
Roder J, Maguire L, Georgantas R, Roder H. Explaining multivariate molecular diagnostic tests via Shapley values. BMC Med Inform Decis Mak 2021;21(1):211.
Kowanetz M, Leng N, Roder J, et al. Evaluation of immune-related markers in the circulating proteomic and their association with atezolizumab efficacy in patients with 2L+ NSCLC. J Immunother Cancer 2018;6(Suppl1):114.
Fehrenbacher L, Spira A, Ballinger M, et al. Atezolizumab versus docetaxel for patients with previously treated non-small-cell lung cancer (POPLAR): a multicentre, open-label, phase 2 randomised controlled trial. Lancet 2016;387(10030):1837–1846.
Rittmeyer A, Barlesi F, Waterkamp D, et al. Atezolizumab versus docetaxel in patients with previously treated non-small-cell lung cancer (OAK): a phase 3, open-label, multicenter randomized controlled trial. Lancet 2017;389(10066):255–265.
Ethics Approval The OAK study that was done in 194 academic medical centers and community oncology practices across 31 countries worldwide. The study was done in full accordance with the guidelines for Good Clinical Practice and the Declaration of Helsinki. All patients gave written informed consent.The POPLAR trial was done at 61 academic medical centers and community oncology practices across 13 countries in Europe and North America. The study was done in full accordance with the guidelines for Good Clinical Practice and the Declaration of Helsinki. Protocol (and modification) approval was obtained from anindependent ethics committee for each site. Patients gave written informed consent.