Article Text

Download PDFPDF

831 Exact Shapley values for explaining complex machine learning based molecular tests of checkpoint inhibitors: potential utility for patients, physicians, and translational research
Free
  1. Heinrich Roder1,
  2. Lelia Net1,
  3. Joanna Roder2,
  4. Thomas Campbell1,
  5. Mark McCleland3,
  6. Wei Zou4,
  7. Minu Srivastava4,
  8. David Shames4,
  9. Laura Maguire1 and
  10. Robert Georgantas1
  1. 1Biodesix, Boulder, CO, USA
  2. 2Biodesix, Inc., Boulder, CO, USA
  3. 3Former employee of Genentech, South San Francisco, CA, USA
  4. 4Genentech, South San Francisco, CA, USA

Abstract

Background Modern machine learning (ML) models based on highly multivariate attribute sets (e.g. unbiased -omics data) can be very successful at generating clinically useful predictions, but at the price of less transparency in how individual attributes are used to make those predictions. In short, ML test algorithms tend to be ”black boxes”. Shapley values (SVs)1 describe the relative importance of the attributes used within a multivariate test to the generation of the test result for an individual patient.2 While typically the calculation of SVs is computationally prohibitive, our ML architecture permits the generation of SVs for large patient cohorts. In this study, we evaluate SVs for the Anti-PD-L1 Response Test (ART), that was shown in independent validation to predict outcomes for patients treated with atezolizumab,3 for the POPLAR Ph2 and OAK Ph3 studies of non-small cell lung cancer (NSCLC) patients .4 5

Abstract 831 Figure 1

Radar plots illustrating the values of the 10 most important SVs for test classification generation for two samples classified as poor (A and B) and two samples classified as good (C and D). Positive SVs are shown in red and negative SVs are shown in blue

Abstract 831 Figure 2

Heatmaps of the SVs for samples classified as poor showing two subgroups (top and bottom, separated by horizontal line) with different patterns of SVs for (A) POPLAR and (B) OAK

Abstract 831 Figure 3

Heatmaps of the SVs for samples classified as good showing three subgroups, top, middle, and bottom, separated by horizontal lines) with different patterns of SVs for (A) POPLAR and (B) OAK

Methods ART results, Good or Poor had been produced for 262 patients in POPLAR (NCT01903993) and 786 patients in OAK (NCT02008227). Exact SVs were generated for each pretreatment serum sample for each of the 93 attributes (proteomic features) used in the test. The distribution of SVs across the cohort was investigated to assess the relative importance of each feature to test classification. Subgroups of patients with similar patterns of SVs were identified using t-sne plots and ML methods in the POPLAR cohort and validated in the OAK cohort.

Results The SV distributions showed that the features influencing ART classification most were similar in both POPLAR and OAK. The relative importance of features to test classification differed between patients (figure 1), but subgroups of patients within test classification groups showed similar patterns of SVs (figures 2 and 3). Such patient subgroups, identified within POPLAR, were also found in the OAK cohort and were associated with differences in outcome and/or differences in patient characteristics.

Conclusions SVs can explain how complex ML-based tests combine molecular attributes to produce individual patient results. Exact SVs can be obtained for certain ML architectures used in molecular test development, revealing the overall relative importance of attributes used in such molecular tests. Subgrouping of patients with the same test classification by different patterns of SVs is possible. This may reveal different biologies contributing to a Good or Poor phenotype and inform translational studies.

ClinicalTrials gov NCT01903993 and NCT02008227

References

  1. Shapley L. A value for n-person games. Contributions to the Theory of Games. 1953;2.28:307–317.

  2. Roder J, Maguire L, Georgantas R, Roder H. Explaining multivariate molecular diagnostic tests via Shapley values. BMC Med Inform Decis Mak 2021;21(1):211.

  3. Kowanetz M, Leng N, Roder J, et al. Evaluation of immune-related markers in the circulating proteomic and their association with atezolizumab efficacy in patients with 2L+ NSCLC. J Immunother Cancer 2018;6(Suppl1):114.

  4. Fehrenbacher L, Spira A, Ballinger M, et al. Atezolizumab versus docetaxel for patients with previously treated non-small-cell lung cancer (POPLAR): a multicentre, open-label, phase 2 randomised controlled trial. Lancet 2016;387(10030):1837–1846.

  5. Rittmeyer A, Barlesi F, Waterkamp D, et al. Atezolizumab versus docetaxel in patients with previously treated non-small-cell lung cancer (OAK): a phase 3, open-label, multicenter randomized controlled trial. Lancet 2017;389(10066):255–265.

Ethics Approval The OAK study that was done in 194 academic medical centers and community oncology practices across 31 countries worldwide. The study was done in full accordance with the guidelines for Good Clinical Practice and the Declaration of Helsinki. All patients gave written informed consent.The POPLAR trial was done at 61 academic medical centers and community oncology practices across 13 countries in Europe and North America. The study was done in full accordance with the guidelines for Good Clinical Practice and the Declaration of Helsinki. Protocol (and modification) approval was obtained from anindependent ethics committee for each site. Patients gave written informed consent.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.