Article Text

Download PDFPDF

1309 Deep learning reveals predictive immune signature of response to checkpoint blockade in multiplexed spatial immunohistochemistry data
  1. John-William Sidhom1,
  2. Jonathan Anker1,
  3. Guray Akturk2,
  4. Sudeh Izadmehr1,
  5. Justin David3,
  6. Saurabh Gupta3,
  7. Seunghee Kim-Schulze1,
  8. Padmanee Sharma4,
  9. Sacha Gnjatic1 and
  10. Matthew Galsky1
  1. 1Mount Sinai Hospital, NY, USA
  2. 2Merck, Demarest, NJ, USA
  3. 3Bristol Myers Squibb, Princeton, NJ, USA
  4. 4The University of Texas MD Anderson Cancer Center, Houston, TX, USA
  • Journal for ImmunoTherapy of Cancer (JITC) preprint. The copyright holder for this preprint are the authors/funders, who have granted JITC permission to display the preprint. All rights reserved. No reuse allowed without permission.


Background While there has been tremendous promise of immunotherapy in treating cancer, most patients do not respond to treatment.1 Biomarker development has grown as the field attempts to better select patients that may benefit from immunotherapy as well as to further understanding of effective use of immunotherapy in cancer.2–4 Here, we utilize a deep learning approach to leverage single-cell information from spatial immunohistochemistry data to query predictive immune signatures of response to immunotherapy.

Methods In this work, we adapt a previously described multiple instance learning (MIL) approach5–7 to analyze single cell tabular data in a supervised machine learning approach, allowing us to not only create predictive models of response but interrogate the specific correlates of response learned by the underlying machine learning model. We utilize this newly described MIL deep learning approach (figure 1a) to analyze single cell data obtained from multiplexed spatial immunohistochemistry data obtained from pre-treatment tumor samples in CheckMate 275, a phase 2 clinical trial of checkpoint inhibition in metastatic urothelial carcinoma, to predict response (via RECIST) in this cohort and reveal insights into an effective immune response at the single cell level and their spatial relationships.

Results Our model was most predictive of response (figure 1b, AUC = 0.81) when applied solely to cells in the extra-tumoral tissue (outside of the tumor bed). When looking at the predictive signature of response, we noted an association of the predictive signature in the extra-tumoral space to key immune markers including CD3/CD8, CD11b, CD68, DC-LAMP, and PDL1 (figure 2a,b), suggesting the importance of this immune signature’s presence in the extra-tumoral tissue as being predictive of response prior to the initiation of immunotherapy. Finally, we interrogated the spatial organization of these predictive cells. By quantifying the level of co-localization of predictive cells via Moran’s Index, we noted that the predictive signature was more co-localized within responders vs non-responder (figure 3a,b), and was an independent correlate of response (figure 3c), suggesting an effective immune response not only requires an immune infiltrated tumor but co-localization of these key immune cells in the extra-tumoral space.

Conclusions These findings highlight the utility of deep learning at the single cell level to identify predictive immune signatures of response and note that while the quantity of the immune infiltration is predictive of response, the spatial organization of this immune response is an independent correlate of response and a hallmark of clinical benefit.


  1. Topalian SL, Hodi FS, Brahmer JR, Gettinger SN, Smith DC, ... Sznol M. Safety, activity, and immune correlates of anti-PD-1 antibody in cancer. New England Journal of Medicine, 2012;366(26):2443–2454

  2. Snyder A, Makarov V, Merghoub T, Yuan J, Zaretsky JM, Desrichard A, ... Chan TA. Genetic basis for clinical response to CTLA-4 blockade in melanoma. New England Journal of Medicine, 2014;371(23):2189–2199

  3. Rizvi NA, Hellmann MD, Snyder A, Kvistborg P, Makarov V, Havel JJ, ... Chan TA. Mutational landscape determines sensitivity to PD-1 blockade in non-small cell lung cancer. Science, 2015;348(6230):124–128

  4. Van Allen EM, Miao D, Schilling B, Shukla SA, Blank C, Zimmer L, ... Garraway LA. Genomic correlates of response to CTLA-4 blockade in metastatic melanoma. Science, 2015;350(6257):207–211

  5. Sidhom JW, Larman HB, Pardoll DM, Baras AS. DeepTCR is a deep learning framework for revealing sequence concepts within T-cell repertoires. Nature communications, 2021;12(1):1605.

  6. Sidhom JW, Oliveira G, Ross-MacDonald P, Wind-Rotolo M, Wu CJ, Pardoll DM, Baras AS. Deep learning reveals predictive sequence concepts within immune repertoires to immunotherapy. Science Advances, 2022;8(37):eabq5089.

  7. Sidhom JW, Siddarthan IJ, Lai BS, Luo A, Hambley BC, Bynum J, ... Shenderov E. Deep learning for diagnosis of acute promyelocytic leukemia via recognition of genomically imprinted morphologic features. NPJ Precision Oncology, 2021;5(1):38.

Ethics Approval CheckMate 275 (NCT02387996) is a BMS-sponsored, multi-center, institutional-review-board-approved, phase 2 single arm clinical trial of nivolumab in patients with metastatic or unresectable urothelial cancer who have progressed or recurred following treatment with a platinum agent.

Abstract 1309 Figure 1

Multiple-instance deep learning model. (A) Architecture of multiple-instance deep learning model. Model takes as an input a feature vector for each cell or instance (i.e. gene expression, antibody staining, etc). Each cell’s set of features passes through multiple fullyconnected dense layers to reach a learned latent representation before being aggregated across all cells in a given sample (i.e. patient). This aggregate latent representation is then passed through a second fully-connected set of layers before being used to make a sample-level (patient-level) classification. (B) When fit in Monte-Carlo cross-validation, this model achieved a classification performance characterized by receiver operating characteristics (ROC) curve with an Area Under the Curve (AUC) of 0.81. This model was also able to stratify survival by (C) OS and (D) PFS.

Abstract 1309 Figure 2

Single-cell UMAP representations. (A) UMAP dimensionality reduction was applied to single cell features (antibody staining from multiplexed IHC) from the cells from the extra-tumoral space. Per-cell assignments of probability of response were derived from the previously fit predictive model (blue = responder signature, red = non-responder signature). (B) UMAP dimensionality reduction colored by antibody expression (red = high expression, blue = low expression). (C) UMAP dimensionality reduction shown for each patient where color of point corresponds to predictive signature from model (as in (A)). Above each plot is the patient identifier and the corresponding probability of response as determined by the model in parenthesis. Plot edge colors are denoted given the patient’s response status as determined by RECIST (blue = CRPR, red = SDPD).

Abstract 1309 Figure 3

Spatial characteristics of predictive immune signature. (A) Moran’s index was computed given the predicted probability of each cell (features) and spatial coordinates in each sample. A high Moran’s Index suggests a high co-localization of predictive cells within a sample while a low Moran’s Index suggests a low co-localization of predictive cells within a sample. A higher Moran’s Index in responders therefore suggests higher co-localization of predictive cells within responders than in non-responders. (B) Spatial visual representation of predictive cells is shown for all patients within cohort, sorted by Moran’s Index in decreasing order. Above each plot is the patient identifier and the corresponding Moran’s Index in parenthesis. Color of cell corresponds with probability of response (red = high probability of response, blue = low probability of response). Plot edge colors are denoted given the patient’s response status as determined by RECIST (blue = CRPR, red = SDPD). (C) Predictive Signature per sample vs Moran’s Index is plotted for each sample in the cohort. (D) When the predicted probability of response (Pred) and Moran’s Index (Moran) were used in multivariate logistic regression modeling, they remain as two independent predictors as evidenced by the fact that the 95% bootstrapped confidence intervals of the model coefficients do not cross 0 (95% CI: Pred = [0.420, 1.472], Moran = [0.029, 0.721])

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.