Article Text

Download PDFPDF

1311 H&E 2.0: deep learning-enabled identification of tumor-specific CD39+CD8+ T cells in marker-free images for predicting immunotherapy response
  1. Felicia Wee1,
  2. Willa Yim1,
  3. Jia Meng1,
  4. Jeffrey Lim1,
  5. Craig Joseph1,
  6. Xinru Lim1,
  7. Kai Soon Ng1,
  8. Jiang Feng Ye1,
  9. Zhen Wei Neo1,
  10. Li Yen Chong1,
  11. Chan Way Ng1,
  12. Kiat Hon Lim2,
  13. Mai Chan Lau1 and
  14. Joe Yeong1
  1. 1Agency for Science, Technology and Research (A*STAR), Singapore 138673, Singapore
  2. 2Singapore General Hospital, Singapore, Singapore
  • Journal for ImmunoTherapy of Cancer (JITC) preprint. The copyright holder for this preprint are the authors/funders, who have granted JITC permission to display the preprint. All rights reserved. No reuse allowed without permission.


Background Several groups, including ours, have shown CD39 to be a tumor-specific CD8+ T cell marker. In non-small cell lung cancer (NSCLC) and colorectal carcinoma (CRC), CD8+ T cells lacking CD39 expression are bystander tumor infiltrating lymphocytes1; while CD39+CD8+ T cells are tumor antigen-specific in treatment-naïve NSCLC2 and triple-negative breast cancer (TNBC).3 Thus, combining CD39+CD8+ T cell abundance and spatial localization is a potential predictor of patient response to PD-1/PD-L1 blockade immunotherapy for numerous cancer types.3–6

To redirect resources from repeatedly conducting laborious and costly multi-marker assays for immunotherapy patient stratification, we developed deep learning (DL) models trained on multiplex immunofluorescence (mIF) and fluorescence imaging data to identify CD39+CD8+ T cells by morphology in hematoxylin and eosin (H&E)-stained tissue images and brightfield images of immune cells from blood samples.

Methods Separate convolutional neural network models were developed to identify CD39+CD8+ T cells in human CRC samples and peripheral blood mononuclear cells (PBMCs) from CT26 tumor-bearing mice (CRC mouse tumor models).

CD39+CD8+ T cells in the CRC samples were first visualized with mIF and subsequently stained with H&E. The DL pipeline stages are: (1) alignment of fluorescence and H&E images, (2) cell segmentation, (3) manual annotation of CD39+CD8+ cells as ground truth labels, (4) extracting each cell as a small image patch, and (5) training a DL model (θH&E) for CD39+CD8+ prediction using 2,426 positive examples and 101,084 negative examples (figure 1A).

The mouse PBMCs were immunostained with fluorescent antibodies and visualized with imaging flow cytometry. The DL pipeline stages are: (1) gating CD8+ and CD39+ positivity based on fluorescence intensity, and (2) training a DL model (θblood) for CD39+CD8+ prediction using 1,985 positive examples and 4,639 negative examples (figure 1B).

The models’ performance was evaluated with F1 scores.

Results The current version of θH&E has a test F1-score of 0.83; θblood has a test F1-score of 0.80.

Conclusions The F1-scores indicate that both DL models can identify CD39+CD8+ T cells from marker-free H&E images and brightfield images, respectively. Ongoing improvements to the models include validating them across independent cohorts with different cancer types and evaluating their predictive capabilities for checkpoint immunotherapy response on pre-treatment patient samples. By implementing cell identification by virtual staining of H&E images (‘H&E 2.0’) and brightfield images of blood samples, high throughput screening of patient samples can be done. If required, downstream stains like immunohistochemistry and/or flow cytometry can be conducted for confirmation (figure 1C).

Acknowledgements This work was supported by the Bioinformatics Institute (BII), Singapore Immunology Network (SIgN), Institute of Molecular and Cell Biology (IMCB), and Agency for Science, Technology and Research (A*STAR).


  1. Simoni Y, Becht E, Fehlings M, et al. Bystander CD8(+) T cells are abundant and phenotypically distinct in human tumour infiltrates. Nature. 2018;557(7706):575–579

  2. Yeong J, Suteja L, Simoni Y, et al. Intratumoral CD39(+)CD8(+) T Cells Predict Response to Programmed Cell Death Protein-1 or Programmed Death Ligand-1 Blockade in Patients With NSCLC. J Thorac Oncol. 2021;16(8):1349–1358

  3. Jia M, Tira T, Jiangfeng Y, et al. 1039 The prognostic value of tumour-specific T cells in Asian TNBC: using CD39(+)CD8(+) T cells as a surrogate marker. J Immunother Cancer. 2022;10(Suppl 2):A1081

  4. Laumont CM, Wouters MCA, Smazynski J, et al. Single-cell Profiles and Prognostic Impact of Tumor-Infiltrating Lymphocytes Coexpressing CD39, CD103, and PD-1 in Ovarian Cancer. Clin Cancer Res. 2021;27(14):4089–4100

  5. Lee YJ, Kim JY, Jeon SH, et al. CD39(+) tissue-resident memory CD8(+) T cells with a clonal overlap across compartments mediate antitumor immunity in breast cancer. Sci Immunol. 2022;7(74):eabn8390

  6. Attrill GH, Owen CN, Ahmed T, et al. Higher proportions of CD39+ tumor-resident cytotoxic T cells predict recurrence-free survival in patients with stage III melanoma treated with adjuvant immunotherapy. J Immunother Cancer. 2022;10(6)

Ethics Approval This study was approved by the Agency of Science, Technology and Research (A*STAR) Human Biomedical Research Office (A*STAR IRB: 2021–161, 2021–188, 2021–112).

Consent De-identified patient data was used in our work. Samples were collected with consent from patients.

Abstract 1311 Figure 1

Two deep learning models for identifying tumor-specific CD39+CD8+ T cells (double-positive cells) in H&E images and single blood cell images. (A) Colorectal carcinoma (CRC) sample sections were visualized for CD8 and CD39 expression by multiplex immunofluorescence and subsequently for morphology by H&E. Individual cells were obtained from H&E images and marked for CD39-CD8 positivity based on immunofluorescence results. These ground truth images were then used to train the DL model. The current version has an F1 score of 0.83, indicating an ability to distinguish double-positive cells from those that are not, which are shown with representative images. (B) Peripheral blood mononuclear cells (PBMCs) isolated from a CRC mouse model were immunostained and visualized with imaging flow cytometry. Double-positive cells were identified based on fluorescence intensity of CD8 and CD39. The model is trained on these ground truth images and reached an F1 score of 0.80, indicating an ability to distinguish double-positive cells from those that are not, which are shown with representative images. (C) DL models that can reliably predict CD39+CD8+ cells can be used to screen large numbers of patient samples before expensive and time-consuming confirmatory analyses like immunohistochemistry and imaging flow cytometry. This will alleviate some pressure on medical resources in the immunotherapy era

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.