Article Text

Download PDFPDF

1282 H&E 2.0 virtual staining of CD3+ T cells: improving cell labeling for precise model training
  1. Xinyun Feng1,2,
  2. Marcia Zhang2,3,
  3. Willa Yim4,
  4. Felicia Wee5,
  5. Menaka Priyadharsani Rajapakse6,
  6. Jeffrey Lim4,
  7. Chan Way Ng6,
  8. Inti Zlobec7,
  9. Bernett Lee8,9,10,
  10. Olaf Rotzchke8,
  11. Joe Yeong4 and
  12. Mai Chan Lau6,11
  1. 1Bioinformatics Institute (BII), Agency of Science, Technology and Research (A*STAR), Singapore 138671., Singapore, Singapore, Singapore
  2. 2National University of Singapore, Singapore 639798., Singapore, Singapore, Singapore
  3. 3A*STAR Bioinformatics Institute, Singapore, Singapore
  4. 4Institute of Molecular and Cell Biology (IMCB), Agency of Science, Technology and Research (A*STAR), Singapore 138673., Singapore, Singapore, Singapore
  5. 5IMCB/A*STAR, Singapore, Singapore, Singapore
  6. 6Singapore Immunology Network (SIgN), Agency of Science, Technology and Research (A*STAR), Singapore 138673., Singapore, Singapore, Singapore
  7. 7University of Bern, Bern 3008, Switzerland, Bern, Switzerland
  8. 8Singapore Immunology Network (SIgN), Agency of Science, Technology and Research (A*STAR), Singapore 138668., Singapore, Singapore, Singapore
  9. 9Centre for Biomedical Informatics, Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore 639798., Singapore, Singapore, Singapore
  10. 10A*STAR Infectious Diseases Labs (ID Labs), Agency of Science, Technology and Research (A*STAR), Singapore 138648., Singapore, Singapore, Singapore
  11. 11IMCB, Singapore, Singapore, Singapore
  • Journal for ImmunoTherapy of Cancer (JITC) preprint. The copyright holder for this preprint are the authors/funders, who have granted JITC permission to display the preprint. All rights reserved. No reuse allowed without permission.


Background Advancement in multiplex immunohistochemistry (mIHC) allows for single-cell tumor microenvironment analysis, greatly enhancing our comprehension of cancer immunology.1 However, clinical application of mIHC remains challenging due to technical variations and cost. Hematoxylin and eosin (H&E) staining, a cost-effective method commonly used in clinics, combined with emerging digital pathology,2 offers promising prospects. Increasing evidence suggests that deep learning (DL) can discern molecular variations from histomorphological features in H&E images.3 4 This has sparked research into applying DL to digital H&E images, a concept referred to as H&E 2.0.5 Model training demands accurate cell labels, but traditional manual annotation methods are labor-intensive, time-consuming, and show significant observer variability.3 6 In this study, we investigate and propose the best methods for image registration and nuclear segmentation to create accurate cell labels for training a predictive model particularly for CD3+ T-cell prediction. Our primary aim is to create an automated workflow that enables robust and precise virtual staining on widely used H&E images.

Methods The cohort includes 19 triple-negative breast cancer samples in a TMA format. The TMA slide underwent mIHC staining, was subsequently washed, stained with H&E, and imaged. The mIHC staining panel consists of DAPI/CD20/CD3/Ki67/CD163/FOXP3 stains. For each of the samples, the H&E and mIHC stained images were first cropped into individual core areas. Using SIFT from OpenCV (v4.7.0.72), the mIHC images were warped to align with their corresponding H&E images. Nuclear segmentation on H&E images was performed using the pretrained ‘2D_versatile_he’ model from StarDist (v0.8.3) where various probability thresholds were assessed (figure 1). All analysis was performed using Python programming.

Results Our findings indicate that the use of SIFT registration significantly enhances the precision of mIHC signal mapping to H&E images, both at the tissue and cellular levels. This is observed through the alignment of the nuclear mask derived from H&E images with the DAPI nuclear signal in mIHC (figure 2). Using the StarDist model on H&E images at a 0.7 probability threshold gave results closest to manual annotation, unlike the over-segmentation seen with DeepCell’s Mesmer algorithm (figure 3).

Conclusions We have presented an optimized approach for generating ground truth with minimal user involvement. This approach can be effortlessly scaled to multiple biomarkers and an increased quantity of training images. In future work, we plan to broaden our focus to include other types of tumors and evaluate the impact on the performance of virtual staining predictions.

Acknowledgements This work was supported by the Bioinformatics Institute, Singapore Immunology Network, and Insitute of Molecular and Cell Biology, Agency for Science, Technology and Research (A*STAR).


  1. Sahardi VK, Armengol G. Molecular Biomarkers in Cancer. Biomolecules. 2022;12(8):1021.

  2. Lee K, Lockhart JH, Xie M, Chaudhary R, Slebos RJC, Flores ER, Chung CH, Tan AC. Deep Learning of Histopathology Images at the Single Cell Level. Front Artif Intell. 2021;4:754641.

  3. Liu Y, Li X, Zheng A, Zhu X, Liu S, Hu M, Luo Q, Liao H, Liu M, He Y, Chen Y. Predict Ki-67 Positive Cells in H&E-Stained Images Using Deep Learning Independently From IHC-Stained Images. Front Mol Biosci. 2020;7:183.

  4. Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyo D, Moreira AL, Razavian N, Tsirigos A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med. 2018;24(10):1559–1567.

  5. Comiter C, Vaishnav ED, Ciampricotti M, Li B, Yang Y, Rodig SJ, Turner M, Pfaff KL, Jané-Valbuena J, Slyper M, Waldman J, Vigneau S, Wu J, Blosser TR, Segerstolpe Å, Abravanel D, Wagle N, Zhuang X, Rudin CM, Klughammer J, Rozenblatt-Rosen O, Kobayash-Kirschvink KJ, Shu J, Regev A. Inference of single cell profiles from histology stains with the Single-Cell omics from Histology Analysis Framework (SCHAF). bioRxiv. 2023:2023.03.21.533680.

  6. Couture HD. Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review. J Pers Med. 2022;12(12):2022.

Abstract 1282 Figure 1

Overview of study design on optimizing cell labeling for virtual staining of CD3+ cells from H&E images

Abstract 1282 Figure 2

Comparison of pre-SIFT registration (top row) and post-SIFT registration images (bottom row). (A) H&E stained (background layer) and mlHC (front layer, green) image pairs were blended together to show degree of alignment. (B) Inset of blended H&E and mlHC image as shown in (A). (C) StarDist nuclear segmentation mask derived from H&E image overlay on H&E image. (D) StarDist nuclear segmentation mask derived from H&E image overlay on counterpart mlHC DAPI signals. The registered image pair shows higher precision in alignment as compared to the non-registered image pair

Abstract 1282 Figure 3

Assessment of StarDist '2D_versatile_he' model nuclear segmentation performance. (A) StarDist nuclear segmentation algorithm performance at varying probability threshold levels (prob) with a non- maximum suppression threshold of 0.35. StarDist performance on H&E images is compared against Mesmer model from DeepCell library on mlHC images. StarDist gave significantly closer results to the ground truth than Mesmer. Ground truth cell counts are manually determined by a histology expert using Indica Labs HALO® image analysis platform. (B) Comparison of StarDist and Mesmer derived nuclear mask overlays on mlHC DAPI signals from H&E and mlHC-stained images respectively. StarDist displays accurate nuclear segmentation whereas Mesmer yields over-segmentation of the nuclei

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.