Article Text

Download PDFPDF

1279 H&E-2.0: an extended application of ST-net to liver cancer and its generalizability
  1. Samuel Lee Cong1,2,
  2. Bok Leong Chua3,
  3. Felicia Wee4,
  4. Chan Way Ng1,
  5. Nicholas Da Zhi Ang1,
  6. Thian Juea Reiner Sim1,5,
  7. Chu Yuan1,2,
  8. Menaka Priyadharsani Rajapakse1,
  9. Solomonraj Wilson1,
  10. Jeffrey Lim6,
  11. Abu Bakr Bakr Azam3,
  12. Yiyu Cai3,
  13. Joe Yeong6 and
  14. Mai Chan Lau1,7
  1. 1Singapore Immunology Network (SIgN), Agency of Science, Technology and Research (A*STAR), Singapore 138673., Singapore, Singapore, Singapore
  2. 2National University of Singapore, Singapore 119077., Singapore, Singapore, Singapore
  3. 3Nanyang Technological University, Singapore 639798., Singapore, Singapore, Singapore
  4. 4IMCB/A*STAR, Singapore, Singapore, Singapore
  5. 5National University of Singapore High School of Math and Science, Singapore 129957., Singapore, Singapore, Singapore
  6. 6Institute of Molecular and Cell Biology (IMCB), Agency of Science, Technology and Research (A*STAR), Singapore 138673., Singapore, Singapore, Singapore
  7. 7IMCB, Singapore, Singapore, Singapore
  • Journal for ImmunoTherapy of Cancer (JITC) preprint. The copyright holder for this preprint are the authors/funders, who have granted JITC permission to display the preprint. All rights reserved. No reuse allowed without permission.


Background Spatial transcriptomics (ST) has greatly advanced gene biomarker discovery1 and understanding of intra-tumoral heterogeneity.2 Deep learning methods applied to H&E images have improved clinical trials,3 inspiring the term H&E 2.0.4 Our work aims to enhance this by applying deep learning to H&E images in ST, reducing costs and enabling retrospective analysis on public H&E datasets without ST labels. Convolutional Neural Network (CNN) based methods like ST-Net5 have shown promise in breast cancer studies. We aim to address two aspects not covered by ST-Net: applicability to other tumors and model generalizability. We have retrained ST-Net on an in-house H&E imaging dataset from a Hepatocellular carcinoma (HCC) patient and tested its gene expression prediction capability on a public dataset, TCGA-LIHC.

Methods A single HCC sample was profiled using the 10x Visium technique. Image patches of 296x296 pixels were produced from the H&E image, each encompassing a Visium spot (figure 1). The ST-Net5 was trained using the image patches and Visium readouts of the top 250 most prevalent genes in this sample. A randomly selected 20% patches were held-out for validation purposes. The genes CD74 and ALB were selected for further investigation due to their status as HCC biomarkers.6 7 The ST-Net model, trained on HCC data, was validated using the public TCGA liver cancer cohort, specifically focusing on the two samples exhibiting the highest and lowest expression of ALB. To manage computational difficulties, we randomly selected 10% of all the image tiles for model prediction.

Results In the 20% held-out regions, the model attained an average Spearman’s correlation of 0.34 across the 250 target genes, with correlations of 0.53 for ALB and 0.34 for CD74 (figure 2A). The gene expression map generated by the model was in agreement with the Visium measurements for both ALB and CD74 (figure 2B). Evaluation on TCGA liver cancer cohort shows promising results. A relatively higher%regions were predicted to have increased ALB expression in samples exhibiting the highest bulk RNAseq values as compared to those exhibiting the lowest bulk RNAseq values, 1.34% and 0.323% vs. 0.234% and 0.0187%, respectively (figure 3).

Conclusions We present an extended application of existing ST prediction mode, ST-Net, in HCC and in public TCGA dataset. Future efforts to improve the model’s performance and generalizability could encompass implementing H&E color normalization and enhancing model explainability, such as through Gradient-weighted Class Activation Mapping (Grad-CAM).

Acknowledgements This work was supported by the Bioinformatics Institute, Singapore Immunology Network, and Insitute of Molecular and Cell Biology, Agency for Science, Technology and Research (A*STAR)


  1. Zhang L, Chen D, Song D, Liu X, Zhang Y, Xu X, Wang X. Clinical and translational values of spatial transcriptomics. Signal Transduct Target Ther. 2022;7(1):1–17. doi: 10.1038/s41392–022-00960-w.

  2. Li Q, Zhang X, Ke R. Spatial transcriptomics for tumor heterogeneity analysis. Front Genet. 2022;13:906158. doi: 10.3389/fgene.2022.906158.

  3. Qaiser T, Lee CY, Vandenberghe M, Yeh J, Gavrielides MA, Hipp J, Scott M, Reischl J. Usability of deep learning and H&E images predict disease outcome-emerging tool to optimize clinical trials. NPJ Precis Oncol. 2022;6(1):1–12. doi: 10.1038/s41698–022-00275–7.

  4. Comiter C, Vaishnav ED, Ciampricotti M, Li B, Yang Y, Rodig SJ, Turner M, Pfaff KL, Jané-Valbuena J, Slyper M, Waldman J, Vigneau S, Wu J, Blosser TR, Segerstolpe Å, Abravanel D, Wagle N, Zhuang X, Rudin CM, Regev A. Inference of single cell profiles from histology stains with the Single-Cell omics from Histology Analysis Framework (SCHAF). bioRxiv. 2023 Mar 21;2023.03.21.533680. doi: 10.1101/2023.03.21.533680.

  5. He B, Bergenstråhle L, Stenbeck L, Abid A, Andersson A, Borg Å, Maaskola J, Lundeberg J, Zou J. Integrating spatial gene expression and breast tumor morphology via deep learning. Nat Biomed Eng. 2020;4(8):827–834. doi: 10.1038/s41551–020-0578-x.

  6. Fu X, Yang Y, Zhang D. Molecular mechanism of albumin in suppressing invasion and metastasis of hepatocellular carcinoma. Liver Int. 2022;42(3):696–709. doi: 10.1111/liv.15115.

  7. Xiao N, Li K, Zhu X, Xu B, Liu X, Lei M, Sun HC. CD74+ macrophages are associated with favorable prognosis and immune contexture in hepatocellular carcinoma. Cancer Immunol Immunother. 2022;71(1):57–69. doi: 10.1007/s00262–021-02962-z.

Abstract 1279 Figure 1

An overview of our proposed study on (A) the extended application of ST-net to liver cancer and (B) its generalizability assessment on public TCGA liver cancer cohort. A resected HCC tissue was profiled using Visium technique, accompanied by a corresponding H&Estained image. The H&E image was tiled and split into an 80/20 train/test sets for re-training and validating the ST-Net. Once the model is trained, a gene expression map can be inferred from unseen image patch input

Abstract 1279 Figure 2

Evaluation of ST-Net performance on the 20% held-out dataset of the HCC sample. (A) This shows the distribution of Spearman’s correlation for the 250 target genes (P < 0.05); each dot on the boxplot represents an individual gene. The highest performing 10 genes are labeled for ease of reference. (B) The gene expression map created by the model (on the left) is compared to the actual Visium measurements (ground truth on the right) for ALB and CD74; individual dots represent model output and Visium spots respectively. The values were center-adjusted to zero and standard normalized to aid visualization

Abstract 1279 Figure 3

Evaluation of model perfomance on public TCGA liver cancer cohort. In the randomly selected 10% regions (image patches), a relatively higher number of regions were predicted to have increased ALB expression (> 0.5) in (top) samples exhibiting the highest bulk RNAseq values as compared to (bottom) samples exhibiting the lowest bulk RNAseq values. Each dot in the figure represents a randomly selected patch. Regions (image patches) with the highest predicted ALB expression are highlighted in yellow and increased in size

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.