Background Spatial omics technologies like multiplexed immunofluorescence (MIF) and spatial transcriptomics produce high dimensional characterizations of tissues. However, the utility of these data are limited by their accessibility and scalability. In this work, we use paired multimodal data and artificial intelligence to (1) discover signatures from MIF data and (2) learn what those signatures look like in histopathology data, a pipeline we call ‘signature transfer.’ Our approach allows for the retrieval of complex molecular signatures in conventional hematoxylin and eosin (HE) images. We demonstrate that our approach identifies specific microenvironments and stratifies patients by survival in unseen head and neck cancer HE cores.

Methods Signature transfer consists of a microenvironment classifier and a function to assign disease prediction scores to histopathology patches. To test this pipeline, MIF and HE images were obtained and co-registered for 308 head and neck cancer patients that underwent surgical intervention. These patients were split into training and validation test sets. The MIF images were segmented into 12 disease relevant microenvironments using a graph neural network approach.¹ A classifier was then trained to classify the different microenvironments from HE images, and a predictive score was assigned for each microenvironment. Finally, the unseen validation set HE images were classified into microenvironments and prediction scores were assigned based on the trained classifier.^2–4

Results Patient survival ROC-AUC scores were between 0.82 and 0.86 for both MIF and HE derived predictions, demonstrating the validity of the signature transfer approach for predicting patient outcomes. Some microenvironments could be translated more easily than others, with an overall accuracy over 0.4 across all validation images. Patients in the validation set were successfully stratified based on microenvironment enrichment from HE images alone (figure 1). The top 25 percentile most enriched patients in microenvironments 2 and 3, as classified by HE, show significantly improved outcomes. The cell-type compositions of each microenvironment are available from the MIF training data. These compositions can be analyzed to develop a mechanistic understanding of the biology influencing patient outcome.

Conclusions We have demonstrated patient stratification of head and neck cancer histopathology images from disease signatures learned and interpreted by MIF images. This represents a novel way to identify patients that are most likely to benefit from particular therapies using routine clinical data. In contrast to other artificial intelligence derived predictive models that only function on HE images, our approach allows a query for specific cellular, molecular, or mechanistically relevant signatures.

References

1. Wu Zhenqin, et al. ‘Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens.’ Nature Biomedical Engineering 2022;6(12):1435–1448.

2. Huang Zhi, et al. ‘A visual–language foundation model for pathology image analysis using medical twitter.’ Nature medicine 2023;29(9):2307–2316.

3. Lu Ming Y, et al. ‘A visual-language foundation model for computational pathology.’ Nature Medicine 2024; 30(3):863–874.

4. Chen Richard J, et al. ‘Towards a general-purpose foundation model for computational pathology.’ Nature Medicine 2024;30(3):850–862.

Ethics Approval Samples and data were collected according to protocols approved by the institutional review board of University of Pennsylvania Medical Center.

• Download figure
• Open in new tab
• Download powerpoint

Abstract 1373 Figure 1

Survival curve for the top 25% and bottom 75% most enriched patients for microenvironment M stratified from HE images using signature transfer