Article Text

Download PDFPDF

818 Using deep learning approaches with mIF images to enhance T cell identification for tumor -automation of infiltrating lymphocytes (TILs) scoring on H&E images
  1. Abu Bakr Azam1,
  2. Yu Qing Chang2,
  3. Matthew Leong Tze Ker1,
  4. Denise Goh3,
  5. Jeffrey Chun Tatt Lim3,
  6. Mai Chan Lau3,
  7. Benedict Tan3,
  8. Lihui Huang1,
  9. Joe Yeong3 and
  10. Yiyu Cai1
  1. 1Nanyang Technological University, Singapore, Singapore
  2. 2Royal College of Surgeons in Ireland, Dublin, Ireland
  3. 3A*Star Singapore, Singapore, Singapore


Background Examining Hematoxylin & Eosin (H&E) images using brightfield microscopes is the gold standard of pathological diagnosis as it is an inexpensive method and provides basic information of tumors and other nuclei. Complementary to H&E-stained images, Immunohistochemical (IHC) images are crucial in identifying tumor subtypes and efficacy of treatment response. Other newer technologies such as Multiplex Immunofluorescence (mIF) in particular, identifies cells such as tumor infiltrating lymphocytes (TILs) which can be augmented via immunotherapy, an evolving form of cancer treatment. Immunotherapy helps in the manipulation of the host immune response and overcome limitations like the PD-1 (Programmed Cell Death-1) receptor induced restrictions on TIL production. If the same biopsy specimen is used for inspection, the higher order features in H&E images can be used to obtain information usually found in mIF images using Convolutional Neural Networks (CNNs), widely used in object detection and image segmentation tasks.

Methods As shown in (figure 1), firstly, a novel optical flow-based image registration paradigm is prepared to co-register H&E and mIF image pairs, aided by adaptive color thresholding and automated color clustering. Secondly, generative adversarial networks (GANs) are adapted to predict TIL (CD3, CD45) regions. For this purpose, a unique dataset is ideated and used in which a given single channel mIF image, e.g., a CD3 channel mIF image is superimposed on the corresponding H&E image. Primarily, the Pix2Pix GAN model is used to predict CD3 and/or CD45 regions.

Results The intensity-based image registration workflow is fast and fully compatible with the given dataset, with an increase in evaluation metric scores after alignment (table 1). Furthermore, this study would be the first implementation of optical flow as the registration algorithm for pathological images. Next, the use of the special dataset not only reduces penalization during the training of the Pix2Pix model, but also helped in gaining repeatable results with high scores in metrics like structural similarity index measure and peak-signal to noise ratio, with minimal effects on location accuracy (table 2 and table 3).

Conclusions This multi-modal pathological image transformation study could potentially reduce dependence on mIF and IHC images for TILs scoring, reducing the amount of tissue and cost needed for examination, as its information is derived directly from inexpensive H&E images automatically – ultimately develop into a pathologist-assisted tool for TILs scoring. This would be highly beneficial in facilities where resources are relatively limited.

Ethics Approval The Agency of Science, Technology and Research, Singapore, provided approval for the use of control tissue materials in this study IRB: 2020 112

Abstract 818 Figure 1

Proposed workflow

Abstract 818 Table 1

Image registration metrics

Abstract 818 Table 2

CD3 negative regions examples

Abstract 818 Table 3

CD3 positive regions examples

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.