Background Examining Hematoxylin & Eosin (H&E) images using brightfield microscopes is the gold standard of pathological diagnosis as it is an inexpensive method and provides basic information of tumors and other nuclei. Complementary to H&E-stained images, Immunohistochemical (IHC) images are crucial in identifying tumor subtypes and efficacy of treatment response. Other newer technologies such as Multiplex Immunofluorescence (mIF) in particular, identifies cells such as tumor infiltrating lymphocytes (TILs) which can be augmented via immunotherapy, an evolving form of cancer treatment. Immunotherapy helps in the manipulation of the host immune response and overcome limitations like the PD-1 (Programmed Cell Death-1) receptor induced restrictions on TIL production. If the same biopsy specimen is used for inspection, the higher order features in H&E images can be used to obtain information usually found in mIF images using Convolutional Neural Networks (CNNs), widely used in object detection and image segmentation tasks.

Methods As shown in (figure 1), firstly, a novel optical flow-based image registration paradigm is prepared to co-register H&E and mIF image pairs, aided by adaptive color thresholding and automated color clustering. Secondly, generative adversarial networks (GANs) are adapted to predict TIL (CD3, CD45) regions. For this purpose, a unique dataset is ideated and used in which a given single channel mIF image, e.g., a CD3 channel mIF image is superimposed on the corresponding H&E image. Primarily, the Pix2Pix GAN model is used to predict CD3 and/or CD45 regions.

Results The intensity-based image registration workflow is fast and fully compatible with the given dataset, with an increase in evaluation metric scores after alignment (table 1). Furthermore, this study would be the first implementation of optical flow as the registration algorithm for pathological images. Next, the use of the special dataset not only reduces penalization during the training of the Pix2Pix model, but also helped in gaining repeatable results with high scores in metrics like structural similarity index measure and peak-signal to noise ratio, with minimal effects on location accuracy (table 2 and table 3).

Conclusions This multi-modal pathological image transformation study could potentially reduce dependence on mIF and IHC images for TILs scoring, reducing the amount of tissue and cost needed for examination, as its information is derived directly from inexpensive H&E images automatically – ultimately develop into a pathologist-assisted tool for TILs scoring. This would be highly beneficial in facilities where resources are relatively limited.

Ethics Approval The Agency of Science, Technology and Research, Singapore, provided approval for the use of control tissue materials in this study IRB: 2020 112

• Download figure
• Open in new tab
• Download powerpoint

Abstract 818 Figure 1

Proposed workflow