Article Text

Download PDFPDF

1302 A machine learning toolkit for automated processing of multiplexed immunofluorescence images
  1. Monee McGrady,
  2. Matthew Bieniosek,
  3. Aaron Mayer and
  4. Alexandro E Trevino
  1. Enable Medicine, Menlo Park, CA, USA


Background Multiplexed immunofluorescence imaging is a powerful spatial biology tool that can produce rich marker expression data at single-cell resolution and whole-slide scales. High parameter images have been particularly useful for characterizing the composition and arrangement of cellular microenvironments and cell interactions in native tissue contexts. These spatial features have been shown to have predictive or prognostic value in clinical tasks, including risk-based cohort stratification, biomarker discovery, therapy response, and survival prediction across different cancers.1–4 Analyzing multiplexed immunofluorescence images can be fraught with both technical challenges, including tissue quality, stain quality, and artifact removal; and computational challenges, such as the processing, curation, and analysis of extremely large raw image files. Excessive time spent manually annotating images, checking image and stain quality, aligning multiple images for multimodal analyses, and creating regions of interest to include or exclude in downstream analysis severely limits the throughput and utility of this technology. To address these challenges, we have developed and tested a suite of deep learning models and image processing utilities in a flexible, performant, and easy-to-use Python toolkit. These tools allow analysts to prepare multiplex immunofluorescence images for accurate analysis at scale in an automated fashion.

Methods Here we present a Python toolkit which includes six functions for processing multiplex immunofluorescence images: a function to calculate image quality based on the NIQE algorithm,5 a deep learning model6 to check the stain quality of six immune markers (CD3e, CD4, CD8, CD45, CD45RO, CD68), automated alignment of different assays using a SIFT method,7 deep learning8 based tissue detection, deep learning based artifact detection, and a thresholding function for creating masks based on biomarker expression.

Results The use of this toolkit reduces turnaround time and analysis time by decreasing manual intervention (visual quality control checks, hand-drawn annotations, manual alignment) and increases the accuracy of results by accurately detecting artifacts to exclude (detects non biological artifacts with 97% accuracy, 95% Jaccard similarity).

Conclusions The analysis of multiplexed immunofluorescence images is key to the advancement of the spatial biology field. Automating the preprocessing steps required to accurately analyze these images is necessary to be able to keep up with the amount of data analysis needed to make these advancements. These data-driven tools should allow biologists and clinicians to focus on research questions rather than technical hurdles. We further expect these models to improve as new data are incorporated.


  1. Lin J-R, Chen Y-A, Campton D, Cooper J, Coy S, Yapp C, et al. High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers. Nat Cancer. 2023 Jun 22;

  2. Wu Z, Trevino AE, Wu E, Swanson K, Kim HJ, D’Angio HB, et al. SPACE-GM: geometric deep learning of disease-associated microenvironments from multiplex spatial protein profiles. BioRxiv. 2022 May 13;

  3. Schürch CM, Bhate SS, Barlow GL, Phillips DJ, Noti L, Zlobec I, et al. Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell. 2020 Sep 3;182(5):1341–1359.e19.

  4. Phillips D, Matusiak M, Gutierrez BR, Bhate SS, Barlow GL, Jiang S, et al. Immune cell topography predicts response to PD-1 blockade in cutaneous T cell lymphoma. Nat Commun. 2021 Nov 18;12(1):6726.

  5. Mittal A, Moorthy AK, Bovik AC. No-reference image quality assessment in the spatial domain. IEEE Trans Image Process. 2012 Dec;21(12):4695–708.

  6. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. arXiv. 2015;

  7. Lowe DG. Distinctive Image Features from Scale-Invariant Keypoints. Int J Comput Vis. 2004 Nov;60(2):91–110.

  8. Chen L-C, Papandreou G, Schroff F, Adam H. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv. 2017;

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.