Research paper
Quality assurance of intracellular cytokine staining assays: Analysis of multiple rounds of proficiency testing

https://doi.org/10.1016/j.jim.2010.08.004Get rights and content

Abstract

When evaluating candidate prophylactic HIV and cancer vaccines, intracellular cytokine staining (ICS) assays that measure the frequency and magnitude of antigen-specific T-cell subsets are one tool to monitor immunogen performance and make product advancement decisions. To assess the inter-laboratory assay variation among multiple laboratories testing vaccine candidates, the NIH/NIAID/DAIDS in collaboration with BD Biosciences implemented an ICS Quality Assurance Program (QAP). Seven rounds of testing have been conducted in which 16 laboratories worldwide participated. In each round, IFN-γ, IL-2 and/or TNF-α responses in CD4+ and CD8+ T-cells to CEF or CMV pp65 peptide mixes were tested using cryopreserved peripheral blood mononuclear cells (PBMC) from CMV seropositive donors. We found that for responses measured above 0.2%, inter-laboratory %CVs were, on average, 35%. No differences in inter-laboratory variation were observed if a 4-color antibody cocktail or a 7-color combination was used. Moreover, the data allowed identification of important sources of variability for flow cytometry-based assays, including: number of collected events, gating strategy and instrument setup and performance. As a consequence, in this multi-site study we were able to define pass and fail criteria for ICS assays, which will be adopted in the subsequent rounds of testing and could be easily extrapolated to QAP for other flow cytometry-based assays.

Introduction

The intracellular cytokine staining (ICS), enzyme-linked Immunospot (ELISpot) assay and staining with HLA-peptide multimers are technologies commonly used for the monitoring of antigen-specific immune responses. ICS has the advantage over these other techniques in that this flow-based application simultaneously permits functional and phenotypic assessment of the responding T-cell populations.

In humans, adaptive cellular immune responses play a vital role in the containment of HIV-1 replication. During primary infection, the appearance of HIV-specific cytotoxic T-lymphocytes (CTL) is correlated with decline from peak viremia (Goonetilleke et al., 2009). Moreover, the long-term, non-progressor status is associated with robust CTL responses (Rinaldo et al., 1995, Harrer et al., 1996, Betts et al., 1999), and the loss of HIV-specific T-cells is associated with rapid progression to AIDS (Klein et al., 1995). Because control of infection is required to prevent disease, and as the best licensed vaccines against other pathogens do not necessarily prevent these infections completely, a successful HIV vaccine will probably also need to elicit cell-mediated immune (CMI) responses capable of controlling HIV infection. Consequently, utilizing validated assays of CMI responses would enhance comparisons among various vaccine developers and enable data-driven prioritization of candidate vaccines.

Numerous vaccine clinical trials, conducted at many sites simultaneously, are currently testing candidate prophylactic HIV vaccines and use ICS to monitor immunogen performance and make product advancement decisions (Cheng et al, Koup et al, De Rosa and McElrath, 2008, McElrath et al., 2008). The interpretation of the results obtained from these ICS assays across different vaccine developers is a difficult task, due to the variety of methods, protocols and statistical criteria available to detect vaccine-specific T-cell responses. To make product advancement decisions, it is necessary to compare data across different trials; consequently, a standardization and Quality Assurance of ICS assay is critical. Moreover, such a Quality Assurance Program (QAP) would provide ongoing proficiency data for participating institutions to meet Good Clinical Laboratory Practice (Ezzelle et al., 2008, Sarzotti-Kelsoe et al., 2009). Benefits of the QAP include: opportunity for participants to monitor their own performance over time; use of the QAP as an internal competency test for staff once trained and qualified; and an ability to compare performance with peers running the same assay.

Published studies have addressed the intra- and inter-assay precision of ICS in whole blood and peripheral blood mononuclear cells (PBMC) (Nomura et al., 2000, Horton et al., 2007, Maecker et al., 2008, Nomura et al., 2008). A recent study by our group on standardization and precision of ICS between laboratories (Maecker et al., 2005) revealed that ICS could be performed by multiple laboratories using a common protocol with good inter-laboratory precision (18–24%). This precision improves as the frequency of responding cells increases. In an effort to standardize the assays across laboratories, in 2005, we created a QAP for ICS assays. This program was developed to assess the inter-laboratory variability when sharing a common standardized protocol and reagents. Here, we present the data from seven consecutive rounds of testing. A total of 16 laboratories from seven different countries participated in the study in which pre-tested PBMC, along with lyophilized antigens and antibodies, were distributed. The laboratories were requested to determine the percentage of cytokine+, CD4+ and CD8+ cells in each sample. The analysis of the data generated in this program has allowed us to identify factors responsible for ICS variability among laboratories that need to be taken into consideration when performing Quality Assurance of flow cytometry assays and reporting data for vaccine clinical trials.

Section snippets

Participating institutions

In the first round of testing, ten laboratories worldwide participated. This number increased to 16 by Round 5. A list of the participants is provided in Table 1. All participants have agreed on the content of this publication. Of note, most of these laboratories were involved in a previous study aimed at standardizing the protocol used in this ICS QAP (Maecker et al., 2005).

PBMC preparation and cryopreservation

Concentrated leukocytes were prepared by machine leukopheresis with anticoagulant ACD-A by BRT Laboratory (Baltimore,

Inter-laboratory variation detected using a 4-color ICS assay

Cytokine responses from the different donors to each stimulus tested were compared across laboratories in each round. The variability across laboratories was calculated as %CV or SD. As noted before (Maecker et al., 2008), %CV were not informative for low cytokine responses. Based on all the data generated in the current study, we established that %CV would stabilize and could only be used when assessing responses higher than 0.2%; hence, SD was used for cytokine responses lower than 0.19%.

The

Discussion

Currently, participation in the ICS QAP described in this study is a NIH/NIAID/DAIDS requirement for those laboratories supporting trials where NIAID holds the IND application. The implementation of this QAP has been a useful tool to determine whether a specific laboratory is capable of providing reliable data for evaluation of T-cell responses to HIV vaccine candidates.

From the data collected during seven rounds of proficiency testing, we were able to evaluate the reproducibility of 4- and

Acknowledgment

This project has been funded in whole or in part with Federal funds from the National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under Contract No. HHSN26620050022C.

References (24)

  • Cheng, C., Gall, J.G., Nason, M., King, C.R., Koup, R.A., Roederer, M., McElrath, M.J., Morgan, C.A., Churchyard, G.,...
  • J.H. Cox et al.

    Results of an ELISPOT proficiency panel conducted in 11 laboratories participating in international human immunodeficiency virus type 1 vaccine trials

    AIDS Res. Hum. Retroviruses

    (2005)
  • Cited by (37)

    • Generation of TCR-engineered reference cell samples to control T-cell assay performance

      2020, Methods in Enzymology
      Citation Excerpt :

      In this context, the inherent complexity of cellular immune assays constitutes a significant challenge. This was highlighted in a range of studies and proficiency panels for most of the commonly used T-cell assays (Britten et al., 2009; Chudley et al., 2014; Jaimes et al., 2011; Janetzki & Britten, 2012; Welters et al., 2012). One key solution may arise from the use of reference samples that contain a defined and stable number of functional Ag-specific T cells and which may serve the need to control immune assay performance over time and across institutions (Maecker et al., 2010; van der Burg et al., 2011).

    • Development of an RNA-based kit for easy generation of TCR-engineered lymphocytes to control T-cell assay performance

      2018, Journal of Immunological Methods
      Citation Excerpt :

      In the application phase, the pre-tested TERS aliquots can be subsequently used to control assay performance in a variety of ways, such as prior to assessing a series of test samples (e.g. patient samples) as well as after the last test sample has been run or in each experiment, on each day, or if required, for each test item depending on the user's preferences. TERS can be used to control and to detect technical sources of assay variation as presented in a range of studies and proficiency panels for most of the commonly used T-cell assays such HLA-peptide multimer staining (Britten et al., 2009), intracellular cytokine flow cytometry (Jaimes et al., 2011; Welters et al., 2012) and ELISpot assay (Chudley et al., 2014; Janetzki and Britten, 2012). Furthermore, TERS can be used for the validation and the day to day quality control of one T-cell assay.

    View all citing articles on Scopus
    1

    Present address: Social and Scientific Systems, Inc., 8757 Georgia Avenue, 12th Floor, Silver Spring, MD 20910, United States.

    View full text