The composition of tumor-targeted T cell infiltrates is a major prognostic factor in colorectal cancer (CRC) outcome; however, the functional role of these populations in prolonging patient survival remains unclear. Here, we evaluated 190 patients with CRC for the presence of functionally active tumor-infiltrating lymphocytes (TILs), the tumor specificity of these TILs, and the correlation between patient TILs and long-term survival. Using intracytoplasmic cytokine staining in conjunction with HLA multimers loaded with tumor peptide and antigen-specific cytokine secretion assays, we determined that TNF-α expression delineates a population of tumor antigen-specific (TA-specific) cytotoxic T lymphocytes (CTLs) present within tumors from patients with CRC. Upregulation of TNF-α expression in TILs strongly correlated with an increase in the total amount of intratumoral TNF-α, which is indicative of tumor-specific CTL activity. Moreover, a retrospective multivariate analysis of 102 patients with CRC, which had multiple immune parameters evaluated, revealed that increased TNF-α concentration was an independent prognostic factor. Together, these results indicate that the prognostic impact of T cell infiltrates for CRC maybe largely based on subpopulations of active TA-specific T cells within the tumor, suggesting causal implication for these cells in patient survival. Additionally, these results support the use of intratumoral TNF-α, which is indicative of T cell function, as a prognostic parameter for CRC.