Various forms of immunotherapy, such as checkpoint blockade immunotherapy, are proving to be effective at restoring T cell-mediated immune responses that can lead to marked and sustained clinical responses, but only in some patients and cancer types1-4. Patients and tumours may respond unpredictably to immunotherapy partly owing to heterogeneity of the immune composition and phenotypic profiles of tumour-infiltrating lymphocytes (TILs) within individual tumours and between patients5,6. Although there is evidence that tumour-mutation-derived neoantigen-specific T cells play a role in tumour control2,4,7-10, in most cases the antigen specificities of phenotypically diverse tumour-infiltrating T cells are largely unknown. Here we show that human lung and colorectal cancer CD8+ TILs can not only be specific for tumour antigens (for example, neoantigens), but also recognize a wide range of epitopes unrelated to cancer (such as those from Epstein-Barr virus, human cytomegalovirus or influenza virus). We found that these bystander CD8+ TILs have diverse phenotypes that overlap with tumour-specific cells, but lack CD39 expression. In colorectal and lung tumours, the absence of CD39 in CD8+ TILs defines populations that lack hallmarks of chronic antigen stimulation at the tumour site, supporting their classification as bystanders. Expression of CD39 varied markedly between patients, with some patients having predominantly CD39- CD8+ TILs. Furthermore, frequencies of CD39 expression among CD8+ TILs correlated with several important clinical parameters, such as the mutation status of lung tumour epidermal growth factor receptors. Our results demonstrate that not all tumour-infiltrating T cells are specific for tumour antigens, and suggest that measuring CD39 expression could be a straightforward way to quantify or isolate bystander T cells.