Discoveries leading to an improved understanding of immune surveillance of the central nervous system (CNS) have repeatedly provoked dismissal of the existence of immune privilege of the CNS. Recent rediscoveries of lymphatic vessels within the dura mater surrounding the brain, made possible by modern live-cell imaging technologies, have revived this discussion. This review emphasizes the fact that understanding immune privilege of the CNS requires intimate knowledge of its unique anatomy. Endothelial, epithelial and glial brain barriers establish compartments in the CNS that differ strikingly with regard to their accessibility to immune-cell subsets. There is a unique system of lymphatic drainage from the CNS to the peripheral lymph nodes. We summarize current knowledge on the cellular and molecular mechanisms involved in immune-cell trafficking and lymphatic drainage from the CNS, and we take into account differences in rodent and human CNS anatomy.