Background: and Purpose Alzheimer’s disease (AD) is the most-prevalent form of dementia and imposes substantial burdens at the personal and societal levels. The apolipoprotein E (APOE) ε4 allele is a genetic factor known to increase AD risk and exacerbate brain atrophy and its symptoms. We aimed to provide a comprehensive review of the impacts of APOE ε4 on brain atrophy in AD as well as in mild cognitive impairment (MCI) as a transitional stage of AD. Methods: We performed a coordinate-based meta-analysis of voxel-based morphometry studies to compare gray-matter atrophy patterns between carriers and noncarriers of APOE ε4. We obtained coordinate-based structural magnetic resonance imaging data from 1,135 individuals who met our inclusion criteria among 12 studies reported in PubMed and Google Scholar. Results: We found that atrophy of the hippocampus and parahippocampus was significantly greater in APOE ε4 carriers than in noncarriers, especially among those with AD and MCI, while there was no significant atrophy in these regions in healthy controls who were also carriers. Conclusions The present meta-analysis has highlighted the significant link between the APOE ε4 allele and hippocampal atrophy in both AD and MCI, which emphasizes the critical influence of the allele on neurodegeneration, especially in the hippocampus. These findings improve the understanding of AD pathology, potentially facilitating progress in early detection, targeted interventions, and personalized care strategies for individuals at risk of AD who carry the APOE ε4 allele.