Objective: This study investigated the effect of TNF-α rs1800629 polymorphism on white matter integrity and memory function in patients with schizophrenia. Methods: Fifty-five participants with schizophrenia were enrolled in this study. They were genotyped for TNF-α rs1800629 polymorphism and underwent diffusion tensor imaging. Memory function was assessed using the Rey–Kim memory test. Participants with schizophrenia were grouped into GG homozygotes and A-allele carriers. Results: Compared to GG homozygotes, A-allele carriers had significantly lower scores for immediate and delayed recall and recognition of verbal memory and showed significantly lower fractional anisotropy in extensive brain regions. Lower total scores in immediate and delayed recall of verbal memory, immediate recall of visual memory, and figure copy of visual memory were significantly correlated with decreased mean fractional anisotropy in the white matter tracts of the corresponding brain regions. Conclusion Our findings suggest that the A-allele, which is associated with higher levels of TNF-α expression, correlates with lower connectivity of the fronto-temporal white matter compared to that in GG homozygotes. Impaired fronto-temporal connectivity may be associated with genetic vulnerability to schizophrenia, leading to verbal and visual memory deficits in patients with schizophrenia.