This study evaluated the relation between physical fitness and survival. This prospective cohort study included 74–88-year-old women who were living at home and who reported no disability in activities of daily living. Subjects completed physical fitness tests assessing walking ability, muscular strength, flexibility, agility, and balance. They were administered a follow-up interview 14 years later. Of 63 older women, 58 died during that 14-year period. Among the 16 physical fitness measurements, significant association with survival time was found for 11 measurements using Cox proportional hazards model (Adjusted for age, BMI, and TMIG index of competence score). Among them, maximum walking time was an indicative main physical fitness factor for predicting the survival of elderly women. The hazard ratio per 1 s of 10m maximum walking time was 1.93 (95% Confidential Interval [CI], 1.54–2.41). Among muscular strength, flexibility, agility, and balance, the most influential physical fitness element to survival was agility (nerve reaction time). Therefore, the functional decline of the nervous system is expected to affect survival strongly. When 10 m maximal walking speed is 8.5 s or more, the median survival time of elderly women is 6.75 years (95%CI, 6.25–7.67), but if the time is less than 8.5 s it is 11.58 years (95%CI, 10.25–13.08, p<0.0001; log-rank test). The ability to move rapidly is identified as the main physical fitness factor associated with survival.