Increasing light-intensity physical activity (LPA) or moderate-to-vigorous physical activity (MVPA) and decreasing sedentary behavior (SB) are important to maintain cognitive function in older adults. However, there is a lack of studies examining the influence of replacing time spent in SB, LPA and MVPA on cognitive function. Therefore, this study applied an isotemporal substitution (IS) model approach to examine the associations of objectively measured SB, LPA, and MVPA on multiple cognitive functions in older adults. We analyzed data from 145 older persons (mean age, 75.1 ± 4.5 years; 61.4% women) in Kasama City, Japan. We assessed SB, LPA, and MVPA for seven consecutive days with a triaxial accelerometer. To evaluate the cognitive function, we used the Five-Cog test (character position referencing task, category cued recall task, clock drawing task, animal name listing task, and analogy task). The IS model of the multiple regression analysis was used to examine the effects of replacing time spent on one activity type with 30 minutes per day of SB, LPA, and MVPA on multiple cognitive functions, adjusting for age, gender, body mass index, education, smoking, sleep duration, history of stroke, medication for hypertension, dyslipidemia, and diabetes and accelerometer wear time. The results showed that reallocation from SB to LPA was significantly related to the category cued recall task score (B = 0.69, 95% confidence interval 0.07 - 1.31). In contrast, other reallocations were not associated with any cognitive tasks. The results suggest that substituting SB with LPA is positively associated with verbal episode memory among older Japanese adults.