The present study aimed to examine the effects of community-based intervention on cognitive function and hand finger dexterity in older adults at different levels of time to go out. Forty men and women (age, 73 ± 1 years) participated in supervised group activity and seated exercise for 60 min per session, once each week during an 8-week intervention. The participants wore an activity monitor for 1 week to determine baseline values and for the 8 weeks of intervention. Mini-mental state examination (MMSE) and pegboard test, which is related to cognitive function, were assessed before and after the intervention. Based on the total time to go out at baseline, the participants were assigned to Control group (> 60 min/day, n = 18) or Short group (≦ 60 min /day, n = 22), and then analyzed. After the 8 weeks of intervention, the Control and Short groups improved physical fitness parameters such as handgrip strength. Although MMSE in the both groups did not reach statistically significant level, these values tended to increase slightly from the baseline. Interestingly, two-way repeated-measures analysis of variance indicated significant interaction of pegboard test, and the score significantly increased only in the Short group. Total physical activity and moderate-vigorous physical activity in the both groups did not change significantly between the baseline and intervention periods. Therefore, these results suggest that the trainability of pegboard test, which is an index of hand finger dexterity and is related to cognitive function, would differ depending on the time to go out at baseline in older adults.