The purpose of this cross-sectional study is to investigate the difference in physical activity among elderly living in different areas in Japan (“Bunkyo Ward in Tokyo” (Bunkyo) and “Fuchu City in Tokyo” (Fuchu) as urban areas, and “Oyama Town in Shizuoka” (Oyama) as a non-urban area). Participants were 1859 community-dwelling residents aged 65-74 years, randomly selected from the residential registry (response rate: 68.9%). A mail survey using self-administered questionnaires was conducted. Multivariate logistic regression analyses were used to calculate the adjusted odds ratios (ORs) and 95% confidence intervals (95% CI) of various types of physical activity (eg, walking (Walking), going out (Going-out), bicycling (Bicycling), exercise habits (Exercise)), according to residential areas (reference category: Fuchu), stratified by gender, adjusting for socio-demographic variables. There was a significant difference in Going-out (ORs = 0.61 (95% CI: 0.44-0.86) for men, 0.48 (0.33-0.69) for women)), and Bicycling (0.04 (0.03-0.07) for men, 0.04 (0.02-0.07) for women) in Oyama compared to Fuchu. Furthermore, for women, there was a significant difference in Walking (0.56 (0.38-0.81)) and Exercise (0.59 (0.41-0.85)) in Oyama compared to Fuchu. There was a significant difference in Bicycling and Going-out for men in Bunkyo compared to Fuchu, but there was not a significant difference in other items. Low physical activity levels were observed in the elderly in the non-urban area compared to urban areas. The association was stronger in women. Regional difference might need to be taken into account for an effective physical activity intervention.