This study aimed to explore the association of breakfast (BF) dietary patterns with physical activity (PA) and fitness in elementary school children. This study enrolled 242 school children of 5th and 6th grade. BF dietary patterns were evaluated by questionnaires on whether the child consumes the following food categories: staple, main dish, side dish, soup, milk or dairy products, and fruit. Individual PA levels were categorized into high and low PA groups using a PA scale for children. The measured 8 assessments of physical fitness scores were converted to Z-scores normalized for sex, age and height as previously reported. High PA group had significantly higher scores in running and muscle strength as well as in the total score of physical fitness. The frequency of eating BF (with or without BF omission) and the level of PA were not significantly related in both sexes. However, in girl children, BF, including staple, main dish, side dish, and soup, was significantly more prevalent in high PA group than low PA group (55.9% vs. 32.1%; χ2 = 5.638, p = 0.018). A similar tendency was observed in girls who had BF, including staple, main dish, and side dish (49.0% vs. 31.1%; χ2 = 3.720, p = 0.054). No associations between dietary patterns and PA were observed in boys. The results suggested that the frequency of eating BF was not associated with PA levels, which was significantly related to physical fitness in Japanese 5th and 6th grade elementary school children. In contrast, the results indicated that a high-quality BF dietary pattern, such as a meal including staple, main dish, side dish, and soup, might be associated with a physically active girl child.