There are large individual differences in arithmetic skills. Although a number of brain-wide association studies have attempted to identify the neural correlates of these individual differences, studies have focused on relatively small sample sizes and have yielded inconsistent results. In the current voxel-based morphometry study, we merged six structural imaging datasets of children and adolescents (from 7.5 to 15 years) whose levels of arithmetic skills were assessed, leading to a combined sample of n = 536. Controlling for individual differences in age, gender, as well as language, and intelligence, we found a unique positive relation between arithmetic skill and gray matter volume in the left inferior frontal gyrus (IFG) and middle temporal gyrus (MTG). Our results suggest that individual differences in arithmetic skills are associated with structural differences in left fronto-temporal areas, rather than in regions of the parietal cortex and hippocampus that are often associated with arithmetic processing.