The structure of the genome shapes the distribution of genetic diversity and sequence divergence. To investigate how the relationship between chromosome size and recombination rate affects sequence divergence between species, we combined empirical analyses and evolutionary simulations. We estimated pairwise sequence divergence among 15 species from three different mammalian clades—Peromyscus rodents, Mus mice, and great apes—from chromosome-level genome assemblies. We found a strong significant negative correlation between chromosome size and sequence divergence in all species comparisons within the Peromyscus and great apes clades but not the Mus clade, suggesting that the dramatic chromosomal rearrangements among Mus species may have masked the ancestral genomic landscape of divergence in many comparisons. Our evolutionary simulations showed that the main factor determining differences in divergence among chromosomes of different sizes is the interplay of recombination rate and selection, with greater variation in larger populations than in smaller ones. In ancestral populations, shorter chromosomes harbor greater nucleotide diversity. As ancestral populations diverge, diversity present at the onset of the split contributes to greater sequence divergence in shorter chromosomes among daughter species. The combination of empirical data and evolutionary simulations revealed that chromosomal rearrangements, demography, and divergence times may also affect the relationship between chromosome size and divergence, thus deepening our understanding of the role of genome structure in the evolution of species divergence.