To elucidate the origin of nematic order in Fe-based superconductors, we report a Raman scattering study of lattice dynamics, which quantify the extent of C4-symmetry breaking, in BaFe2As2 and FeSe. FeSe possesses a nematic ordering temperature Ts and orbital-related band-energy split below Ts that are similar to those in BaFe2As2, but unlike BaFe2As2 it has no long-range magnetic order. We find that the Eg phonon-energy split in FeSe becomes substantial only well below Ts, and its saturated value is much smaller than that in BaFe2As2. Together with reported results for the Ba(Fe1−xCox)2As2 family, the data suggest that magnetism exerts a major influence on the lattice.