Spontaneous symmetry breaking—the phenomenon in which an infinitesimal perturbation can cause the system to break the underlying symmetry—is a cornerstone concept in the understanding of interacting solid-state systems. In a typical series of temperature-driven phase transitions, higher-temperature phases are more symmetric due to the stabilizing effect of entropy that becomes dominant as the temperature is increased. However, the opposite is rare but possible when there are multiple degrees of freedom in the system. Here, we present such an example of a symmetry-ascending phenomenon upon cooling in a magnetic kagome metal FeGe by utilizing neutron Larmor diffraction and Raman spectroscopy. FeGe has a kagome lattice structure with simple A-type antiferromagnetic order below Néel temperature TN≈400 K and a charge density wave (CDW) transition at TCDW≈110 K, followed by a spin-canting transition at around 60 K. In the paramagnetic state at 460 K, we confirm that the crystal structure is indeed a hexagonal kagome lattice. On cooling to around TN, the crystal structure changes from hexagonal to monoclinic with in-plane lattice distortions on the order of 10−4 and the associated splitting of the double-degenerate phonon mode of the pristine kagome lattice. Upon further cooling to TCDW, the kagome lattice shows a small negative thermal expansion, and the crystal structure gradually becomes more symmetric upon further cooling. A tendency of increasing the crystalline symmetry upon cooling is unusual; it originates from an extremely weak structural instability that coexists and competes with the CDW and magnetic orders. These observations are against the expectations for a simple model with a single order parameter and hence can only be explained by a Landau free energy expansion that takes into account multiple lattice, charge, and spin degrees of freedom. Thus, the determination of the crystalline lattice symmetry as well as the unusual spin-lattice coupling is a first step towards understanding the rich electronic and magnetic properties of the system, and it sheds new light on intertwined orders where the lattice degree of freedom is no longer dominant.