As a promising candidate for nonvolatile memory devices, the hafnia-based ferroelectric system has recently been a hot research topic. Although significant progress has been made over the past decade, the endurance problem is still an obstacle to its final application. In perovskite-based ferroelectrics, such as the well-studied Pb[ZrxTi1−x]O3 (PZT) family, polarization fatigue has been discussed within the framework of the interaction of charged defects (such as oxygen vacancies) with the moving domains during the switching process, particularly at the electrode-ferroelectric interface. Armed with this background, a hypothesis is set out to test that a similar mechanism can be in play with the hafnia-based ferroelectrics. The conducting perovskite La-Sr-Mn-O is used as the contact electrode to create La0.67Sr0.33MnO3 / Hf0.5Zr0.5O2 (HZO)/ La0.67Sr0.33MnO3 capacitor structures deposited on SrTiO3-Si substrates. Nanoscale X-ray diffraction is performed on single capacitors, and a structural phase transition from polar o-phase toward non-polar m-phase is demonstrated during the bipolar switching process. The energy landscape of multiphase HZO has been calculated at varying oxygen vacancy concentrations. Based on both theoretical and experimental results, it is found that a polar to non-polar phase transformation caused by oxygen vacancy redistribution during electric cycling is a likely explanation for fatigue in HZO.