Quantum optics is the field of physics that studies the interaction between light and matter under a fully quantum mechanical treatment. In recent decades, relentless advances in the technologies used for trapping, cooling, and controlling atoms and molecules have allowed us to finally start exploring experimentally the physics displayed by different cornerstone models of quantum optics originally proposed in the second half of the 20th century.

In this thesis, we propose multiple extensions to those fundamental models, expanding the already interesting physics displayed by them into new and exciting directions. More specficially, we study these generalized models in the context of quantum phase transitions both in and out of equilibrium and show how new phases of matter can be unlocked in these extended models.

In Chapters 1 and 2 we briefly introduce the reader to the realms of quantum phase transitions and quantum optics. Once the fundamental aspects of both of these fields are covered, we present the superradiant phase transition displayed in the Dicke and Rabi models in Chapter 3. This phase transition will be the central topic we will be orbiting around in the subsequent chapters. In Chapter 4 we propose a generalization of the quantum Rabi model, where N cavities, instead of one, are connected to each other through photon hopping. In this system, named the quantum Rabi ring, interesting chiral superradiant phases can develop as a function of the hopping phase and the geometry of the ring. In particular, we find that geometrical frustration plays a key role in determining the stable phases in equilibrium. Moreover, frustration manifests itself in the quantum Rabi ring through unconventional non-symmetric critical exponents, a phenomenon reported before, but hardly observed in condensed matter materials. Finally, in Chapter 5, we introduce a different extension to the Dicke model where instead of two-level atoms, three-level atoms are considered. In this system, which we have named the tricritical Dicke model, the superradiant phase transition can occur through first-order, second-order, and third-order critical points. We study the critical properties of the system in and out of equilibrium and find that in the open-system description, the tricritical points are still present but are now accompanied by regions of bistability where the final fate depends highly on the initial conditions. These regions of bistability represent potentially interesting environments to investigate dynamical features and steady-state preparation schemes.

The critical phenomena displayed by the extended models proposed here, such as multicriticality, geometrically frustrated quantum phases, and chiral phases breaking time-reversal symmetry, are all features that have been observed, although not abundantly, or proposed in condensed matter physics. However, the unmatched tunability and control of atomic and molecular experimental platforms, where the models presented in this thesis are proposed to be realized, causes these extended models to become promising candidates for investigating these features in parameter regimes and conditions never explored before.