In recent years, there has been an explosion of interest in frustrated magnetic systems on the honeycomb lattice, a geometry well known to be associated with graphene. In this class of spin systems, different types of spin interactions such as Heisenberg exchange, bond-dependent anisotropic magnetic exchanges, and antisymmetric spin interactions may appear at the same time. The varying values of these parameters can greatly affect the magnetic phases and their properties. Understanding the role played by these interactions is significant and valuable for explaining the various experimental phenomena and for providing routes to potentially new honeycomb magnetic materials. In this thesis, I will present our study of three novel phenomena in spin systems on honeycomb lattices.

In the first part, I will present our work on explaining the mechanism of an unusual noncollinear magnetic order of Ni2+ S=1 moments with a nontrivial angle between adjacent spins appearing in a non-centrosymmetric honeycomb nickelate Ni2Mo3O8. With the help of first principles electronic structure calculations and crystal field analysis, we construct an effective spin-1 bilinear-biquadratic model with estimated exchange parameters and single ion anisotropy. By performing the variational mean-field and linear spin-wave theory calculations, we find that the crucial key to explaining the observed noncollinear spin structure is the inclusion of the Dzyaloshinskii-Moriya interaction between the neighboring spins.

In the second part, I will present our study of the topological properties and magnon Hall effect in a three-dimensional ferromagnet CrI3 that crystallizes in the ABC stacked honeycomb layers. We find that the magnon band structure and Chern numbers of the magnon branches are significantly affected by the interlayer coupling Jc. Intriguingly, we find several Weyl magnon phases separating the non-equivalent Chern insulating phases, tuned by the ratio of the interlayer coupling Jc and the third-neighbor Heisenberg interaction J3. We further show that the topological character of the magnon bands results in non-zero thermal Hall conductivity, whose sign and magnitude depend on Jc and the intra-layer couplings. Since the interlayer coupling strength Jc can be easily tuned by applying pressure to the quasi-2D material, this provides a potential route to tuning the magnon thermal Hall effect in an experiment.

In the last part, I will show our study of the possible magnon thermal Hall effect in a Kitaev model candidate material α-RuCl3. Because the model parameters of this system are uncertain, we investigate the behavior of the magnon thermal Hall conductivity κxy/T in a wide parameter regime. Through the minimization of the classical energy and linear spin-wave theory, we determine the magnetic phases and compute κxy/T in each set of parameters. We further compare the temperature and magnetic field dependence of κxy/T to the experimental results, which shows that the magnon thermal Hall effect may not be enough to reproduce the experimental data.