While turbulence remains the oldest unsolved mystery in physics, recent efforts in building high-resolution physics-based simulation models, availability of highquality observational data, coupled with advances in data-driven scientific computing, offers significant hope for modeling and predicting turbulent flow. On the other hand, data-driven methods, primarily fueled by the unprecedented success of deep learning, often suffer from the lack of interpretability and rigorous theoretical understanding of their inner working-mechanism. In this thesis, I propose, a principled approach to constructing data-driven deep learning algorithms leveraging rigorous theories of deep learning, that are motivated from established theories of numerical analysis and fluid physics, to build prediction models for complex turbulent flows in both engineering applications and geophysical fluid dynamics. I would discuss how these data-driven models can be interpreted from the lenses of physics, how they can fail under certain circumstances, and how these failure modes can be mitigated once we understand their inner working mechanism. In conclusion, I would discuss how deep learning-based algorithms can be reliably used for scientific applications specifically focusing on highly chaotic, nonlinear geophysical turbulent flows on several systems of increasing complexity from canonical systems, fully-coupled climate models, to actual atmospheric observations.