In this thesis, we explore the role of third-order interactions in neural computations, emphasizing their significance as a reflection of such generative processes in the physical world. We also propose using third-order interactions in probabilistic graphical models (PGMs) within the exponential family as a normative way to define a gating mechanism in generative probabilistic graphical models. By going a step beyond pairwise interactions, it empowers much more computational efficiency, like a transistor expands possible digital computations. We also demonstrate the use of third-order PGM for explaining observed properties of neural computations, particularly in context-dependent flexible divisive normalization and attention. Both can be conceptualized as a gating mechanism. As a concrete example, we show that a graphical model with three-way interactions provides a normative explanation for observed divisive normalization properties in the macaque primary visual cortex. Inference in such PGMs is nontrivial. We define Recurrent Factor Graph Neural Network (RF-GNN), a machine learning approach developed for fast approximate inference in PGMs with higher-order interactions. Experimental results on several families of graphical models demonstrate the out-of-distribution generalization capability of our method to different-sized graphs and indicate the domain in which our method outperforms Belief Propagation (BP). Moreover, we test the RF-GNN on a real-world Low-Density Parity-Check dataset as a benchmark along with other baseline models including BP variants and a stacked GNN method. Overall we find that RF-GNNs outperform other methods under high noise levels.