Microstructural evolution is a key aspect of understanding and exploiting the processing-structure-property relationship of materials. Modeling microstructure evolution usually relies on coarse-grained simulations with evolution principles described by partial differential equations (PDEs). Here we demonstrate that convolutional recurrent neural networks can learn the underlying physical rules and replace PDE-based simulations in the prediction of microstructure phenomena. Neural nets are trained by self-supervised learning with image sequences from simulations of several common processes, including plane-wave propagation, grain growth, spinodal decomposition, and dendritic crystal growth. The trained networks can accurately predict both short-term local dynamics and long-term statistical properties of microstructures assessed herein and are capable of extrapolating beyond the training datasets in spatiotemporal domains and configurational and parametric spaces. Such a data-driven approach offers significant advantages over PDE-based simulations in time-stepping efficiency and offers a useful alternative, especially when the material parameters or governing PDEs are not well determined.