We demonstrate that memorization (perfectly fitting the training data) is necessary to avoid mode collapse in generative networks. Using a straightforward measure of the distance between the training data points and the closest point in the range of the generator, we study how well current generative models memorize in terms of the training dataset size, data distribution, and generator architecture. An important hallmark of our GOoF measure is that it does not require a second, trained model as with Frechet Inception Distance or Inception Score. The GOoF measure quantifies that the successful, popular generative models DCGAN, WGAN, and BigGAN fall far short of memorization. Our analysis inspires a new method to circumvent mode collapse by subsampling the training data (either randomly or with k-means clustering); we discuss the links to overparameterization.

Mixtures of generative adversarial networks (GANs) are closely related to subsampling methods. We study these mixtures in the context of memorization and density estimation to show that mixtures of GANs are superior to training a single GAN under certain assumptions. Furthermore, we construct a theoretic framework that explains how single GANs, mixtures of GANs, conditional GANs, and Gaussian mixture GANs are all related to each other by modifying the typical GAN optimization problem. Finally, we show empirically that our modified optimization problem has a memorization sweet spot which can be found with hyperparameter tuning.