Browsing by Author "Nguyen, Minh Tan"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    On the Momentum-based Methods for Training and Designing Deep Neural Networks
    (2020-09-14) Nguyen, Minh Tan; Baraniuk, Richard G
    Training and designing deep neural networks (DNNs) are an art that often involves expensive search over candidate architectures and optimization algorithms. In my thesis, we develop novel momentum-based methods to speed up deep neural networks training and facilitate the process of designing them. For training DNNs, stochastic gradient descent (SGD) algorithms with constant momentum and its variants such as Adam are the optimization methods of choice for training DNNs. There is great interest in speeding up the convergence of these methods due to their high computational expense. Nesterov accelerated gradient (NAG) improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst. We propose scheduled restart SGD (SRSGD), a new NAG-style scheme for training DNNs. SRSGD replaces the constant momentum in SGD by the increasing momentum in NAG but stabilizes the iterations by resetting the momentum to zero according to a schedule. Using a variety of models and benchmarks for image classification, we demonstrate that, in training DNNs, SRSGD significantly improves convergence and generalization; for instance, in training ResNet-200 for ImageNet classification, SRSGD achieves an error rate of 20.93% vs. the benchmark of 22.13%. These improvements become more significant as the network grows deeper. Furthermore, on both CIFAR and ImageNet, SRSGD reaches similar or even better error rates with significantly fewer training epochs compared to the SGD baseline. For designing DNNs, we focus on the recurrent neural networks (RNNs) and establish a connection between the hidden state dynamics in an RNN and gradient descent (GD). We then integrate momentum into this framework and propose a new family of RNNs, called MomentumRNNs. We theoretically prove and numerically demonstrate that MomentumRNNs alleviate the vanishing gradient issue in training RNNs. We also demonstrate that MomentumRNN is applicable to many types of recurrent cells, including those in the state-of-the-art orthogonal RNNs. Finally, we show that other advanced momentum-based optimization methods, such as Adam and NAG with a restart, can be easily incorporated into the MomentumRNN framework for designing new recurrent cells with even better performance.
  • Thumbnail Image
    Item
    The Deep Rendering Model: Bridging Theory and Practice in Deep Learning
    (2018-10-10) Nguyen, Minh Tan; Baraniuk, Richard G
    A grand challenge in machine learning is the development of computational algorithms that match or outperform humans in perceptual inference tasks such as visual object and speech recognition. The key factor complicating such tasks is the presence of numerous nuisance variables, for instance, the unknown object position, orientation, and scale in object recognition or the unknown voice pronunciation, pitch, and speed in speech recognition. Recently, a new breed of deep learning algorithms has emerged for high-nuisance inference tasks; they are constructed from many layers of alternating linear and nonlinear processing units and are trained using large-scale algorithms and massive amounts of training data. The recent success of deep learning systems is impressive— they now routinely yield pattern recognition systems with near or super-human capabilities — but a fundamental question remains: Why do they work? Intuitions abound, but a coherent framework for understanding, analyzing, and synthesizing deep learning architectures has remained elusive. We answer this question by developing a new probabilistic framework for deep Learning, namely the Deep Rendering Model (DRM), based on a Bayesian generative probabilistic model that explicitly captures variation due to nuisance variables. The graphical structure of the model enables it to be learned from data using classical expectation-maximization techniques. Furthermore, by relaxing the generative model to a discriminative one, we can recover deep convolutional neural networks (DCNs) as well as its variants including the deep residual networks (ResNet) and the densely connected convolutional networks (DenseNet), providing insights into their successes and shortcomings as well as a principled route to their improvement. The DRMM is also applicable to semi-supervised and unsupervised learning tasks, achieving results that are state-of-the-art in several categories on the MNIST benchmark and comparable to state of the art on the CIFAR10 benchmark.