Browsing by Author "Wang, Qihan"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    LoFT: Finding Lottery Tickets through Filter-wise Training
    (2022-05-04) Wang, Qihan; Kyrillidis, Anastasios
    Recent work on pruning techniques and the Lottery Ticket Hypothesis (LTH) shows that there exist “winning tickets” in large neural networks. These tickets represent versions of the full model that can be trained separately to achieve comparable accuracy with respect to the full models. However, in practice the process of finding these tickets can be a burdensome task, especially when the original neural network gets larger: Often one has to pretrain the large model for at least a number of epochs. In this paper, we explore how we can empirically identify when such winning tickets emerge, and use this heuristic to design efficient pretraining algorithms. Our focus in this work is on convolutional neural networks (CNNs): To identify good filters within winning tickets, we propose a novel filter distance metric that well-represents the model convergence, without the need to know the true winning ticket or training the model in full. Our filter analysis behaves consistently with recent findings of neural network learning dynamics. Motivated by this metric, we present the LOttery ticket through Filter-wise Training algorithm, dubbed as LoFT. LoFT is a model-parallel pretraining algorithm that partitions convolutional layers in CNNs by filters to train them independently on different distributed workers, leading to reduced memory and communication costs during pretraining. Experiments show that LoFT achieves non-trivial savings in communication, while maintaining comparable or even better accuracy compared to other model-parallel training methods.