Lin, Yingyan2024-01-222024-01-222023-082023-12-06August 202Puckett, Daniel. "TACoS: Transformer and Accelerator Co-Search Towards Ubiquitious Vision Transformer Acceleration." (2023) Master's thesis, Rice University. https://hdl.handle.net/1911/115349https://hdl.handle.net/1911/115349Recent works have combined pruned Vision Transformer (ViT) models and specialized accelerators to achieve strong accuracy/latency tradeoffs in many computer vision tasks. However, it takes a significant amount of expert labor to adapt these systems to real-world scenarios with specific accuracy, latency, power, and/or area constraints. Automating the design and exploration of these systems is a promising solution but is hampered by two unsolved problems: 1) Existing methods of pruning the attention maps of a ViT model involve fully training the model, pruning its attention maps, then fine-tuning the model. This is infeasible when exploring a design space containing millions of model architectures. 2) The design space is complicated and the system’s area efficiency, scalability, and data movement are hurt because we lack a unified accelerator template that efficiently computes each operation in sparse ViT models. To solve these problems, I propose TACoS: Transformer and Accelerator Co-Search, the first automated method to co-design pruned ViT model and accelerator pairs. TACoS answers the above challenges using 1) a novel ViT search algorithm that simultaneously prunes and fine-tunes many models at many different sparsity ratios, and 2) the first unified ViT accelerator template, which efficiently accelerates each operation in sparse ViT models using adaptable PEs and reconfigurable PE lanes. With these innovations, the TACoS framework quickly and automatically designs state-of-the-art systems for real-world applications and achieves accuracy/latency tradeoffs superior to hand-crafted ViT models and accelerators.application/pdfengCopyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.Vision Transformerhardware acceleratormachine learningpruningsparsityThesis2024-01-22