Declarative Machine Learning with Einsummable

Simple item page
dc.contributor.advisorJermaine, Chris Men_US
dc.creatorBourgeois, Daniel Christopheren_US
dc.date.accessioned2024-08-30T15:58:35Zen_US
dc.date.available2024-08-30T15:58:35Zen_US
dc.date.created2024-08en_US
dc.date.issued2024-07-23en_US
dc.date.submittedAugust 2024en_US
dc.date.updated2024-08-30T15:58:35Zen_US
dc.description.abstractModern tensor-based machine learning (ML) systems such as PyTorch and TensorFlow have high performance but significant limitations for large-scale ML. These systems require a programmer to manually decompose ML computations so that they can run on multiple machines. Not only is this challenging for end-users, but moving from one hardware setup to the next requires writing a lot of code. We introduce a new end-to-end ML system called ``Einsummble'' that automatically decomposes computations to match the available hardware. Unlike existing systems, we are guided by one fundamental design principle: at all costs, the user may only say what they want to compute, not how it is to be computed. Instead of painstakingly building a ``model parallel'' or ``data parallel'' implementation, a user of Einsummable needs only build their computation in our Einsummable language. To make Einsummable a reality, we designed an Einsummable language for users to interact with, to create what we call EinGraphs. The Einsummable language is built on the extended Einstein summation notation, familiar to many ML practitioners. Our language is expressive enough to represent state of the art generative ML models, including Llama. In addition, we support automatic differentiation. On the other end of the abstraction spectrum, we created a compute graph specification for machines to execute, called TaskGraphs. TaskGraphs are designed to be executed by distributed, asynchronous compute engines. For our experiments, we built a distributed CPU execution engine, scaling to 32 machines, each with 64 processors. Even though we targeted CPU clusters, the TaskGraph abstraction is also suitable for clusters of GPUs. Most importantly, given hardware parameters, we compile EinGraphs into \\ TaskGraphs without user intervention. The discovered TaskGraph solution may very well include the common model or data parallel solutions. Our main algorithm for this is called EinDecomp, which decomposes EinGraphs so that the computation exposes enough parallelism to keep all processors busy without also introducing undue communication burden.en_US
dc.format.mimetypeapplication/pdfen_US
dc.identifier.citationBourgeois, Daniel Christopher. Declarative Machine Learning with Einsummable. (2024). PhD diss., Rice University. https://hdl.handle.net/1911/117775en_US
dc.identifier.urihttps://hdl.handle.net/1911/117775en_US
dc.language.isoengen_US
dc.rightsCopyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.en_US
dc.subjectMachine Learningen_US
dc.subjectDeep Learningen_US
dc.subjectAutomatic Parallelismen_US
dc.subjectDistributed Computingen_US
dc.titleDeclarative Machine Learning with Einsummableen_US
dc.typeThesisen_US
dc.type.materialTexten_US
thesis.degree.departmentStatisticsen_US
thesis.degree.disciplineEngineeringen_US
thesis.degree.grantorRice Universityen_US
thesis.degree.levelDoctoralen_US
thesis.degree.nameDoctor of Philosophyen_US
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
BOURGEOIS-DOCUMENT-2024.pdf
Size:
2.28 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
5.85 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
LICENSE.txt
Size:
2.98 KB
Format:
Plain Text
Description:
Download
Collections
Rice University Theses and Dissertations