Max-Affine Splines Insights Into Deep Learning

Simple item page
dc.contributor.advisorBaraniuk, Richard G.
dc.creatorBalestriero, Randall
dc.date.accessioned2021-05-03T21:43:27Z
dc.date.available2021-05-03T21:43:27Z
dc.date.created2021-05
dc.date.issued2021-04-28
dc.date.submittedMay 2021
dc.date.updated2021-05-03T21:43:27Z
dc.description.abstractWe build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs), which provide a powerful portal through which to view and analyze their inner workings. For instance, conditioned on the spline partition region containing the input signal, the output of a MASO DN can be written as a simple affine transformation of the input. Studying the geometry of those regions allows to obtain novel insights into different regularization techniques, different layer configurations or different initialization schemes. Going further, this spline viewpoint allows to obtain precise geometric insights in various domains such as the characterization of the Deep Generative Networks's generated manifold, the understanding of Deep Network pruning as a mean to simplify the DN input space partition or the relationship between different nonlinearities e.g. ReLU-Sigmoid Gated Linear Unit as simply corresponding to different MASO region membership inference algorithms. The spline partition of the input signal space that is implicitly induced by a MASO directly links DNs to the theory of vector quantization (VQ) and $K$-means clustering, which opens up new geometric avenues to study how DNs organize signals in a hierarchical fashion.
dc.format.mimetypeapplication/pdf
dc.identifier.citationBalestriero, Randall. "Max-Affine Splines Insights Into Deep Learning." (2021) Diss., Rice University. <a href="https://hdl.handle.net/1911/110439">https://hdl.handle.net/1911/110439</a>.
dc.identifier.urihttps://hdl.handle.net/1911/110439
dc.language.isoeng
dc.rightsCopyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.
dc.subjectdeep learning
dc.subjectdeep networks
dc.subjectaffine splines
dc.titleMax-Affine Splines Insights Into Deep Learning
dc.typeThesis
dc.type.materialText
thesis.degree.departmentElectrical and Computer Engineering
thesis.degree.disciplineEngineering
thesis.degree.grantorRice University
thesis.degree.levelDoctoral
thesis.degree.nameDoctor of Philosophy
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
BALESTRIERO-DOCUMENT-2021.pdf
Size:
7.82 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
5.85 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
LICENSE.txt
Size:
2.61 KB
Format:
Plain Text
Description:
Download
Collections
Rice University Electronic Theses and Dissertations