Max-Affine Splines Insights Into Deep Learning

Simple item page
dc.contributor.advisorBaraniuk, Richard G.en_US
dc.creatorBalestriero, Randallen_US
dc.date.accessioned2021-05-03T21:43:27Zen_US
dc.date.available2021-05-03T21:43:27Zen_US
dc.date.created2021-05en_US
dc.date.issued2021-04-28en_US
dc.date.submittedMay 2021en_US
dc.date.updated2021-05-03T21:43:27Zen_US
dc.description.abstractWe build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs), which provide a powerful portal through which to view and analyze their inner workings. For instance, conditioned on the spline partition region containing the input signal, the output of a MASO DN can be written as a simple affine transformation of the input. Studying the geometry of those regions allows to obtain novel insights into different regularization techniques, different layer configurations or different initialization schemes. Going further, this spline viewpoint allows to obtain precise geometric insights in various domains such as the characterization of the Deep Generative Networks's generated manifold, the understanding of Deep Network pruning as a mean to simplify the DN input space partition or the relationship between different nonlinearities e.g. ReLU-Sigmoid Gated Linear Unit as simply corresponding to different MASO region membership inference algorithms. The spline partition of the input signal space that is implicitly induced by a MASO directly links DNs to the theory of vector quantization (VQ) and $K$-means clustering, which opens up new geometric avenues to study how DNs organize signals in a hierarchical fashion.en_US
dc.format.mimetypeapplication/pdfen_US
dc.identifier.citationBalestriero, Randall. "Max-Affine Splines Insights Into Deep Learning." (2021) Diss., Rice University. <a href="https://hdl.handle.net/1911/110439">https://hdl.handle.net/1911/110439</a>.en_US
dc.identifier.urihttps://hdl.handle.net/1911/110439en_US
dc.language.isoengen_US
dc.rightsCopyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.en_US
dc.subjectdeep learningen_US
dc.subjectdeep networksen_US
dc.subjectaffine splinesen_US
dc.titleMax-Affine Splines Insights Into Deep Learningen_US
dc.typeThesisen_US
dc.type.materialTexten_US
thesis.degree.departmentElectrical and Computer Engineeringen_US
thesis.degree.disciplineEngineeringen_US
thesis.degree.grantorRice Universityen_US
thesis.degree.levelDoctoralen_US
thesis.degree.nameDoctor of Philosophyen_US
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
BALESTRIERO-DOCUMENT-2021.pdf
Size:
7.82 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
5.85 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
LICENSE.txt
Size:
2.61 KB
Format:
Plain Text
Description:
Download
Collections
Rice University Theses and Dissertations