Ridge Regularization by Randomization in Linear Ensembles
| dc.contributor.advisor | Baraniuk, Richard G | en_US |
| dc.creator | LeJeune, Daniel | en_US |
| dc.date.accessioned | 2023-01-03T21:20:27Z | en_US |
| dc.date.available | 2023-01-03T21:20:27Z | en_US |
| dc.date.created | 2022-12 | en_US |
| dc.date.issued | 2022-11-21 | en_US |
| dc.date.submitted | December 2022 | en_US |
| dc.date.updated | 2023-01-03T21:20:27Z | en_US |
| dc.description.abstract | Ensemble methods that average over a collection of independent predictors that are each limited to random sampling of both the examples and features of the training data command a significant presence in machine learning, such as the ever-popular random forest. Combining many such randomized predictors into an ensemble produces a highly robust predictor with excellent generalization properties; however, understanding the specific nature of the effect of randomization on ensemble method behavior has received little theoretical attention. We study the case of an ensembles of linear predictors, where each individual predictor is a linear predictor fit on a randomized sample of the data matrix. We first show a straightforward argument that an ensemble of ordinary least squares predictors fit on a simple subsampling can achieve the optimal ridge regression risk in a standard Gaussian data setting. We then significantly generalize this result to eliminate essentially all assumptions on the data by considering ensembles of linear random projections or sketches of the data, and in doing so reveal an asymptotic first-order equivalence between linear regression on sketched data and ridge regression. By extending this analysis to a second-order characterization, we show how large ensembles converge to ridge regression under quadratic metrics. | en_US |
| dc.format.mimetype | application/pdf | en_US |
| dc.identifier.citation | LeJeune, Daniel. "Ridge Regularization by Randomization in Linear Ensembles." (2022) Diss., Rice University. <a href="https://hdl.handle.net/1911/114188">https://hdl.handle.net/1911/114188</a>. | en_US |
| dc.identifier.uri | https://hdl.handle.net/1911/114188 | en_US |
| dc.language.iso | eng | en_US |
| dc.rights | Copyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder. | en_US |
| dc.subject | ensembles | en_US |
| dc.subject | ridge regression | en_US |
| dc.subject | sketching | en_US |
| dc.subject | random projections | en_US |
| dc.subject | proportional asymptotics | en_US |
| dc.subject | random matrix theory | en_US |
| dc.title | Ridge Regularization by Randomization in Linear Ensembles | en_US |
| dc.type | Thesis | en_US |
| dc.type.material | Text | en_US |
| thesis.degree.department | Electrical and Computer Engineering | en_US |
| thesis.degree.discipline | Engineering | en_US |
| thesis.degree.grantor | Rice University | en_US |
| thesis.degree.level | Doctoral | en_US |
| thesis.degree.name | Doctor of Philosophy | en_US |
Files
Original bundle
1 - 1 of 1