Browsing by Author "Yin, Wotao"
Now showing 1 - 20 of 56
Results Per Page
Sort Options
Item A Block Coordinate Descent Method for Multi-Convex Optimization with Applications to Nonnegative Tensor Factorization and Completion(2012-08) Xu, Yangyang; Yin, WotaoThis paper considers block multi-convex optimization, where the feasible set and objective function are generally non-convex but convex in each block of variables. We review some of its interesting examples and propose a generalized block coordinate descent method. Under certain conditions, we show that any limit point satisfies the Nash equilibrium conditions. Furthermore, we establish its global convergence and estimate its asymptotic convergence rate by assuming a property based on the Kurdyka-Lojasiewicz inequality. The proposed algorithms are adapted for factorizing nonnegative matrices and tensors, as well as completing them from their incomplete observations. The algorithms were tested on synthetic data, hyperspectral data, as well as image sets from the CBCL and ORL databases. Compared to the existing state-of-the-art algorithms, the proposed algorithms demonstrate superior performance in both speed and solution quality. The Matlab code is available for download from the authors' homepages.Item A Comparison of Three Total Variation Based Texture Extraction Models(2007-01) Yin, Wotao; Goldfarb, Donald; Osher, StanleyThis paper qualitatively compares three recently proposed models for signal/image texture extraction based on total variation minimization:the Meyer, Vese-Osher, and TV-L1 models. We formulate discrete versions of these models as second-order cone programs (SOCPs) which can be solved efficiently by interior-point methods. Our experiments with these models on 1D oscillating signals and 2D images reveal their differences: the Meyer model tends to extract oscillation patterns in the input, the TV-L1 model performs a strict multiscale decomposition, and the Vese-Osher model has properties falling in between the other two models.Item A Curvilinear Search Method for p-Harmonic Flows on Spheres(2008-01) Goldfarb, Donald; Wen, Zaiwen; Yin, WotaoThe problem of finding p-harmonic flows arises in a wide range of applications including micromagnetics, liquid crystal theory, directional diffusion, and chromaticity denoising. In this paper, we propose an innovative curvilinear search method for minimizing p-harmonic energies over spheres. Starting from a flow (map) on the unit sphere, our method searches along a curve that lies on the sphere in a manner similar to a standard inexact line search descent method. We show that our method is globally convergent if the step length satisfies the Armijo-Wolfe conditions. Computational tests are presented to demonstrate the efficiency of the proposed method and a variant of it that uses Barzilai-Borwein steps.Item A Fast Algorithm for Edge-Preserving Variational Multichannel Image Restoration(2008-07) Yang, Junfeng; Yin, Wotao; Zhang, Yin; Wang, YilunWe generalize the alternating minimization algorithm recently proposed in [32] to effciently solve a general, edge-preserving, variational model for recovering multichannel images degraded by within- and cross-channel blurs, as well as additive Gaussian noise. This general model allows the use of localized weights and higher-order derivatives in regularization, and includes a multichannel extension of total variation (MTV) regularization as a special case. In the MTV case, we show that the model can be derived from an extended half-quadratic transform of Geman and Yang [14]. For color images with three channels and when applied to the MTV model (either locally weighted or not), the per-iteration computational complexity of this algorithm is dominated by nine fast Fourier transforms. We establish strong convergence results for the algorithm including finite convergence for some variables and fastᅠq-linear convergence for the others. Numerical results on various types of blurs are presented to demonstrate the performance of our algorithm compared to that of the MATLAB deblurring functions. We also present experimental results on regularization models using weighted MTV and higher-order derivatives to demonstrate improvements in image quality provided by these models over the plain MTV model.Item A Fast TVL1-L2 Minimization Algorithm for Signal Reconstruction from Partial Fourier Data(2008-10) Yang, Junfeng; Zhang, Yin; Yin, WotaoRecent compressive sensing results show that it is possible to accurately reconstruct certain compressible signals from relatively few linear measurements via solving nonsmooth convex optimization problems. In this paper, we propose a simple and fast algorithm for signal reconstruction from partial Fourier data. The algorithm minimizes the sum of three terms corresponding to total variation, $\ell_1$-norm regularization and least squares data fitting. It uses an alternating minimization scheme in which the main computation involves shrinkage and fast Fourier transforms (FFTs), or alternatively discrete cosine transforms (DCTs) when available data are in the DCT domain. We analyze the convergence properties of this algorithm, and compare its numerical performance with two recently proposed algorithms. Our numerical simulations on recovering magnetic resonance images (MRI) indicate that the proposed algorithm is highly efficient, stable and robust.Item A Feasible Method for Optimization with Orthogonality Constraints(2010-11) Wen, Zaiwen; Yin, WotaoMinimization with orthogonality constraints (e.g., X'X = I) and/or spherical constraints (e.g., ||x||_2 = 1) has wide applications in polynomial optimization, combinatorial optimization, eigenvalue problems, sparse PCA, p-harmonic flows, 1-bit compressive sensing, matrix rank minimization, etc. These problems are difficult because the constraints are not only non-convex but numerically expensive to preserve during iterations. To deal with these difficulties, we propose to use a Crank-Nicholson-like update scheme to preserve the constraints and based on it, develop curvilinear search algorithms with lower per-iteration cost compared to those based on projections and geodesics. The efficiency of the proposed algorithms is demonstrated on a variety of test problems. In particular, for the maxcut problem, it exactly solves a decomposition formulation for the SDP relaxation. For polynomial optimization, nearest correlation matrix estimation and extreme eigenvalue problems, the proposed algorithms run very fast and return solutions no worse than those from their state-of-the-art algorithms. For the quadratic assignment problem, a gap 0.842% to the best known solution on the largest problem "256c" in QAPLIB can be reached in 5 minutes on a typical laptop.Item A Fixed-Point Continuation Method for L_1-Regularization with Application to Compressed Sensing(2007-05) Hale, Elaine T.; Yin, Wotao; Zhang, YinWe consider solving minimization problems with L_1-regularization: min ||x||_1 + mu f(x) particularly for f(x) = (1/2)||Ax-b||M2, where A is m by n and m < n. Our goal is to construct efficient and robust algorithms for solving large-scale problems with dense data, and our approach is based on two powerful algorithmic ideas: operator-splitting and continuation. This paper establishes q-linear convergence rates for our algorithm applied to problems with f(x) convex, but not necessarily strictly convex. We present numerical results for several types of compressed sensing problems, and show that our algorithm compares favorably with three state-of-the-art algorithms when applied to large-scale problems with noisy data.Item A Matlab Implementation of a Flat Norm Motivated Polygonal Edge Matching Method using a Decomposition of Boundary into Four 1-Dimensional Currents(2009-09) Morgan, Simon; Yin, Wotao; Vixie, KevinWe describe and provide code and examples for a polygonal edge matching method.Item A New Alternating Minimization Algorithm for Total Variation Image Reconstruction(2007-06) Wang, Yilun; Yang, Junfeng; Yin, Wotao; Zhang, YinWe propose, analyze and test an alternating minimization algorithm for recovering images from blurry and noisy observa- tions with total variation (TV) regularization. This algorithm arises from a new half-quadratic model applicable to not only the anisotropic but also isotropic forms of total variation discretizations. The per-iteration computational complexity of the algorithm is three Fast Fourier Transforms (FFTs). We establish strong convergence properties for the algorithm including finite convergence for some variables and relatively fast exponential (or q-linear in optimization terminology) convergence for the others. Furthermore, we propose a continuation scheme to accelerate the practical convergence of the algorithm. Extensive numerical results show that our algorithm performs favorably in comparison to several state-of-the-art algorithms. In particular, it runs orders of magnitude faster than the Lagged Diffusivity algorithm for total-variation-based deblurring. Some extensions of our algorithm are also discussed.Item A Rapid and Robust Numerical Algorithm for Sensitivity Encoding with Sparsity Constraints: Self-Feeding Sparse SENSE(2010-11) Huang, Feng; Chen, Yunmei; Yin, Wotao; Lin, Wei; Ye, Xiaojing; Guo, Weihong; Reykowski, ArneThe method of enforcing sparsity during magnetic resonance imaging reconstruction has been successfully applied to partially parallel imaging (PPI) techniques to reduce noise and artifact levels and hence to achieve even higher acceleration factors. However, there are two major problems in the existing sparsity-constrained PPI techniques: speed and robustness. By introducing an auxiliary variable and decomposing the original minimization problem into two subproblems that are much easier to solve, a fast and robust numerical algorithm for sparsity-constrained PPI technique is developed in this work. The specific implementation for a conventional Cartesian trajectory data set is named self-feeding Sparse Sensitivity Encoding (SENSE). The computational cost for the proposed method is two conventional SENSE reconstructions plus one spatially adaptive image denoising procedure. With reconstruction time approximately doubled, images with a much lower root mean square error (RMSE) can be achieved at high acceleration factors. Using a standard eight-channel head coil, a net acceleration factor of 5 along one dimension can be achieved with low RMSE. Furthermore, the algorithm is insensitive to the choice of parameters. This work improves the clinical applicability of SENSE at high acceleration factors.Item Accelerated High-Performance Compressive Sensing using the Graphics Processing Unit(2011) Reyna, Nabor; Yin, WotaoThis thesis demonstrates the advantages of new practical implementations of compressive sensing (CS) algorithms tailored for the graphics processing unit (CPU) using a software platform called Jacket. There exist many applications which utilize CS including medical imaging, signal processing and data acquisition which have benefited from advancements in CS. However, as problems become larger not only do they become more difficult to solve but also more computationally expensive. In light of tins, existing CS algorithms are augmented for practical use on the CPU, reaping performance gains from the highly parallel architecture of the GPU. I discuss the issues associated with this transition and analyze the effects of such a movement, as well as provide results exhibiting advantages of using CPU-based methods.Item Algorithms to Find the Girth and Cogirth of a Linear Matroid(2014-09-18) Arellano, John David; Hicks, Illya V; Tapia, Richard A; Yin, Wotao; Baraniuk, Richard GIn this thesis, I present algorithms to find the cogirth and girth, the cardinality of the smallest cocircuit and circuit respectively, of a linear matroid. A set covering problem (SCP) formulation of the problems is presented. The solution to the linear matroid cogirth problem provides the degree of redundancy of the corresponding sensor network, and allows for the evaluation of the quality of the network. Hence, addressing the linear matroid cogirth problem can lead to significantly enhancing the design process of sensor networks. The linear matroid girth problem is related to reconstructing a signal in compressive sensing. I provide an introduction to matroids and their relation to the degree of redundancy problem as well as compressive sensing. I also provide an overview of the methods used to address linear matroid cogirth/girth problems, the SCP, and reconstructing a signal in compressive sensing. Computational results are provided to validate a branch-and-cut algorithm that addresses the SCP formulation as well as an algorithm which uses branch decompositions and dynamic programming to find the girth of a linear matroid.Item Alternating Direction Augmented Lagrangian Methods for Semidefinite Programming(2009-12) Wen, Zaiwen; Goldfarb, Donald; Yin, WotaoWe present an alternating direction method based on an augmented Lagrangian framework for solving semidefinite programming (SDP) problems in standard form. At each iteration, the algorithm, also known as a two-splitting scheme, minimizes the dual augmented Lagrangian function sequentially with respect to the Lagrange multipliers corresponding to the linear constraints, then the dual slack variables and finally the primal variables, while in each minimization keeping the other variables fixed. Convergence is proved by using a fixed-point argument. A multiple-splitting algorithm is then proposed to handle SDPs with inequality constraints and positivity constraints directly without transforming them to the equality constraints in standard form. Finally, numerical results for frequency assignment, maximum stable set and binary integer quadratic programming problems are presented to demonstrate the robustness and efficiency of our algorithm.Item An Alternating Direction Algorithm for Matrix Completion with Nonnegative Factors(2011-01) Xu, Yangyang; Yin, Wotao; Wen, Zaiwen; Zhang, YinThis paper introduces a novel algorithm for the nonnegative matrix factorization and completion problem, which aims to nd nonnegative matrices X and Y from a subset of entries of a nonnegative matrix M so that XY approximates M. This problem is closely related to the two existing problems: nonnegative matrix factorization and low-rank matrix completion, in the sense that it kills two birds with one stone. As it takes advantages of both nonnegativity and low rank, its results can be superior than those of the two problems alone. Our algorithm is applied to minimizing a non-convex constrained least-squares formulation and is based on the classic alternating direction augmented Lagrangian method. Preliminary convergence properties and numerical simulation results are presented. Compared to a recent algorithm for nonnegative random matrix factorization, the proposed algorithm yields comparable factorization through accessing only half of the matrix entries. On tasks of recovering incomplete grayscale and hyperspectral images, the results of the proposed algorithm have overall better qualities than those of two recent algorithms for matrix completion.Item An Efficient Augmented Lagrangian Method with Applications to Total Variation Minimization(2012-07) Li, Chengbo; Yin, Wotao; Jiang, Hong; Zhang, YinBased on the classic augmented Lagrangian multiplier method, we propose, analyze and test an algorithm for solving a class of equality-constrained non-smooth optimization problems (chiefly but not necessarily convex programs) with a particular structure. The algorithm effectively combines an alternating direction technique with a nonmonotone line search to minimize the augmented Lagrangian function at each iteration. We establish convergence for this algorithm, and apply it to solving problems in image reconstruction with total variation regularization. We present numerical results showing that the resulting solver, called TVAL3, is competitive with, and often outperforms, other state-of-the-art solvers in the field.Item Augmented L1 and Nuclear-Norm Models with a Globally Linearly Convergent Algorithm(2012-01) Lai, Ming-Jun; Yin, WotaoThis paper studies the models of minimizing $||x||_1+1/(2\alpha)||x||_2^2$ where $x$ is a vector, as well as those of minimizing $||X||_*+1/(2\alpha)||X||_F^2$ where $X$ is a matrix and $||X||_*$ and $||X||_F$ are the nuclear and Frobenius norms of $X$, respectively. We show that they can efficiently recover sparse vectors and low-rank matrices. In particular, they enjoy exact and stable recovery guarantees similar to those known for minimizing $||x||_1$ and $||X||_*$ under the conditions on the sensing operator such as its null-space property, restricted isometry property, spherical section property, or RIPless property. To recover a (nearly) sparse vector $x^0$, minimizing $||x||_1+1/(2\alpha)||x||_2^2$ returns (nearly) the same solution as minimizing $||x||_1$ almost whenever $\alpha\ge 10||x^0||_\infty$. The same relation also holds between minimizing $||X||_*+1/(2\alpha)||X||_F^2$ and minimizing $||X||_*$ for recovering a (nearly) low-rank matrix $X^0$, if $\alpha\ge 10||X^0||_2$. Furthermore, we show that the linearized Bregman algorithm for minimizing $||x||_1+1/(2\alpha)||x||_2^2$ subject to $Ax=b$ enjoys global linear convergence as long as a nonzero solution exists, and we give an explicit rate of convergence. The convergence property does not require a solution solution or any properties on $A$. To our knowledge, this is the best known global convergence result for first-order sparse optimization algorithms.Item Autofocus for Synthetic Aperture Radar(2013-06-05) Gallardo Palacios, Ricardo; Borcea, Liliana; Symes, William W.; Yin, WotaoIn this thesis, I compare the performance of three different autofocus techniques for Synthetic Aperture Radar (SAR). The focusing is done by estimating phase errors in SAR data. The first one, the Phase Gradient Autofocus, is the most popular in the industry, it has been around for more than 20 years and it relies on the redundancy of the phase error in the SAR images. The second one, the Entropy-based minimization, uses measurements of image sharpness to focus the images and it has been available for about 10 years. The last, the Phase-space method, uses the Wigner transform and the ambiguity function of the SAR data to estimate the phase perturbations and it was recently introduced. Additionally, I develop a criteria for filtering the data for the cases in which the Phase-space method does not capture the entirety of the error.Item Block Coordinate Descent for Regularized Multi-convex Optimization(2013-09-16) Xu, Yangyang; Yin, Wotao; Tapia, Richard A.; Zhang, Yin; Baraniuk, Richard G.This thesis considers regularized block multi-convex optimization, where the feasible set and objective function are generally non-convex but convex in each block of variables. I review some of its interesting examples and propose a generalized block coordinate descent (BCD) method. The generalized BCD uses three different block-update schemes. Based on the property of one block subproblem, one can freely choose one of the three schemes to update the corresponding block of variables. Appropriate choices of block-update schemes can often speed up the algorithm and greatly save computing time. Under certain conditions, I show that any limit point satisfies the Nash equilibrium conditions. Furthermore, I establish its global convergence and estimate its asymptotic convergence rate by assuming a property based on the Kurdyka-{\L}ojasiewicz inequality. As a consequence, this thesis gives a global linear convergence result of cyclic block coordinate descent for strongly convex optimization. The proposed algorithms are adapted for factorizing nonnegative matrices and tensors, as well as completing them from their incomplete observations. The algorithms were tested on synthetic data, hyperspectral data, as well as image sets from the CBCL, ORL and Swimmer databases. Compared to the existing state-of-the-art algorithms, the proposed algorithms demonstrate superior performance in both speed and solution quality.Item Block Coordinate Update Method in Tensor Optimization(2014-08-19) Xu, Yangyang; Yin, Wotao; Zhang, Yin; Allen, Genevera; Tapia, RichardBlock alternating minimization (BAM) has been popularly used since the 50's of last century. It partitions the variables into disjoint blocks and cyclically updates the blocks by minimizing the objective with respect to each block of variables, one at a time with all others fixed. A special case is the alternating projection method to find a common point of two convex sets. The BAM method is often easy yet efficient particularly if each block subproblem is simple to solve. However, for certain problems such as the nonnegative tensor decomposition, the block subproblems can be difficult to solve, or even if they are solved exactly or to high accuracies, BAM can perform badly on solving the original problem, in particular on non-convex problems. On the other hand, in the literature, the BAM method is mainly analyzed for convex problems. Although it has been shown numerically to work well on many non-convex problems, theoretical results of BAM for non-convex optimization are still lacked. For these reasons, I propose different block update schemes and generalize the BAM method for non-smooth non-convex optimization problems. Which scheme is the most efficient depends on specific applications. In addition, I analyze convergence of the generalized method, dubbed as block coordinate update method (BCU), with different block update schemes for non-smooth optimization problems, in both convex and non-convex cases. BCU has found many applications, and the work in this dissertation is mainly motivated by tensor optimization problems, for which the BCU method is often the best choice due to their block convexity. I make contributions in modeling, algorithm design, and also theoretical analysis. The first part is about the low-rank tensor completion, for which I make a novel model based on parallel low-rank matrix factorization. The new model is non-convex, and it is difficult to guarantee global optimal solutions. However, the BAM method performs very well on solving this model. Global convergence in terms of KKT conditions is established, and numerical experiments demonstrate the superiority of the proposed model over several state-of-the-art ones. The second part is towards the solution of the nonnegative tensor decomposition. For this problem, each block subproblem is a nonnegative least squares problem and not simple to solve. Hence, the BAM method may be inefficient. I propose a block proximal gradient (BPG) method. In contrast to BAM that solves each block subproblem exactly, BPG solves relaxed block subproblems, which are often much simpler than the original ones and can thus make BPG converge faster. Through the Kurdyka-Lojasiewicz property, I establish its global convergence with rate estimate in terms of iterate sequence. Numerical experiments on sparse nonnegative Tucker decomposition demonstrates its superiority over the BAM method. The last part is motivated by tensor regression problems, whose block partial gradient is expensive to evaluate. For such problems, BPG becomes inefficient, and I propose to use inexact partial gradient and generalize BPG to a block stochastic gradient method. Convergence results in expectation are established for general non-convex case in terms of first-order optimality conditions, and for convex case, a sublinear convergence rate result is shown. Numerical tests on tensor regression problems show that the block stochastic gradient method significantly outperforms its deterministic counterpart.Item Block Stochastic Gradient Iteration for Convex and Noncovex Optimization(2014-08) Xu, Yangyang; Yin, WotaoThe stochastic gradient (SG) method can minimize an objective function composed of a large number of differentiable functions or solve a stochastic optimization problem, very quickly to a moderate accuracy. The block coordinate descent/update (BCD) method, on the other hand, handles problems with multiple blocks of variables by updating them one at a time; when the blocks of variables are (much) easier to update individually than together, BCD has a (much) lower per-iteration cost. This paper introduces a method that combines the great features of SG and BCD for problems with many components in the objective and with multiple (blocks of) variables. Specifocally, a block stochastic gradient (BSG) method is proposed for both convex and nonconvex programs. At each iteration, BSG approximates the gradient of the differentiable part of the objective by randomly sampling a small set of data or sampling a few functions in the objective, and then, using the approximate gradient, it updates all the blocks of variables in either a deterministic or a randomly shuffed order. Its convergence for convex and nonconvex cases is established in different senses. In the convex case, the proposed method has the same order of convergence rate as the SG method. In the nonconvex case, its convergence is established in terms of the expected violation of a first-order optimality condition. The proposed method was numerically tested on problems including stochastic least squares and logistic regression, which are convex, as well as low-rank tensor recovery and bilinear logistic regression, which are nonconvex. On the convex problems, it performed as well as, and often significantly better, than the SG method. On the nonconvex problems, the proposed method BSG significantly outperformed the deterministic BCD method because the latter tends to slow down or stagnate near bad local minimizers. Overall, BSG inherits the benefits of both stochastic gradient approximation and block-coordinate updates.
- «
- 1 (current)
- 2
- 3
- »