Consistency of trace norm minimization

Francis Bach INRIA - Willow project D´epartement d’Informatique, Ecole Normale Sup´erieure 45, rue d’Ulm 75230 Paris, France

Abstract Regularization by the sum of singular values, also referred to as the trace norm, is a popular technique for estimating low rank rectangular matrices. In this paper, we extend some of the consistency results of the Lasso to provide necessary and sufficient conditions for rank consistency of trace norm minimization with the square loss. We also provide an adaptive version that is rank consistent even when the necessary condition for the non adaptive version is not fulfilled.

1. Introduction

[email protected]

tor [Yuan and Lin, 2007, Zhao and Yu, 2006, Zou, 2006]. When learning on rectangular matrices, the rank is a natural extension of the cardinality, and the sum of singular values, also known as the trace norm or the nuclear norm, is the natural extension of the ℓ1 -norm; indeed, as the ℓ1 -norm is the convex envelope of the ℓ0 -norm on the unit ball (i.e., the largest lower bounding convex function) [Boyd and Vandenberghe, 2003], the trace norm is the convex envelope of the rank over the unit ball of the spectral norm [Fazel et al., 2001]. In practice, it leads to low rank solutions [Fazel et al., 2001, Srebro et al., 2005] and has seen recent increased interest in the context of collaborative filtering [Srebro et al., 2005], multi-task learning [Abernethy et al., 2006, Argyriou et al., 2007] or classification with multiple classes [Amit et al., 2007].

In recent years, regularization by various non Euclidean norms has seen considerable interest. In particular, in the context of linear supervised learning, norms such as the ℓ1 -norm may induce sparse loading vectors, i.e., loading vectors with low cardinality or ℓ0 -norm. Such regularization schemes, also known as the Lasso [Tibshirani, 1994] for least-square regression, come with efficient path following algorithms [Efron et al., 2004]. Moreover, recent work has studied conditions under which such procedures consistently estimate the sparsity pattern of the loading vec-

In this paper, we consider the rank consistency of trace norm regularization with the square loss, i.e., if the data were actually generated by a lowrank matrix, will the matrix and its rank be consistently estimated? We provide necessary and sufficient conditions for the rank consistency that are extensions of corresponding results for the Lasso [Yuan and Lin, 2007, Zhao and Yu, 2006, Zou, 2006] and the group Lasso [Bach, 2007a]. We do so under two sets of sampling assumptions detailed in the full paper: a full i.i.d assumption and a non i.i.d assumption which is natural in the context of collaborative filtering.

Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute.

As for the Lasso and the group Lasso, the necessary condition implies that such procedures do

Consistency of trace norm minimization

not always estimate the rank correctly; following the adaptive version of the Lasso and group Lasso [Zou, 2006], we design an adaptive version to achieve n−1/2 -consistency and rank consistency, with no consistency conditions. Finally, we present a smoothing approach to convex optimization with the trace norm, and we show simulations on toy examples to illustrate the consistency results. The full paper (submitted to JMLR) [Bach, 2007b] may be downloaded from http://hal. archives-ouvertes.fr/hal-00179522

References J. Abernethy, F. Bach, T. Evgeniou, and J.-P. Vert. Low-rank matrix factorization with attributes. Technical Report N24/06/MM, Ecole des Mines de Paris, 2006. Y. Amit, M. Fink, N. Srebro, and S. Ullman. Uncovering shared structures in multiclass classification. In Proceedings of the Twenty-fourth International Conference on Machine Learning, 2007. A. Argyriou, T. Evgeniou, and M. Pontil. Multitask feature learning. In Adv. NIPS 19, 2007. F. Bach. Consistency of the group lasso and multiple kernel learning. Technical Report HAL00164735, HAL, 2007a. F. Bach. Consistency of trace norm minimization. Technical Report HAL-00179522, HAL, 2007b. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge Univ. Press, 2003. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32:407, 2004. M. Fazel, H. Hindi, and S. P. Boyd. A rank minimization heuristic with application to minimum order system approximation. In Proceedings American Control Conference, volume 6, pages 4734–4739, 2001. N. Srebro, J. D. M. Rennie, and T. S. Jaakkola. Maximum-margin matrix factorization. In Adv. NIPS 17, 2005. R. Tibshirani. Regression shrinkage and selection

via the lasso. Journal Royal Statististics, 58(1): 267–288, 1994. M. Yuan and Y. Lin. On the non-negative garrotte estimator. Journal of The Royal Statistical Society Series B, 69(2):143–161, 2007. P. Zhao and B. Yu. On model selection consistency of lasso. Journal of Machine Learning Research, 7:2541–2563, 2006. H. Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101:1418–1429, December 2006.

Consistency of trace norm minimization

learning, norms such as the ℓ1-norm may induce ... When learning on rectangular matrices, the rank ... Technical Report HAL-00179522, HAL, 2007b. S. Boyd ...

47KB Sizes 4 Downloads 305 Views

Recommend Documents

Consistency of trace norm minimization
and a non i.i.d assumption which is natural in the context of collaborative filtering. As for the Lasso and the group Lasso, the nec- essary condition implies that ...

Consistency of trace norm minimization Francis Bach
Trace norm - optimization problem. • n observations (Mi,zi), i = 1,...,n, where zi ∈ R and Mi ∈ R p×q. • Optimization problem on W ∈ R p×q. : min. W∈R p×q. 1.

TRACE FOSSILS
Smooth the surface of the sediment using a flat piece of plastic or wood; a cheap ruler should work well. 4. Allow the animal to run, walk, or crawl across the ...

Norm Hord.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. Norm Hord.pdf.

socializing consistency
often rather interact with a person than a machine: Virtual people may represent a ..... (Cook, 2000), an active topic of discussion as telephone-based call.

socializing consistency
demonstrates that as interfaces become more social, social consistency .... action with any number of such complex beings on a daily basis. .... media (stereotypically gender neutral), and computers (stereotypically male) ... In line with predictions

Childrens Tattling: The Reporting of Everyday Norm ... - Jesse Bering
refusing to share objects that they themselves owned or were using. A report of ..... the federal civil service: New evidence of the public service ethic. Journal of ...

PDF Books Norm Of The North
... Skills for improved productivity employment growth and development iv Chapter 5 Skills policies as drivers of development Nonlinear Dynamics and Statistical Physics focuses on both fundamental and applied problems involving interacting many body

Fair Simulation Minimization - Springer Link
Any savings obtained on the automaton are therefore amplified by the size of the ... tions [10] that account for the acceptance conditions of the automata. ...... open issue of extending our approach to generalized Büchi automata, that is, to.

Consistency of individual differences in behaviour of the lion-headed ...
1999 Elsevier Science B.V. All rights reserved. Keywords: Aggression .... the data analysis: Spearman rank correlation co- efficient with exact P values based on ...

Consistency Without Borders
Distributed consistency is a perennial research topic; in recent years it has become an urgent practical matter as well. The research literature has focused on enforcing various flavors of consistency at the I/O layer, such as linearizability of read

Distributed Asymptotic Minimization of Sequences of ...
Feb 21, 2012 - planning, acoustic source localization, and environmental modeling). ... steps. First, to improve the estimate of a minimizer, agents apply a ...... Dr. Cavalcante received the Excellent Paper Award from the IEICE in 2006 and the ...

Minimization of thin film contact resistance - MSU College of Engineering
Nov 19, 2010 - lack of analytical scaling that readily gives an explicit evalu- ation of thin .... our Fourier representation data reveal that the sole depen- dence of ...

Cyclone Trace Supplement.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. Cyclone Trace ...

On the Consistency of Deferred Acceptance when Priorities are ... - Csic
Roth, A.E., and M.A.O. Sotomayor (1990): Two-Sided Matching: A Study in Game-Theoretic. Modeling and Analysis. Econometric Society Monograph Series.

The Basis of Consistency Effects in Word Naming
Kenseidenberg. Mark S Journal of Memory and Language; Dec 1, 1990; 29, 6; Periodicals Archive Online pg. 637 ..... (consistent vs. inconsistent) and frequency.

condorcet consistency of approval voting: a counter ...
mine voters' best responses and hence the equilibria of the game. Myerson .... component bğcŞ of vector b accounts for the number of voters who vote for bal- lot c. .... Notice that the magnitude of an outcome must be inferior or equal to zero,.

On the Consistency of Deferred Acceptance when ... - Semantic Scholar
An allocation µ Pareto dominates another allocation µ′ at R if µiRiµ′ ... at (R,Ch). Since Ch is substitutable, the so-called deferred acceptance rule, denoted ...

On the Consistency of Deferred Acceptance when ...
There is a set of agents N and a set of proper object types O. There is also a null object ... An allocation is a vector µ = (µi)i∈N assigning object µi ∈ O ∪ {∅} to.

Client-centric benchmarking of eventual consistency for cloud storage ...
Client-centric benchmarking of eventual consistency for cloud storage systems. Wojciech Golab1, Muntasir Raihan Rahman2, Alvin AuYoung3,. Kimberly Keeton3, Jay J. ... J. López, G. Gibson, A. Fuchs, and B. Rinaldi. YCSB++: benchmarking and performanc