A norm-concentration argument for non-convex regularisation

Ata Kab´ an Robert J. Durrant School of Computer Science The University of Birmingham Birmingham B15 2TT, UK

ICML/UAI/COLT Workshop on Sparse Optimization and Variable Selection Helsinki, 9 July 2008.

Introduction L1-regularisation - a workhorse in machine learning • sparsity • convexity • logarithmic sample complexity Non-convex norm regularisation - seems to have added value • statistics (Fan & Li, ’01): oracle property • signal proc. (Chartland, ’07), signal reconstruction (Wipf & Rao, ’05) • 0-norm SVM classification (Weston et al., ’03) (results data-dependent) • genomic data classification (Liu et al., ’07)

Regularised regression in high dimensions Training set {(xj , yj )}nj=1 , where xj ∈ Rm are m-dimensional inputs and yj ∈ {−1, 1} are their labels. Scenario of interest: few r << m relevant features, small sample size n << m. Consider regularised logistic regression for concreteness: max w

n X

log p(yj |xj , w) subject to ||w||q ≤ A

(1)

j=1

where w ∈ R1×m are unknown parameters, p(y|wT x) = 1/(1 + exp(−ywT x)), P q 1/q . and ||w||q = ( m i=1 |wi | ) If q = 2: L2-regularised (’ridge’) logistic regression. If q = 1: L1-regularised (’lasso’) logistic regression. If q < 1: Lq<1-regularised logistic regression: non-convex, non-differentiable at 0

A word on some recent estimation algorithms 2.5

2.0 2.0

1.5 1.5

1.0 1.0

0.5

0.5

K

4

K

2

0

w

i

2

4

K

4

K

2

0

w

2

4

i

Local quadratic (Fan & Li,’01) vs. local linear (Zou & Li,’08) bound, tangent at ±3. [Despite the latter appears to be a closer approximation, framing the iterative estimation into the E-M methodology framework, it turns out they are in fact equivalent (Kaban & Durrant, ECML’08)]

Sample complexity bound H = {h(x, y) = − log p(y|wT x) : x ∈ Rm , y ∈ {−1, 1}} the function class erP (h) = E(x,y)∼iidP [h(x, y)] the true error of h P er ˆ z (h) = n1 ni=1 h(xi , yi ) the sample error of h on training set z of size n optP (H) = inf h∈H erP (h) the approximation error of H

L(z) = minh∈H er ˆ z (h) function returned by the learning algorithm

Theorem (A.Ng,’04, extended from L1 to Lq<1). ∀ǫ > 0, ∀δ > 0, ∀m, n ≥ 1, in order to ensure that erP (L(z)) ≤ optP (H) + ǫ with probability 1 − δ, it is enough to have n = Ω((log m) × poly(A, r 1/q , 1/ǫ, log(1/δ)) - logarithmic in dimensionality m; - polynomial in #relevant features, but growing with r 1/q

(2)

test logloss

0−1 error

30 20

validation logloss

1.5

40

1 0.5

10 0.2

0.4

0.6 q

0.8

1

0.2

r=5

r=10

0.4

0.6 q r=30

0.8

1 r=50

1.4 1.2 1 0.8 0.6 0.4 0.2 0.2

0.4

0.6 q

0.8

r=100

Experiments on m = 200 dimensional data sets, varying the number of relevant features r ∈ {5, 10, 30, 50, 100}. The medians of 60 independent trials are shown and the error bars represent one standard error. The 0-1 errors are out of 100.

1

A norm concentration view Consider the un-regularised version of the problem. Because n << m, the system is under-determined, and so, m − n components of w can be set arbitrarily. We can model the arbitrary components of w as being i.i.d. Uniform: wi ∼ Unif[−a, a], ∀i ∈ {n + 1, ..., m} with some large a. The regularisation term is meant to constrain the problem to make it well-posed. However, in very high dimensions, a counter-intuitive phenomenon known as the concentration of distances and norms comes into play. The regularisation term becomes essentially the same for all the infinitely many possible maximisers of the likelihood term.

Distance concentration Distance concentration is the counter-intuitive phenomenon that, as the data dimensionality increases without bounds, all pairwise distances between points become identical. This phenomenon affects every area, where high-dimensional data processing is required — e.g. database indexing & retrieval, data analysis, statistical machine learning. Concentration of the L2-norm (Demartinez,’94) Let x ∈ Rm a random vector with i.i.d. components of any distribution. Then, lim

E[||x||2]

m→∞ m1/2

= const.;

lim Var[||x||2] = const.

m→∞

(3)

Concentration of arbitrary dissimilarity functions in arbitrary multivariate distributions (Beyer et al.,99). (m)

(m)

Let Fm , m = 1, 2, . . . be an infinite sequence of data distributions and x1 , . . . , xn a random sample of n independent data vectors distributed as Fm . For each m, let ||.|| : dom(Fm) → R+ be a function that takes a point from the domain of Fm and returns a positive real value. p > 0 an arbitrary positive constant Assume that E[||x(m) ||p ] and Var[||x(m) ||p ] are finite and E[||x(m) ||p ] 6= 0. Var[(||x(m) ||)p] = 0, then, If lim m→∞ E[(||x(m)»||)p]2 – (m) (m) ∀ǫ > 0, lim P max ||xj || ≤ (1 + ǫ) min ||xj || = 1. m→∞ 1≤j≤n 1≤j≤n

Sample estimate of Var[x2] / E[x2]2

0.5 0.4 0.3 0.2 0.1 0

0

100

200

300 400 dimensions (m)

500

600

700

0

100

200

300 400 dimensions (m)

500

600

700

log ( DMAXm / DMINm )

5 4 3 2 1 0

Var[(||x||q )p] (p) Applying this to our problem. Denote RVm (||x||q ) = E[(||x||q )p ]2

Using the independence of wn+1, ..., wm, we get: Pm P n Pn q] q q Var[|w | Cov[|w | , |w | ] + i i j i=n+1 j=1 i=1 (q) P m Pm RVm (||w||q ) = q q j=1 E[|wi | ]E[|wj | ] i=1 which converges to 0 as m → ∞.

Hence, the problem remains ill-posed despite the regularisation.

The effect of q Fortunately, not all norms concentrate at the same rate. x ~ Unif[0,1]

Sample estimate of Var[||x||2] / E[||x||2]2

0.25 L2−norms L1−norms L0.5−’norms’ L0.1−’norms’ 0.2

0.15

0.1

0.05

0

0

5

10

15

20

25

30

dimensions (m)

35

40

45

50

Theorem (Francois et al.’07, extended). If w ∈ Rm is a random vector with no more than n < m non-iid components, where n is finite, and all the other components being i.i.d, then Var[||w||q ] 1 σ2 = 2 2 lim m m→∞ E[||w||q ]2 q µ

(4)

where µ = E[wn+1], σ 2 = Var[wn+1] and n + 1 is one of the i.i.d. dimensions of w. Applying this to w, we can use (4) to approximate Var[||w||q ] 1 1 σ2 ≈ = ... 2 2 2 E[||w||q ] mq µ for some large m.

(5)

Computing µ and σ 2 for wn+1 ∼ Unif[−a, a]: Z a aq q 1 q = |wn+1 | µ = E[|wn+1 | ] = 2a q+1 −a a2q q 2 2 2q q 2 σ = E[|wn+1 | ] − E[|wn+1 | ] = (2q + 1)(q + 1)2

So, Var[||w||q ] 1 1 ≈ E[||w||q ]2 m 2q + 1

(6)

(Conveniently, a cancels out in this computation.)

Observe this is a decreasing function of q. Thus, the smaller the q the better, from the point of countering the concentration of the norm in regularisation.

3 relevant features

8

Test 0−1 errors

Test 0−1 errors

1 relevant feature

6 4 2 0

200

400 q=0.1

600

10 5

800 1000

200

q=0.3

q=0.5

0.3

400

600

q=0.7

800 1000 q=1

0.4

Test logloss

Test logloss

15

0.2 0.1

200

400 600 800 1000 Data dimensions

0.3 0.2 0.1 200

400 600 800 1000 Data dimensions

Comparative results on 1000-dimensional synthetic data from (Ng,’04). Each point is the median of > 100 indep. trials. The 0-1 errors are out of 100.

Train + valid. set size = 52+23

Train + valid. set size = 35+15

35 40 0−1 test errors

0−1 test errors

30 25 20 15 10

30 20 10

5 0

0 q=0.1

q=0.5

q=1

q=0.1

Train + valid. set size = 52+23

q=0.5

q=1

Train + valid. set size = 35+15 0.8

0.6 0.6 test logloss

test logloss

0.5 0.4 0.3 0.2

0.4 0.2

0.1 0

0 q=0.1

q=0.5

q=1

q=0.1

q=0.5

q=1

Results on 5000-dimensional synthetic data with only one relevant feature and even smaller sample size. The improvement over L1 becomes larger. (The 0-1 errors are out of 100.)

3 relevant features

4 2 0.4

0.6 0.8 q 1 relevant feature

1

0.25 0.2

0.2

0.15 0.1

0.6 0.8 q 3 relevant features

0.05 0.2

0.4

0.6 q

0.8

0.2 0.1 0.2

40 20 0 1 2 3 4 5 6 7 8 9 10 q*10

0.6 q

0.8

12 10 8 0.6 0.8 q exp decay relevance

40 20 0

1 2 3 4 5 6 7 8 9 10 q*10

0.4

0.2

0.4

1

0.35 0.3 0.25

1

60

0.2

0.4

3 relevant features

# features retained

# features retained

1 relevant feature

0.4

14

1

0.3

1

60

0.4

0.6 q

0.8

1

exp decay relevance

# features retained

0.2

Test logloss

Test logloss

0

Test 0−1 errors

6

exp decay relevance

14 12 10 8 6 4 2

Test logloss

8

Test 0−1 errors

Test 0−1 errors

1 relevant feature

40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 q*10

Results on synthetic data from (A.Ng,’04). Training set size n1 = 70, validation set size=30, and the out-of-sample test set size=100. The statistics are over 10 independent runs with dimensionality ranging from 100 to 1000.

Discussion & further work The learning-theoretic sample complexity bound for generalization is an (loose) upper-bound only. Our analysis based on norm-concentration so far only used that n << m. Further work should examine of the effect of r << m from this perspective. The phenomenon of concentration of norms and distances in very high dimensions impacts all high dimensional problems. Its implications for learning and generalization (and for other areas) is and open question.

References C.C. Aggarwal, A. Hinneburg, & D.A. Keim. On the surprising behavior of distance metrics in high dimensional space. Proc. Int. Conf. Database Theory, 2001, pp. 420-434. K. Beyer, J. Goldstein, R. Ramakrishnan, & U. Shaft. When is nearest neighbor meaningful? Proc. Int. Conf. Database Theory, pp. 217-235, 1999. D Fran¸ cois, V Wertz, & M Verleysen. The concentration of fractional distances. IEEE Trans. on Knowledge and Data Engineering, vol 19, no 7, July 2007 A Kab´ an and R.J Durrant. Learning with Lq<1 vs. L1 -norm regularization with exponentially many irrelevant features. Proc. ECML 2008, to appear. Z Liu, F Jiang, G Tian, S Wang, F Sato, S.J Meltzer, M Tan. Sparse Logistic Regression with Lp Penalty for Biomarker Identification. Statistical Applications in Genetics and molecular Biology. Vol.6, Issue 1, 2007. A.Y. Ng. Feature selection, L1 vs. L2 regularization, and rotational invariance. Proc. ICML 2004. Hui Zou and Runze Li: One-step sparse estimates in non-concave penalized likelihood models. The Annals of Statistics, 2008.