Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Non-convex Optimization for Linear System with Pregaussian Matrices and Recovery from Multiple Measurements Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
University of Georgia July 15, 2010
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
`q -minimization
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
problem
In compressed sensing, we want to recover a sparse or compressible signal via solving the minimization problem minimizex∈RN where
kxk0
kxk0
subject to
Ax = b
is the the number of non-zero entries of vector
namely the sparsity of
x,
and
A
is a matrix of size
m×N
x,
with
m N.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
`q -minimization
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
problem
In compressed sensing, we want to recover a sparse or compressible signal via solving the minimization problem minimizex∈RN where
kxk0
kxk0
subject to
Ax = b
is the the number of non-zero entries of vector
namely the sparsity of
x,
and
A
is a matrix of size
m×N
x,
with
m N. `q -approach with 0 < q ≤ 1, The
is to consider the
minimizex∈RN
kxkq
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
`q -minimization
subject to
problem
Ax = b.
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Null space property for SMV The null space property has been used to quantify the error of approximations (Cohen, Dahmen, and DeVore, 2009), and it also guarantees the exact recovery. Proposition (Null space property)
Let S ⊆ {1, 2, · · · , N } be a xed index set. Then a vector x with support (x) ⊆ S can be uniquely recovered from Ax = b using `1 -minimization if for all non-zero v in the null space of A, kvS k1 < kvS c k1 ,
where S c is the complement of S in {1, 2, · · · , N }.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Proof of the proposition Proof. We know for any non-zero
z
in the null space of
A,
kxS k1 ≤ kzS k1 + k(z + x)S k1 . By the assumption
kzS k1 < kzS c k1 ,
we then have
kxS k1 < k(z + x)S k1 + kzS c k1 . But
x
vanishes on
Sc,
thus
kxk1 = kxS k1 < k(z + x)S k1 + k(z + x)S c k1 = kz + xk1 , and so
x ∈ RN
is the unique solution to the
`1 -minimization
problem. Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Remarks to the proposition
Remark
`1
can be replaced by
`q
.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Remarks to the proposition
Remark
`1
can be replaced by
`q
.
Remark The converse is also true.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
MMV problem
De nition Given a set of
r
measurements
Ax(k) = b(k) Find the vectors
x(k)
for
k = 1, · · · , r.
which are jointly sparse.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
MMV problem
De nition Given a set of
r
measurements
Ax(k) = b(k) Find the vectors
x(k)
for
k = 1, · · · , r.
which are jointly sparse.
The MMV problem arises in biomedical engineering, especially in neuromagnetic imaging.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
`1,2 -minimization
The approach of mixed
`1,2 -minimization
to recover jointly sparse
vectors from MMV is solving the optimization problem
minimize
N q X x21,j + · · · + x2r,j
subject to
Ax(k) = b(k)
j=1 for
k = 1, · · · , r.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Null space property for MMV Theorem (BergFriedlander, 2009)
Let A be a real matrix of m × N and S ⊂ {1, 2, · · · , N } be a xed index set. Denote by S c the complement set of S in {1, 2, · · · , N }. Let k · k be any norm. Then all x(k) with support x(k) in S for k = 1, · · · , r can be uniquely recovered using the following minimize
N X
k(x1,j , · · · , xr,j )k subject to Ax(k) = b(k)
j=1
for k = 1, · · · , r if and only if for all vectors
(u(1) , · · · , u(r) ) ∈ (N (A))r \{(0, 0, · · · , 0)} satisfy the following X X k(u1,j , · · · , ur,j )k < k(u1,j , · · · , ur,j )k, . j∈S Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
j∈S c Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Real vs. complex null space properties Theorem (FoucartGribonval, 2009)
Let A be a matrix of size m × N and S ⊂ {1, · · · , N } be the support of the sparse vector y. The complex null space property: for any u ∈ N (A), w ∈ N (A) with (u, w) 6= 0, Xq Xq u2j + wj2 < u2j + wj2 , j∈S c
j∈S
where u = (u1 , u2 , · · · , uN )T and w = (w1 , w2 , · · · , wN )T is equivalent to the following standard null space property: for any u in the null space N (A) with u 6= 0, X
|uj | <
j∈S Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
X
|uj |.
j∈S c Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Questions raised by Foucart and Gribonval
1
Sparse recovery can also be achieved by
0 < q < 1.
Its success on a set
space property
S
kuS kq < kuS c kq
`q -minimization
for
is characterized by the null
u ∈ ker (A) \ {0}. 0 < q < 1? (Non-convex
for all
Does one have the equivalence for
Problem, Hahn-Banach theorem not applicable) 2
n-tuple (x1 , · · · , xN ) of vectors in RN , each of which supported on the same set S , is the unique solution of the `1,2 -minimization problem i a mixed `1,2 null space property holds. Does the real null space property imply the mixed `1,2 null space property when r ≥ 3? (Higher Dimensional
Every
Problem, perimeter infeasible)
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
`q -minimization
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
for MMV
We may consider a joint recovery from MMV via
minimize
N q q X x21,j + · · · + x2r,j
subject to
Ax(k) = b(k) ,
j=1
k = 1, · · · , r, for all 0 < q ≤ 1, where x(k) = (xk,1 , · · · , xk,N )T for k = 1, · · · , r.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
`q -minimization
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
for MMV
We may consider a joint recovery from MMV via
minimize
N q q X x21,j + · · · + x2r,j
subject to
Ax(k) = b(k) ,
j=1
k = 1, · · · , r, for all 0 < q ≤ 1, where x(k) = (xk,1 , · · · , xk,N )T for k = 1, · · · , r. Note that
lim x21,j + · · · + x2r,j
q
2
q→0+ Let
s
( 1 = 0
x1,j , · · · , xr,j nonzero x1,j , · · · , xr,j all zero.
if any of if
be the joint sparsity of the solution vectors
lim
q→0+
x(k) ,
then
N q q X x21,j + · · · + x2r,j = s. j=1
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Null Space property of `q -minimization for MMV Theorem (NSP of MMV
`q -minimization)
Let A be a real matrix of m × N and S ⊂ {1, 2, · · · , N } be a xed index set. Fix q ∈ (0, 1]. Then all x(k) supported on S for k = 1, · · · , r can be uniquely recovered using (10) if and only if for all vectors (u(1) , · · · , u(r) ) ∈ (N (A))r \{(0, 0, · · · , 0)} q q X q X q u21,j + · · · + u2r,j < u21,j + · · · + u2r,j .
(1)
j∈S c
j∈S
Furthermore, the above null space property is equivalent to the following condition: for any vector z ∈ N (A) with z 6= 0, X
|zj |q <
j∈S Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
X
|zj |q .
(2)
j∈S c Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Comparison theorem In order to prove the main theorem, let us rst show Theorem (Comparison theorem)
Given 0 < q ≤ 1 and any 2 × n matrix B with columns c1 , c1 , · · · , cn ∈ R2 , k(x, y)BS kq < k(x, y)BS c kq
(3)
for all (x, y) ∈ R2 \ {0} and some S ⊆ {1, 2, · · · , N }. Then X
kck kq2 <
k∈S
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
X
kck kq2 .
(4)
k∈S c
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Proof of the comparison theorem Proof.
B =: (bi,j )2×N , and without loss of S := {1, 2, · · · , s}. By the assumption, Let
s X
|b1,j x + b2,j y|q <
j=1 for all
(x, y) ∈ R2 \ {0}
N X
generality we can assume
|b1,j x + b2,j y|q
j=s+1 . Then by normalizing
(b1,j , b2,j ),
N q s q q q X X b21,j + b22,j |hvj , ξi|q < b21,j + b22,j |hvj , ξi|q . j=1
j=s+1
(continued on next slide) Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Proof of the comparison theorem (continued) Proof. Taking the integral of (2) over the unit circle
s X
b21,j
+
b22,j
j=1 Note that
q
ˆ q
|hvj , ξi| dξ <
2
S1
|h·, ξi|q dξ
b21,j
+
j=s+1
S1
´
N X
S1 ,
we have
b22,j
q
ˆ |hvj , ξi|q dξ.
2
S1
is constant. Therefore
s q N q q q X X b21,j + b22,j < b21,j + b22,j . j=1
j=s+1
That completes the proof.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem
Proof of the main theorem (sketch)
Proof. For
r = 2,
the null space property (2) implies that assumption (3)
in the comparison theorem is satis es;
r > 2, by using q |h·, ξi| dξ is constant, Sr−1 r−1 r where S denotes the unit sphere in R , and the rest are analogue to the case of r = 2. The comparison theorem can be generalized to
result from integral geometry that
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
´
a
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Restricted isometry property
For an integer
s ≤ n, the restricted isometry δ which satis es
constant
δs (A)
is
the smallest number
(1 − δ) kxk2 ≤ kAxk2 ≤ (1 + δ) kxk2 for any
s-sparse
vector
x ∈ Rn .
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Restricted isometry property
For an integer
s ≤ n, the restricted isometry δ which satis es
constant
δs (A)
is
the smallest number
(1 − δ) kxk2 ≤ kAxk2 ≤ (1 + δ) kxk2 for any
s-sparse
vector
x ∈ Rn .
Equivalently, the inequality
√
1 − δ ≤ smin (AS ) ≤ smax (AS ) ≤
holds for any
m×s
submatrix
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
√
1+δ
AS .
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Largest singular value in `2
Geman'1980 and Yin, Bai, and Krishnaiah'1988 showed that the largest singular value of random matrices of size independent entries of mean
√
m+
√
N
0
and variance
1
m×N
with
converges to
almost surely.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Largest
q -singular
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
value
De nition The largest
q -singular
value
(q)
s1 (A) : = for an
m×N
matrix
sup x∈RN , kxkq =1
kAxkq
A.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Some Challenges for `q and pregaussian
Pregaussian random variable has slower decay on the tail probabilities than subgaussian and gaussian random variables. Geometrically, perpendicular in
`2
needs to be generalized in
`p . The analogue of the central limit theorem in
`q
for
q 6= 2
becomes infeasible.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Upper tail probability of the largest
1-singular
Theorem (Upper tail probability of the largest
value
1-singular
value)
Let ξ be a pregaussian variable normalized to have variance 1 and A be an m × m matrix with i.i.d. copies of ξ in its entries, then (1) P s1 (A) ≥ Cm ≤ exp −C 0 m
for some C , C 0 > 0 only dependent on the pregaussian variable ξ .
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
A property of largest
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
q -singular
value
Lemma
For any 0 < q ≤ 1, the largest q -singular value is a quasi-norm on the space of matrices. In particular, we have
(q)
s1 (A + B)
q
q q (q) (q) ≤ s1 (A) + s1 (B)
for any m × N matrices A and B . Moreover (q)
s1 (A) = max kaj kq j
for 0 < q ≤ 1, where aj , j = 1, · · · , n, are the column vectors of matrix A. Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Proof of the lemma
Proof. For
0 < q ≤ 1, kAxkqq
≤
N X
|xj |q · kaj kqq ≤ kxkqq max kaj kqq . j
j=1 On the other hand,
(5)
(q)
maxj ||aj ||q ≤ s1 (A).
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Upper tail probability of largest
q -singular
value
The previous theorem allows us to estimate the largest value for
q -singular
0 < q < 1.
Theorem (Upper tail probability of the largest
q -singular
value)
Let ξ be a pregaussian variable normalized to have variance 1 and A be an m × m matrix with i.i.d. copies of ξ in its entries, then for any 0 < q < 1, 1 (q) P s1 (A) ≥ Cm q ≤ exp −C 0 m
for some C , C 0 > 0 only dependent on the pregaussian variable ξ .
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Sum of random variables
Lemma (Linear bound for partial binomial expansion)
For every positive integer n, n X +1 k=b n 2c
n k
xk (1 − x)n−k ≤ 8x
for all x ∈ [0, 1].
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Sum of random variables
Lemma
Suppose ξ1 , ξ2 , · · · , ξn are i.i.d. copies of a random variable ξ , then for any ε > 0, P
n X i=1
nε |ξi | ≤ 2
!
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
≤ 8P (|ξ| ≤ ε) .
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Lower tail probability of largest
1-singular
Theorem (Lower tail probability of the largest
value
1-singular
value)
Let ξ be a pregaussian variable normalized to have variance 1 and A be an m × N matrix with i.i.d. copies of ξ in its entries, then for any ε > 0, there exists K > 0 such that (1) P s1 (A) ≤ Km ≤ ε
where K only depends on ε and the pregaussian variable ξ .
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Lower tail probability of largest
1-singular
value
Proof. Since
aij
is pregaussian with variance
exists some
δ>0
1,
then for any
(1)
s1 (A) =
there
ε . 8
P (|aij | ≤ δ) ≤ Since
ε > 0,
such that
Pm
i=1 |aij0 | for some
j0 ,
by the previous lemma, we
have
! m X mδ mδ (1) P s1 (A) ≤ ≤ P ≤ 8P (|aij | ≤ δ) ≤ ε. |aij0 | ≤ 2 2 i=1
Finally let
K=
δ 2 , then the claim follows.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value
Lower tail probability of largest
For general
0 < q ≤ 1,
q -singular
value
we have
Theorem (Lower tail probability of the largest
q -singular
value )
Let ξ be a pregaussian variable normalized to have variance 1 and A be an m × N matrix with i.i.d. copies of ξ in its entries, then for every 0 < q ≤ 1 and any ε > 0, there exists K > 0 such that 1 (q) P s1 (A) ≤ Km q ≤ ε
where K only depends on q , ε and the pregaussian variable ξ .
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Smallest
q -singular
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
value
De nition The smallest
q -singular
value of an
s(q) n (A) : =
n×n inf
matrix
x∈Rn , kxkq =1
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
kAxkq .
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Smallest singular value in `2
Rudelson and Vershynin rst showed the following result Theorem (RudelsonVershynin, 2008)
If A is a matrix of size n × n whose entries are independent random variables with variance 1 and bounded fourth moment. Then for any δ > 0, there exists > 0 and integer n0 > 0 such that P sn (A) ≤ √ ≤ δ, n
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
∀n ≥ n0 .
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Smallest singular value in `2
Later, they proved the following result Theorem (RudelsonVershynin, 2008)
Let A be an n × n matrix whose entries are i.i.d. centered random variables with unit variance and fourth moment bounded by B. Then, for every δ > 0 there exist K > 0 and n0 which depend (polynomially) only on δ and B, and such that K ≤ δ, P sn (A) > √ n
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
∀n ≥ n0
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Lower tail probabilistic estimate on the smallest
q -singular
value For tall rectangular matrices, we have the lower tail probabilistic of exponential decay on the smallest
q -singular
value
Theorem
Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an m × n matrix with i.i.d. copies of ξ in its entries. Then for any ε > 0, there exist some γ > 0 and c > 0 and r ∈ (0, 1) dependent on q and ε, such that 1 q P s(q) (A) < γm < e−cm n
if n ≤ rm. Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Lower tail probabilistic estimate on the smallest
q -singular
value
Theorem
Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an n × n matrix with i.i.d. copies of ξ in its entries. Then for any ε > 0 and 0 < q ≤ 1, there exist some K > 0 and c > 0 dependent on q and ε, such that 1 − 1q P s(q) < Cε + Cαn + P kAk > Kn− 2 . n (A) < εn
where α ∈ (0, 1) and C > 0 depend only on the pregaussian variable and K . Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Lower tail probabilistic estimate on the smallest
q -singular
value
Theorem (Lower tail probabilistic estimate on the smallest
q -singular
value )
Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an n × n matrix with i.i.d. copies of ξ in its entries. Then for any ε > 0, there exists some γ > 0 such that − 1q P s(q) (A) < γn < ε, n
where γ only depends on q , ε and the pregaussian variable ξ .
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Upper tail probability of the smallest
q -singular
value
Theorem (Upper tail probabilistic estimate on the smallest
q -singular
value)
Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an n × n matrix with i.i.d. copies of ξ in its entries. Then for any K > e, there exist some C > 0, 0 < c < 1, and α > 0 only dependent on pregaussian variable ξ , q , such that − 21 P s(q) (A) > Kn ≤ n
C (ln K)α + cn . Kα
In particular, for any ε > 0, there exist some K > 0 and n0 , such that for all n ≥ n0 , − 21 < ε. P s(q) (A) > Kn n Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
Lower tail probability of the largest
p-singular
value
Theorem (Lower tail probability of the largest -singular value,
p > 1)
Let ξ be a pregaussian random variable normalized to have variance 1 and A be an m × N matrix with i.i.d. copies of ξ in its entries, then for every p > 1 and any ε > 0, there exists γ > 0 such that 1 (p) ≤ ε P s1 (A) ≤ γm p
where γ only depends on p, ε and the pregaussian random variable
ξ.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing
A Remark
Remark By the duality that for any
p≥1
and
(p)
(q)
s1 (A) = s1 where
1 p
+
1 q
= 1,
in particular, for
n×n AT
matrix
A,
(∞)
p = ∞, s1
(A)
is of order
n
with high probability.
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
2-singular For
value
p = 2,
we plot the largest
matrices of size
n × n,
Figure: Largest
where
2-singular
2-singular value of Gaussian n runs from 1 through 100.
random
value of Gaussian random matrices
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
1-singular
value
For p = 1, in the rst numerical experiment we plot the largest 1-singular value of Gaussian random of size n × n, where n runs from 1 through 100.
Figure: Largest
1-singular
value of Gaussian random matrices:
Experiment 1
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
1-singular
value
In the second numerical experiment for
1-singular value of Gaussian n runs from 1 through 200.
Figure: Largest
1-singular
p = 1,
we plot the largest
random matrices of size
n × n,
where
value of Gaussian random matrices:
Experiment 2
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
1-singular
value
In the third experiment for
p = 1,
we plot the largest
value of Gaussian random matrices of size from
1
through
Figure: Largest
n × n,
1-singular n runs
where
400.
1-singular
value of Gaussian random matrices:
Experiment 3
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
∞-singular For
value
p = ∞,
we plot the largest
random matrices of size
Figure: Largest
n × n,
∞-singular
∞-singular value of Gaussian where n runs from 1 through 500.
value of Gaussian random matrices
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
1 3 -singular value
For
p=
1 1 3 , we plot the largest 3 -singular value of Gaussian random
matrices of size
n × n,
Figure: Largest
where
n
runs from
1
through
500.
1 3 -singular value of Gaussian random matrices
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
1 4 -singular value
For
p=
1 1 4 , we plot the largest 4 -singular value of Gaussian random
matrices of size
n × n,
Figure: Largest
where
n
runs from
1
through
300.
1 4 -singular value of Gaussian random matrices
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments
1 4 -singular value
For rectangular matrices, we also plot the largest of Gaussian random matrices of size from
1
through
Figure: Largest
m × n,
1 4 -singular value
where
m
and
n
run
100.
1 4 -singular value of rectangular Gaussian random matrices
Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)
Sparse Recovery by `q -minimization and q -Singular Values
