A Note on Convex Relaxations for Non-Metric Smoothness Priors Christopher Zach, Microsoft Research Cambridge∗ August 13, 2012 In [2] the following convex relaxation for labeling problems with non-metric but isotropic smoothness costs is proposed: ( ) X X X def i T i i i R(u) = max (ps ) ∇us + qs us , (1) (ps ,qs )∈Cs

s

i

i

where def  Cs = (ps , qs ) : kpis − pjs k2 + qsi ≤ θij ∀i, j .

(2)

Here θij ≥ 0 are the costs for a transition from label i to j. us ∈ ∆ is a (relaxed) one-hot encoding (or node marginal) of chosing label i at pixel s (uis = 1, ujs = 0 for j 6= i). Since the objective above decouples into independent problems over pixels s, we focus on a single pixel and drop the subscript indicating s in the def def following. We introduce rij = pi − pj and sij = q i , then one term in the objective above can be written as   X  X X X X max (pi )T ∇ui + q i ui − ı{krij k2 + sij ≤ θij } − ı{pi − pj − rij = 0} − ı{sij − q i = 0} .  (p,q)  i

i

i,j

i,j

i,j

(3) We will derive the primal of this expression using the following variant of Fenchel duality, min f (Ax) = max −f ∗ (y). x

(4)

y:AT y=0

The primal variables correspond to Lagrange multipliers y ij for pi − pj − rij = 0 and z ij for sij − q i = 0. We have X X X −f ∗ (p, q, r, s) = (pi )T ∇ui + q i ui − ı{krij k2 + sij ≤ θij }, (5) i

i

i,j

where each summand is now independent and f can be computed term-wise. Since the convex conjugate of y 7→ aT y is x 7→ ı{x = a}, the first two terms translate to constraints (Ay y + Az z)pi = −∇ui and (Ay y + Az z)qi = −ui in the primal. Here we decomposed A = [Ay |Az ], and e.g. (Ay y)pi refers to the rows def

in Ay y corresponding to the dual variable pi . The conjugate of h∗θ (z, w) = ı{kzk2 + w ≤ θ} requires a little derivation: hθ (x, y) =

max z,w:kzk2 +w≤θ

xT z + yw =

xT z + yw

max z,w,ε≥0:kzk2 +w+ε=θ

= max xT z + y(θ − kzk2 − ε) = θy + max xT z − y(kzk2 + ε) z,ε≥0 z,ε≥0  T = θy + max x z − ykzk2 + max −εy Ansatz: z = cx, c ≥ 0 z ε≥0  = θy + max ckxk22 − ckxk2 y + ı{y ≥ 0} = θy + kxk2 max {c(kxk2 − y)} + ı{y ≥ 0} c≥0

c≥0

= θy + kxk2 ı{y ≥ kxk2 } + ı{y ≥ 0} = θy + ı{y ≥ kxk2 }. ∗I

am grateful to Christian H¨ ane for discussion and proof-reading.

1

(6)

We used the particular ansatz, since xT z − ykzk2 will be maximized w.r.t. z if z is pointing into the same direction as x. In order to finally derive the primal energy, we need to specify the rows of Ay y + Az z. First note that the constraints in the dual form two separate partitions, hence (Ay y)qi = 0, (Ay y)sij = 0 and (Az z)pi = 0, (Az z)rij = 0. Since pi appears in constraints y ij (for j 6= i) with +1 coefficient, and in y ji for j 6= i with -1 coefficient (note that pi cancels in constraint y ii ), we have X X (Ay y)pi = y ij − y ji . (7) j:j6=i

j:j6=i

Since rij appears exactly in the constraint corresponding to the multiplier y ij (with -1), P we have (Ay y)rij = −y ij . q i appears with a -1 coefficient in all constraints z ij for j, and (Az z)qi = − j z ij . Thus, we can already state the two linear constraints appearing in the primal problem, X X X ∇ui = y ji − y ij and ui = z ij . j:j6=i

j

j:j6=i

The dual capacity constraints krij k2 + sij ≤ θij translate to X
X ij ij
θij z ij + ı{z ij ≥ k−y ij k2 } = θ z + ı{z ij ≥ ky ij k2 } . i,j

(8)

i,j

Overall we can state R(u) in its primal form XX
R(u) = min θij z ij + ı{z ij ≥ ky ij k2 } x,y

s.t. ∇uis =

s

X j:j6=i

(9)

i,j

ysji −

X

ysij ,

zsij ≥ kysij k2 ,

uis =

X

zsij .

j

j:j6=i

The first set of constraints are the differential marginalization constraints as derived in [4]. The second set of constraints forces zsij to be no smaller than kysij k2 . If the final set of constraints is dropped, any minimizer will satisfy zsij = kysij k2 and we have Etight in [4] (which is the primal form of the saddlepoint energy proposed in [1]). The last set of constraints tell us, that the zsij are pairwise pseudo-marginals for a transition from label i to j regardless of the orientation of the discontinuity. This also means, that zsij is bounded from above by 1 (since uis ≤ 1). Hence, the contribution of a jump from i to j at pixel s is at most θij regardless of the direction. In particular, diagonal boundaries on regular pixel grids will be underestimated in their cost. Our understanding therefore is, that R(u) is not a stronger relaxation than Etight addressing non-metric smoothness (since even for metric pairwise potentials the minimizer will be in general not the same as Etight ), but a different relaxation in the first place. This is not a contradiction with Section 2.4 in [2], since there the qsi are explicitly forced to be 0, which again leads directly to the primal energy Etight . Etight-marginals as proposed in [3] is a proper way to strengthen Etight such that minimizers for metric smoothness costs coincide in both settings.

References [1] A. Chambolle, D. Cremers, and T. Pock. A convex approach for computing minimal partitions. Technical report, Ecole Polytechnique, 2008. [2] E. Strekalovskiy, C. Nieuwenhuis, and D. Cremers. Nonmetric priors for continuous multilabel optimization. In Proc. ECCV, 2012. to appear. [3] C. Zach, C. H¨ ane, and M. Pollefeys. What is optimized in convex relaxations for multi-label problems: Connecting discrete and continuously-inspired MAP inference. Technical report, MSR Cambridge, 2012. [4] C. Zach, C. H¨ ane, and M. Pollefeys. What is optimized in tight convex relaxations for multi-label problems? In Proc. CVPR, 2012.

2