Geometry Motivated Variational Segmentation for Color Images Alexander Brook1, Ron Kimmel2 , and Nir A. Sochen3 1
3
Dept. of Mathematics, Technion, Israel
[email protected] 2 Dept. of Computer Science, Technion, Israel Dept. of Applied Mathematics, Tel-Aviv University, Israel
Abstract. We propose image enhancement, edge detection, and segmentation models for the multi-channel case, motivated by the philosophy of processing images as surfaces, and generalizing the Mumford-Shah functional. Refer to http://www.cs.technion.ac.il/~sova/canada01/ for color figures.
1
Introduction
We provide a general variational framework for color images, generalizing the Mumford-Shah functional. Our goal is to provide a theoretical background for the model proposed and implemented in [25]. In Section 2 we give a review of variational segmentation and color edge detection. Section 3 offers a summary of the theory of the Mumford-Shah functional and of numerical minimization methods devised for this functional. We propose two generalizations of the Mumford-Shah functional in Section 4 and show some numerical results. A few words on the notation. The norm | · | is the usual Euclidean norm of any object: a number, a vector, or a matrix. In particular, for a function ∂ui 2 1/2 n ( ∂xj ) . L is the Lebesgue measure on Rn . u : Rn → Rm we put |∇u| = Hn−1 is the (n − 1)-dimensional Hausdorff measure, which is a generalization of the area of a submanifold.
2
Image Segmentation
We consider images as functions from a domain in R2 into some set, that will be called the feature space. When needed, we suppose that the domain is [0, 1]2 . Some examples of feature spaces are [0, 255] or [0, ∞), for gray-level images, or [0, 1]3 for color images in RGB. In [28] Mumford and Shah suggested segmenting an image by minimizing a functional of the form Ω\K ( ∇u 2 + α u − w 2) + β length(K), where K is the union of edges in the image. Thay conjectured that there are minimizers over M. Kerckhove (Ed.): Scale-Space 2001, LNCS 2106, pp. 362–370, 2001. c Springer-Verlag and IEEE/CS 2001
Geometry Motivated Variational Segmentation for Color Images
363
u ∈ C 1 (Ω \ K) and K being finite union of smooth arcs, and decribed possible conjectured configurations for endpoints and crossings in K. The minimization of the Mumford-Shah functional poses a difficult problem, both theoretical and numerical, because it contains both area and length terms and is minimized with respect to two variables: a function f : Ω → R and a set K ⊂ Ω. This kind of functionals was introduced in [18]. The usual mean of providing coupling between the color channels is by defining a suitable edge indicator function e, that is supposed to be small in the smooth parts of the image and large in the vicinity of an edge. A typical example is e(x) = |∇f (x)|2 , and the integral e usually constitutes the smoothing term. One of the promising frameworks to derive and justify edge indicators is to consider images as embedded manifolds and to look at the induced metric for qualitative measurements of image smoothness. This idea first appeared in [21], and was extended in [16]. This interpretation was formulated in the most general way and implemented in [31]. The n-dimensional m-valued image is considered as an n-dimensional n+m , h) given by (x1 , . . . , xn , f1 (x1 , . . . , xn ), . . . , fm (x1 , . . . , xn )), manifold √ in (R and det g, where g is the metric on the manifold induced by the √ metric h from Rn+m , is taken to be the edge indicator function. The integral det g gives the n-dimensional volume of the manifold, and its minimization brings on a kind of non-isotropic diffusion, which the authors called the Beltrami flow. As pointed out in [31], when implementing such a diffusion, one must decide what is the relationship between unit lengths along the xi axes and along the fj axes. The significance of the ratio of the scales is discussed in detail in [31]. We will denote this coefficient by γ. In the case of gray-level images this framework was first introduced in [22]. Here the image is a surface in R3 , the edge indicator is the area element (1 + fx2 + fy2 )1/2 , and the flow is closely related to the mean curvature flow. In a number of works (e.g. [11, 32]) another problem is considered, leading to very similar equations. It is the problem of smoothing, scaling and segmenting an image in a “non-flat” feature space, like a circle, a sphere or a projective line. Other related formulations and models for color images were proposed in [26, 5, 30, 16, 12, 7, 8]
3
Mumford-Shah Functional
The core difficulty in proving this conjecture is that the functional is a sum of an area and a curvilinear integrals, and the curve of integration is one of the variables. There is yet another problem, namely that the proposed domain of u and K is too restrictive and lacks some convenient properties (compactness, lower semicontinuity of the functionals in question). This one, however, is ordinary; instead of imposing that K is a finite union of smooth arcs, we should drop this requirement, and prove later that a minimizing K must be smooth. This also
364
Alexander Brook, Ron Kimmel, and Nir A. Sochen
necessitates replacing length(K) with something defined on non-smooth sets; the most natural replacement is H1 (K), the one-dimensional Hausdorff measure of K. The crucial idea in overcoming the difficulties of interaction between the area and the length terms is to use a weak formulation of the problem. First, we let K be the set of jump points of u: K = Su . The functional thus depends on u only. Second, we relax the functional in L2 , that is, we consider 2
L ¯ E(u) = inf{lim inf E(uk , Suk ) : uk → u, uk ∈ C 1 (Ω \ Suk ), H1 (Suk \ Suk ) = 0}. k→∞
It turns out (see [4]) that this functional has an integral representation ¯ E(u) = (|∇u|2 + α|u − w|2 ) + βH1 (Su ) Ω
¯ and if E(u) is finite then u ∈ SBV, the space of special functions of bounded variation. For the definition of SBV, and also of BV, GBV, and GSBV spaces we refer the reader to the book [2] In this weak setting it was shown in [19] that there are indeed minimizers of E¯ and that at least some of them are regular enough (with K closed and it was proven for the more general case of Ω ⊂ Rn , u ∈ C 1 (Ω \ K)). Actually, 2 n 2, and F (u) = Ω (|∇u| + α|u − w|2 ) + βHn−1 (Su ). An interesting and important limiting case of the Mumford-Shah functional is the problem F¯ = α|u − w|2 + βHn−1 (Su ), ∇u = 0 on Ω \ Su Ω
of approximating g by a piecewise-constant function. For this functional, the Mumford-Shah conjecture was proved already in the original paper [28]; an elementary constructive proof can be found in [27]. Existence of minimizers for any n 2 was shown in [15]. The main difficulty that hampers attempts to minimize the Mumford-Shah functional E(u, K) numerically is the necessity to somehow store the set K, keep track of possible changes of its topology, and calculate it’s length. Also, the number of possible discontinuity sets is enormous even on a small grid. We can, however, try to find another functional approximating the MumfordShah functional that will also be more amenable to numerical minimization. The framework for this kind of approximation is Γ -convergence, introduced in [20] (also see the book [17]). Consider a metric space (X, d). A sequence of functionals Fi : X → R+ is said to Γ -converge to F : X → R+ (Γ-lim Fi = F ) if for any f ∈ X ∀fi → f : lim inf Fi (fi ) F (f )
and ∃fi → f : lim sup Fi (fi ) F (f ).
We can extend this definition to families of functionals depending on a continuous parameter ε ↓ 0, requiring convergence of Fεi to F (x) on every sequence εi ↓ 0.
Geometry Motivated Variational Segmentation for Color Images
365
It is important to notice that Γ -limit depends on what kind of convergence we have on X. Sometimes, to avoid ambiguities, it is designated as Γ (X)- or Γ (d)-limit. For us, the most important property of Γ -convergence is that if Γ-lim Fi = F , fi minimizes Fi and fi → f , f minimizes F . We come back to the task of approximating the Mumford-Shah functional by a nicer functional. However, we can not approximate F (u) with functionals of the usual local integral form Fε (u) = Ω fε (∇u, u) for u ∈ W 1,1 (see [10, p. 56]). One of the possibilities to overcome this is to introduce a second auxiliary variable, which was done in [3, 4]. The approximation proposed in [4] is (v − 1)2 2 2 2 2 + ε|∇v| + α|u − w| dx. v |∇u| + β (1) Fε (u, v) = 4ε Ω The meaning of v in this functional is clear—it approximates 1 − χSu , being close to 0 when |∇u| is large and 1 otherwise. This functional is elliptic and is relatively easy to minimize numerically. A finite-element discretization was proposed in [6], with a proof that the discretized functionals also Γ -converge to F (u) if the mesh-size is o(ε). Other works, suggesting approximations and numerical methods for the Mumford-Shah functional are [10, 29, 9, 33, 27, 13, 14].
4
Generalizing Mumford-Shah Functional to Color
The most obvious way to generalize the Mumford-Shah functional to color images u : Ω → R3 is to use (|∇u|2 + α|u − w|2 ) + βHn−1 (Su ). (2) F (u) = Ω
In this case the only coupling between the channels is through the common jump set Su . The approximation results from Section 3 translate to this case without change (as noted in [4]) and we can use the elliptic approximation (v − 1)2 + ε|∇v|2 + α|u − w|2 dx, v 2 |∇u|2 + β (3) Fε (u, v) = 4ε Ω to find minimizers of F (u). We minimize Fε by steepest descent. A result of numerical minimization is shown in Figure 1. The original image was a noisy color image also shown in Figure 1. We want to generalize the Mumford-Shah functional (|∇u|2 + α|u − w|2 ) + βHn−1 (Su ) Ω
to color images, using the “image as a manifold” interpretation, while the length term Hn−1 (Su ) remains the same.
366
Alexander Brook, Ron Kimmel, and Nir A. Sochen
Fig. 1. Results with ε = 0.05, α = 0.7, β = 0.022, t = 20 and the original image.
The fidelity term that is most consistent with the geometric approach would be the Hausdorff distance between the two surfaces, or at least Ω d(u(x), w(x)), where d(·, ·) is the geodesic distance in the feature space, as in [8]. Yet, both these approaches seem computationally intractable. The suggestion of Ω u − w 2h , made in [25], (here hij is the metric on the feature space, and · h is the corresponding norm on the tangent space) is easy to implement, but lacks mathematical validity: u−w is not in the tangent space. We will use the simplest reasonable alternative, Ω |u − w|2 . √ The smoothing term is the area Ω det g or the energy Ω det g, where g is the metric induced on the manifold by h—the metric on the feature space. In the case where h is a Euclidean metric on Ω × [0, 1]3 and U (x, y) = (x, y, R, G, B) = (x, y, u1 (x, y), u2 (x, y), u3 (x, y)) is the embedding, we get det g = det(dU ∗ ◦ h ◦ dU ) = γ 2 + γ
|∇ui |2 +
i
|∇ui × ∇uj |2
i,j
= γ 2 + γ(|ux |2 + |uy |2 ) + |ux × uy |2 = γ 2 + γ|∇u|2 + |ux × uy |2 . Thus, we have two models: γ 2 + γ|∇u|2 + |ux × uy |2 + α |u − w|2 + βHn−1 (Su ), F 1 (u) = Ω Ω 2 2 2 F (u) = (γ|∇u| + |ux × uy | ) + α |u − w|2 + βHn−1 (Su ). Ω
Ω
Note that γ 2 was dropped in the second functional, since in this case it merely adds a constant to the functional. However, the theory of functionals on SBV or GSBV seems to be unable to deal with these models at the moment. It is necessary to establish lower semicontinuity of the functionals, both to ensure existence of minimizers, and as an important component in the Γ -convergence proofs. Though, theorems on lower semicontinuity of functionals on SBV exist only for isotropic functionals
Geometry Motivated Variational Segmentation for Color Images
367
(depending only on |∇u| and not on ∇u itself), or at least functionals with constant rate of growth, i.e. c|∇u|r f (∇u) C(1 + |∇u|)r for some C > c > 0 and r > 1. The term |ux × uy |2 is of order |∇u|4 , yet we can not bound it from below by c|∇u|4 for some c > 0, therefore we can not use these theorems. The role of the term |ux ×uy |2 is explored in [24]. If we assume the Lambertian light reflection model, then u(x, y) = (n(x, y) · l)ρ(x, y), where n(x, y) is the unit normal to the surface, l is the source direction, and ρ(x, y) captures the characteristics of the material. Assuming that for any given object ρ(x, y) = ρ = const we have u(x, y) = (n(x, y) · l)ρ, hence Im u ⊂ span{ρ} and rank du 1. This is equivalent to ux × uy = 0. Thus, the term |ux ×uy |2 in the edge indicator enforces the Lambertian model on every smooth surface patch. It also means that taking rather small γ makes sense, since we expect |ux × uy |2 to be (almost) 0, and |∇u|2 to be just small. A generalization of the Mumford-Shah functional proposed here is an attempt to combine the nice smoothing-segmenting features of the geometric model with the existing Γ -convergence results for the elliptic approximation of the original Mumford-Shah functional. We pay for that by the loss of some of the geometric intuition behind the manifold interpretation. The proposed models are just F 1 and F 2 with |ux × uy |2 replaced by |ux × uy |, that is
(γ|∇u| + |ux × uy |) + α |u − w|2 + βHn−1 (Su ), Ω Ω 2 2 2 G (u) = γ + γ|∇u| + |ux × uy | + α |u − w|2 + βHn−1 (Su ). G1 (u) =
2
Ω
Ω
Note that |ux × uy | enforces the Lambertian model, just as |ux × uy |2 . The new functional G2 seems to violate another important requirement, necessary for lower semicontinuity with respect to L1 convergence: being quasiconvex. Besides, since the smoothing term is of linear growth, approximation similar to those in Section 3 will converge to a functional with more interaction between the area and the length terms, and depending on the Cantor part of Du. We thus propose the functional G3 (u) =
γ + |∇u|2 + α |u − w|2 + β Ω
Ω
Su
|u+ −u− | 1+|u+ −u− |
dHn−1 + |Dc u|(Ω).
The elliptic approximation for G1 is provided in [23]: (v − 1)2 2 = v (γ|∇u| + |ux × uy |) + β ε|∇v| + + α|u − w|2 . 4ε Ω
G1ε (u, v)
2
2
Results of numerical minimization by steepest descent are shown in Figure 2. A functional similar to the Mumford-Shah functional, but with linear growth in the gradient is examined in [1], and it is proved in particular that Γ-lim Gε = G
368
Alexander Brook, Ron Kimmel, and Nir A. Sochen
Fig. 2. Results for G1 with ε = 0.25, α = 0.05, β = 0.002, γ = 0.01, t = 20. (with respect to L1 convergence), where (1 − v)2 2 2 v f (|∇u|) + β ε|∇v| + Gε (u, v) = 4ε Ω if u, v ∈ H 1 (Ω) and 0 v 1 a.e., and +∞ otherwise, |u+ −u− | 1 c G(u, v) = f (|∇u|) + β 1+|u+ −u− | dH + |D u|(Ω) Ω
Su
if u ∈ GBV(Ω) and v = 1 a.e., and +∞ otherwise, and f : [0, +∞) → [0, +∞) is convex, increasing, and limz→∞ f (z)/z = 1. With the aim of generalizing this result to color images we define f (z) = γ + z 2 , |u+ −u− | n−1 G3 (u) = γ + |∇u|2 + α |u − w|2 + β + |Dc u|(Ω), 1+|u+ −u− | dH Ω Ω Su (1 − v)2 G3ε (u, v) = v 2 γ + |∇u|2 + α|u − w|2 + β ε|∇v|2 + , 4ε Ω with domains as above. Upon inspection of the proofs in [1], it seems that everything remains valid for the vectorial case, except one part, that establishes the lower inequality for the one-dimensional case (n = 1) in a small neighborhood of a jump point. We can, however, provide a “replacement” for this part (the second part of Proposition 4.3 in [1], beginning with (4.4)). A result of numerical minimization of G3ε is shown in Figure 3.
References [1] R. Alicandro, A. Braides, and J. Shah. Free-discontinuity problems via functionals involving the L1 -norm of the gradient and their approximation. Interfaces and Free Boundaries, 1(1):17–37, 1999. [2] L. Ambrosio, N. Fusco, and D. Pallara. Functions of Bounded Variation and Free Discontinuity Problems. Oxford Mathematical Monographs. Oxford University Press, 2000.
Geometry Motivated Variational Segmentation for Color Images
369
Fig. 3. Results for G3 with ε = 0.01, α = 3, β = 0.02, γ = 0.001, t = 4. [3] L. Ambrosio and V. M. Tortorelli. Approximation of functionals depending on jumps by elliptic functionals via Γ -convergence. Comm. Pure Appl. Math., 43(8):999–1036, 1990. [4] L. Ambrosio and V. M. Tortorelli. On the approximation of free discontinuity problems. Boll. Un. Mat. Ital. B (7), 6(1):105–123, 1992. [5] C. Ballester and M. Gonz´ alez. Texture segmentation by variational methods. In ICAOS ’96. 12th International Conference on Analysis and Optimization of Systems. Images, Wavelets and PDEs, pages 187–193. Springer Verlag, 1996. [6] G. Bellettini and A. Coscia. Discrete approximation of a free discontinuity problem. Numer. Funct. Anal. Optim., 15(3-4):201–224, 1994. [7] P. Blomgren and T. F. Chan. Color TV: total variation methods for restoration of vector-valued images. IEEE Trans. Image Processing, 7(3):304–309, 1998. [8] A. Bonnet. On the regularity of edges in image segmentation. Ann. Inst. H. Poincar´e Anal. Non Lin´eaire, 13(4):485–528, 1996. [9] B. Bourdin and A. Chambolle. Implementation of an adaptive finite-element approximation of the Mumford-Shah functional. Numer. Math., 85(4):609–646, 2000. [10] A. Braides. Approximation of free-discontinuity problems. Lecture Notes in Math., 1694. Springer-Verlag, Berlin, 1998. [11] T. Chan and J. Shen. Variational restoration of non-flat image features: models and algorithms. Technical Report 99-20, UCLA CAM, 1999. [12] T. F. Chan, B. Y. Sandberg, and L. A. Vese. Active contours without edges for vector-valued images. J. Visual Communication Image Representation, 11(2):130– 141, 2000. [13] T. F. Chan and L. A. Vese. An active contour model without edges. In M. Nielsen, P. Johansen, O. F. Olsen, and J. Weickert, editors, Scale-Space Theories in Computer Vision, Second International Conference, Scale-Space’99, volume 1682 of Lecture Notes in Comp. Sci., pages 141–151. Springer, 1999. [14] T. F. Chan and L. A. Vese. Image segmentation using level sets and the piecewiseconstant Mumford-Ssah model. Technical Report 00-14, UCLA CAM, 2000. [15] G. Congedo and I. Tamanini. On the existence of solution to a problem in multidimensional segmentation. Ann. Inst. Henri Poincar´e Anal. Non Lin´eaire, 8(2):175– 195, 1991. [16] A. Cumani. Edge detection in multispectral images. CVGIP: Graphical Models and Image Processing, 53(1):40–51, 1991. [17] G. Dal Maso. An introduction to Γ -convergence. Birkh¨ auser Boston Inc., Boston, MA, 1993.
370
Alexander Brook, Ron Kimmel, and Nir A. Sochen
[18] E. De Giorgi and L. Ambrosio. New functionals in the calculus of variations. Atti Accad. Naz. Lincei Rend. Cl. Sci. Fis. Mat. Natur. (8), 82(2):199–210 (1989), 1988. [19] E. De Giorgi, M. Carriero, and A. Leaci. Existence theorem for a minimum problem with free discontinuity set. Arch. Rational Mech. Anal., 108(3):195–218, 1989. [20] E. De Giorgi and T. Franzoni. Su un tipo di convergenza variazionale. Atti Accad. Naz. Lincei Rend. Cl. Sci. Fis. Mat. Natur. (8), 58(6):842–850, 1975. [21] S. Di Zenzo. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing, 33(1):116–125, 1986. [22] A. I. El-Fallah and G. E. Ford. The evolution of mean curvature in image filtering. In Proceedings of IEEE International Conference on Image Processing, volume 1, pages 298–302, 1994. [23] M. Focardi. On the variational approximation of free-discontinuity problems in the vectorial case. Technical report, SNS, Pisa, 1999. [24] R. Kimmel, R. Malladi, and N. Sochen. Images as embedded maps and minimal surfaces: Movies, color, texture, and volumetric medical images. Int. J. Computer Vision, 39(2):111–129, 2000. [25] R. Kimmel and N. Sochen. Geometric-variational approach for color image enhancement and segmentation. In M. Nielsen, P. Johansen, O. F. Olsen, and J. Weickert, editors, Scale-space theories in computer vision, volume 1682 of Lecture Notes in Comp. Sci., pages 294–305. Springer, 1999. [26] T. S. Lee, D. Mumford, and A. Yuille. Texture segmentation by minimizing vectorvalued energy functionals: the coupled-membrane model. In G. Sandini, editor, Computer vision – ECCV ’92, volume 558 of LNCS, pages 165–173. Springer Verlag, 1992. [27] J.-M. Morel and S. Solimini. Variational methods in image segmentation. Birkh¨ auser Boston Inc., Boston, MA, 1995. With seven image processing experiments. [28] D. Mumford and J. Shah. Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math., 42(5):577–685, 1989. [29] M. Negri. The anisotropy introduced by the mesh in the finite element approximation of the Mumford-Shah functional. Numer. Funct. Anal. Optim., 20(9-10):957– 982, 1999. [30] G. Sapiro and D. L. Ringach. Anisotropic diffusion of multivalued images with application to color filtering. IEEE Trans. Image Processing, 5(11):1582–1586, Oct. 1996. [31] N. Sochen, R. Kimmel, and R. Malladi. A general framework for low level vision. IEEE Trans. Image Processing, 7(3):310–318, 1998. [32] B. Tang, G. Sapiro, and V. Caselles. Diffusion of general data on non-flat manifolds via harmonic maps theory: The direction diffusion case. Int. J. Computer Vision, 36(2):149–161, 2000. [33] A. Tsai, A. Yezzi, and A. S. Willsky. Curve evolution, boundary value stochastic processes, the Mumford-Shah problem, and missing data applications. In IEEE International Conference on Image Processing, volume 3, pages 588–591. IEEE, 2000.
