PROGRESSIVE COLOR REMOVAL USING POISSON EMBEDDING FOR SEAMLEASS GLOBAL/LOCAL INTEGRATION Yilun Chen1, Qiong Yang2, Chao Wang3 1 Tsinghua University, Beijing, 100084, China 2 Microsoft Research Asia, Beijing, 100080, China 3 University of Science and Technology of China, Heifei, 230027, China ABSTRACT When converting color images into grayscales, traditional global methods may not preserve local salience very well. To achieve good results both globally and locally, we propose a progressive color removal approach, where we divide the problem into substages: global optimization and local optimization with global constraints, and then we use Poisson embedding to integrate the above two results. The approach elegantly combines the strengths of both global and local operations, where we greatly enhance the local salience without change of the global effect. It also provides the user a coarse-to-fine framework to refine the results step by step. The idea of using Poisson embedding for global/local integration can also be adopted in other applications, where it is applied in image compression as an example.

Index Terms— Color graphics, image color analysis, image representation

1. INTRODUCTION Converting color images to grayscales is necessary in many tasks, such as using a low-price grayscale device to display or print a color image, limited-band transmission, data compression, and colorization. Many algorithms exist to find the grayscale mapping of color images[1-4]. However, they mostly tend to solve the problem via a global optimization on the whole image. For instance, in [2], the linear mapping function is obtained by choosing the principal axis which is the optimal projection vector with least reconstruction error on all pixels. As a result, it cannot preserve contrast of local regions very well when the color difference of local region falls on the non-principal axis.(see Fig.1e, the right two color bands take the same gray level). Similar effect can be seen in the methods based on nonlinear mapping function using statistical analysis of all pixels [1]. Other examples are the color-difference mapping methods, such as [3] and [4], which tried to maintain the color differences between pixels via a global minimizing framework. During the optimization process on the whole image, contributions from local regions may be neglected, thus local features would be lost accordingly (see Fig.1c, some letters of “ACM MM" become indiscernible in Color2Gray result). The local salience, however, is quite important, because human visual system is more sensitive to the relative difference between neighboring pixels rather than distant pixels and their absolute values. In pursuit of a good result both globally and locally, the integration of global method and local method is necessary. The global/local integration is useful in many fields,

such as pattern recognition [8], image retrieval [9], phase unwrapping [10] and etc. Traditionally, a weighted objective function optimization is adopted to assign larger weights on the more interested local region [10]. In such a framework, a tradeoff has to be made due to the fact that local improvement is at the cost of global effect, i.e, the global result is degenerated when the local result is emphasized. In fact, this tradeoff is not necessary, if we handle the problem in a progressive way. We first conduct the optimization on the whole image to achieve an optimal result, and then conduct a constrained optimization on the interested local region with global constraints to enhance the local salience. After that, we apply Poisson equation to seamlessly embed the local result into the global one. In this way, we achieve good result both globally and locally. The main advantage of the proposed framework is that the local salience is enhanced without change of global effect; therefore a global/local tradeoff does not need to be made, which is distinctively different from the previous global/local integration methods. Furthermore, the progressive framework can be also useful for many other applications, such as image fusion, image compression and reconstruction, image enhancement, and image denoising, where it is applied in image compression as an example.

2. FRAMEWORK Let Φ (Ω) denote the color image Φ on the region Ω ; let

φ (Ω) be the corresponding grayscale. The conversion from the color image into grayscale can be defined as a function h : Φ ( Ω ) a φ ( Ω ) , which satisfies E (Φ, φ ) → min , where E (⋅, ⋅) is the objective function defined over Ω . During the optimization on global region Ω , some local salience may be lost. Suppose the local region to be Ω L . Using

φ ( Ω L ) as the constraint, we conduct the optimization on the local region by V (Φ,ψ ) → min , where V (⋅, ⋅) is the local objective function over region Ω L . We then get another function

g : Φ ( Ω L ) a ψ ( Ω L ) for salience preserving in the local region. Without loss of global effect, the new result on the local region ψ ( Ω L ) is embedded into φ (Ω) to achieve a good result φ * (Ω) both globally and locally:

φ * (Ω) = f (φ (Ω),ψ (Ω L ) ) ,

(1)

(a)

(b)

(c)

(d)

(e)

Figure 1. Color removal (“banner”). (a)Color source, (b)Photoshop Gray, (c)Color2Gray, (d)Proposed, (e)PCA Gray

(a) Color source

(b) Color2Gray (c) Direct embedding (d) Poisson embedding (e) Photoshop Gray Figure 2. Color removal results “voiture” (row 1) and “house” (row 2). 1

(a) Color source

(b) Color2Gray (c) 1st local refinement (d) 2nd local refinement Figure 3. Progressive results on “harbor”.

where f (⋅, ⋅) denotes the embedding function. In this paper, we employ Poisson equation for seamless embedding. Poisson equation has been widely applied in image and graphics, such as Poisson image editing[5], Poisson matting[6], and Poisson mesh[7]. It requires φ * (Ω L ) to be the solution of the Poisson equation with Dirichlet boundary condition ∆φ * = ∆ψ over Ω L with φ * |∂Ω L = φ |∂ΩL , (2)


Given

*

− ∇ψ |2 with φ * |∂Ω =φ |∂Ω . L

L

(3)

In this way, the reconstructed image φ * interpolates the boundary conditions inwards, while following the spatial variation of the target image ψ as closely as possible. Accordingly, it effectively removes the artifact in direct embedding. The flowchart of the proposed method can be described as follows:

the initial grayscale result by φ (


For

t = 1, 2,...

-φ

(t )

0)

(Ω ) = h ( Φ (Ω))

(t )

( Ω ) = g ( Φ ( Ω )) (t ) L

(Ω ) =

(t )

(

f φ(

(t ) L

t −1)

( Ω ) ,ψ (t ) ( Ω(Lt ) ) )

(t )

( Ω ) is satisfied (t )
The final result is φ ( Ω ) = φ (Ω ) . until the grayscale φ *

boundary condition:

∫∫ | ∇φ

the color image Φ ( Ω ) , and set t = 0 .


Get

-ψ

Ω L . This is equivalent to find the solution whose gradient is the closest, in L2 -norm, to the guidance gradient ∇ψ under given

φ* Ω L

(e) 3rd local refinement

- Select the local region Ω(Lt )

where ∆(⋅) is the Laplacian operator and ∂Ω L is the boundary of

min

(f) PCA Gray

In finding h(⋅) , we adopt the following method [3]:


Convert the color image Φ ( Ω ) to CIE L*a*b color space. Denote ∆Lij , ∆Aij and ∆Bij as the channel differences between a pair of pixels (i , j ) respectively.




uuuur ∆Cij = ( ∆Aij , ∆Bij )

Given

uur vθ = ( cos θ ,sin θ )

,

,

crunch ( x ) = α tanh ( x / α ) , define the color-difference between pixel i and pixel j as

uuuur if ∆Lij > crunch || ∆Cij || uuuur uur if ∆Cij , vθ ≥ 0

(

 ∆Lij  uuuur  δ ij = crunch || ∆Cij ||  uuuur − crunch || ∆Cij || 

(

)

(




)

)

otherwise.

h : Φ ( Ω ) a φ ( Ω ) that minimizes the following energy function E ( Φ , φ ) , where

Find

a

mapping

function

κ is a set of ordered pixel pairs ( i, j ) :

∑κ ( (φ − φ ) − δ )

E (Φ, φ ) =

i

j

2

(4).

ij

( i , j )∈

Here, ξ = (α , θ , κ ) , which is the parameter set. θ controls whether chromatic differences are mapped to increases or decreases in luminance value, α determines how much chromatic variation is allowed to change the source luminance value and κ sets the neighborhood. The function g (⋅) can be found similarly, but adding

φ ( Ω L ) as constraint in eq. (4) by minimizing V (Φ,ψ ) =

∑κ ( (ψ

( i , j )∈

,

i

−ψ j ) − δ ij

)

2

+ λ ∑ (ψ i − φi

2

),

(5)

i∈B

i , j∈Ω L

where B is the boundary region between Ω and Ω L , λ is a constant for balancing the salience preserving inward the local region Ω L (the first term in eq.(5)) and the local compatibility to the global result on the boundary region B (the second term in eq.(5)) The procedure of the proposed method can be demonstrated by Fig.2. In the figure, (a) is the color image, (b) shows the initial gray images obtained by h(⋅) , which is the optimal result on the global image. The red rectangle shows the selected local regions, and (c) shows the local refinement result with direct embedding. The Poisson embedding result is shown in (d). One key idea of the proposed framework is the adoption of Poisson embedding. We compare the result before and after Poisson embedding in Fig.4, where we can see that the Poisson embedding (Fig.4b) effectively remove the artifacts when embedding directly (Fig.4a).

3. EXPERIMENTAL RESULTS We conduct experiments on a number of images: “banner", “voiture", “house" and “harbor". We first compare our method with Gooch's Color2Gray method [3]. Here we take Fig.2 as an example. Although Color2Gray can provide effective mapping over major large regions (such as slope vs. hill in “voiture", and lawn vs. car in “house"), local detail is lost during the global optimization; e.g. the trees on the hill in “voiture", and the color change on the faultage in “house" become indecipherable (see Fig.2b). Compared with Color2Gray, our method provides more satisfactory results, which achieves good results both globally and locally (Fig.2d). It also effectively eliminates the artifact on the

(a) (b) Figure 4. Direct embedding (a) v.s Poisson embedding (b) boundary which is perceivable in direct embedding (Fig.2c). The same conclusions are also shown in Fig.1. In Fig.1, although the color band is successfully transferred to gray image by Color2Gray, some key letters, such as “A" and “C", are hard to identify (Fig.1c). Using progressive framework with local refinement, both the color band and the key letter are properly transferred (Fig.1d). By using Poisson embedding, the visual quality is also well preserved with no observable artifact. In the “harbor" image, the letter “A" and “B" around the mast are fragmentary in the Color2Gray result (Fig.3b), while our method preserves their salience. We also compare our result with other color removal methods, such as Photoshop Gray and PCA-Gray in Fig.1 and Fig.2. From the figures, we can see that Photoshop Gray is prone to lose the chrominance since it uses luminance only (such as the color band in “banner", the slope vs hill in “voiture" as Fig.1b and Fig.2e show). Although PCA-Gray enhances the global contrast, yet there is no guarantee about preserving local contrast; such as the yellow leaves among the green leaves behind the house and the tree on the hill(see the marked circle in Fig.2f “house”). Also, it may bring some side effect such as violation of luminance consistency with the color source image (Fig.1e and Fig.2f “voiture”). Compared with the above two methods, Color2Gray can achieve good global results, and it gives an effective salience preserving mapping over major large regions; for instance, the contrast between the slope and the hill in “voiture", the contrast between the lawn and the car in “house" are greatly enhanced (Fig.2b). However, some local salience may be lost as the red rectangle shows in Fig.2b. Among them, our method achieves the best result. The new approach also provides a progressive framework to allow the user to refine the result step by step where the user might be interested in. As Fig.3 displays, we can first refine the mast and letters (“A" and “B") (Fig.3c), then the building (Fig.3d), and then the fire hydrant (Fig.3e) to get a satisfactory result.

4. OTHER APPLICATIONS The idea of using Poisson embedding for global/local integration can also be extended to other applications, such as image compression and reconstruction, image enhancement, and image denoising. The proposed method can also be useful for multichannel data fusion in medical image processing or remote sensing applications, since color removal can be basically deemed as a multi-channel data fusion problem.

Take image compression as an example. Fig.5a is the source image, we first apply DCT compression on the whole image with PSNR=48.5dB. Then we divide the image into blocks with size 32 by 32, and compute the mean square reconstruction error in each block. The local region is selected to be the block which contains the largest error (the red rectangle in Fig.5f), and then we apply a DCT compression on the local region at the same level of PSNR. After that, we embed the local reconstructed image into the global reconstructed result seamlessly via Poisson embedding, which is shown in Fig.5c. The error map is displayed in Fig. 5f. With the same compression rate as Fig.5c, the reconstructed result using the global DCT only is shown in Fig.5b with its error map in Fig.5e. We can see that the global/local integration method shows more promising results, from both visual sense and error map. We also present the results using refinement on three blocks which contains the top 3 largest errors (Fig.5d and its error map Fig.5g) at the same compression rate. Better results are achieved by using multiple local regions.

5. CONCLUSION AND FUTURE WORK In this paper, a new method is proposed to convert color image into grayscales. In the method we use Poisson embedding for seamless global/local integration. The result after integration combines the strengths of both global and local operations, where we greatly enhance the local salience without change of the global effect, and thereby the global-local tradeoff is elegantly avoided. The idea of using Poisson embedding for global/local integration may be valuable for many fields, such as image compression and reconstruction, image enhancement, and image denoising. Furthermore, based the proposed color removal method, we can add chrominance to the removed grayscales to produce augmented

(a)

(b)

color images. Experiments on image compression are conducted as an example and promising results are obtained. The future interesting work might be the selection of region of interested. We are now devising a smarter region selection method to avoid the boundary of the selected region boundary cutting off edges.

REFERENCES [1] A. Teschioni, C.S. Regazzoni and E. Stringa, "A Markovian approach to color image restoration based on space filling curves", ICIP, vol. 2, pp. 462-465, 1997. [2] U. Schmiedl, D.A. Orthendahl, A.S. Mark and et al. “The utility of principal component analysis for the image display of brain lesions: A preliminary, comparative study”, Magnetic Resonance in Medicine, pp. 471-486, 1987. [3] A. Gooch, S. Olsen, J. Tumblin and et al., "Color2Gray: Salience-Preserving Color Removal", SIGGRAPH 2005. [4] K. Rasche, R. Geist and J. Westall, "Re-coloring images for gamuts of lower dimension", Eurographics 2005. [5] P. Perez, M. Gangnet and A. Blake, "Poisson image editing", ACM Trans. on Graphics, pp. 313-318, 2003. [6] J. Sun, J. Jia, C.K. Tang, and et. al, “Poisson matting”, ACM Trans. on Graphics, pp. 315-321, August 2004. [7] Y. Yu, K. Zhou, D. Xu and et.al, “Mesh editing with poissonbased gradient field manipulation”, ACM Trans. on Graphics, Vol. 23, pp. 641-648, 2004. [8] X. Jiang, W.Y. Yau, “Fingerprint Minutiae Matching Based on the Local and Global Structures”, ICPR, vol. 2, 2000. [9] K. Idrissi, J. Ricard, A. Anwander and et.al, “An image retrieval system based on local and global color descriptors”, PAM, pp. 55-62, 2001. [10] D.C. Ghiglia and M.D Pritt, “Two-dimensional phase unwrapping: theory, algorithms, and software”, Wiley , 1998.

(c)

(d)

(e) (f) (g) Figure 5. Poisson embedding for Global/Local integration in image compression. (a) is the source image; (b) is the reconstructed image of Global DCT, (c) and (d) are the reconstructed image using Poisson embedded for global/local integration. 1 block with largest reconstruction errors is selected as the local region in (c) and 3 blocks with the top 3 largest reconstruction errors are selected in (d) as the red rectangles show. (e), (f), and (g) are error maps of (b), (c), and (d) respectively.