Interactive Image Colorization using Laplacian Coordinates Wallace Casaca12 , Marilaine Colnago1 , Luis Gustavo Nonato1 1

Institute of Mathematics and Computer Sciences - University of S˜ ao Paulo (USP), 13566-590, S˜ ao Carlos, SP, Brazil 2

School of Engineering - Brown University, 182 Hope St., Providence, RI 02912, United States

Abstract. Image colorization is a modern topic in computer vision which aims at manually adding colors to grayscale images. Techniques devoted to colorize images differ in many fundamental aspects as they often require an excessive number of image scribbles to reach pleasant colorizations. In fact, spreading lots of scribbles in the whole image consists of a laborious task that demands great efforts from users to accurately set appropriate colors to the image. In this work we present a new framework that only requires a small amount of image annotations to perform the colorization. The proposed framework combines the highadherence on image contours of the Laplacian Coordinates segmentation approach with a fast color matching scheme to propagate colors to image partitions. User can locally manipulate colored regions so as to further improve the segmentation and thus the colorization result. We attest the effectiveness of our approach through a set of practical applications and comparisons against existing colorization techniques.

1

Introduction

Colorization is a computer-supervised process by which colors are imparted to grayscale images or to black-and-white films. It has been successfully used in photo editing and scientific illustration, to modernize old motion pictures and to enhance the visual appear of an image. Traditionally, colorization is tedious, time consuming and it requires artistic skills to precisely set suitable colors to an off-color image. Aiming at making the colorization task simpler and less laborious, several computational systems have been proposed in the last two decades, which can be roughly divided into two major groups: example-based [1–6], and scribblebased [6–11]. Example-based methods accomplish the colorization process by matching the luminance of the monochromatic image with the luminance of a reference color image used to drive the color propagation. In scribble-based methods, the user guides the colorization by manually defining colored strokes onto the grayscale image.

Considering the flexibility to operate arbitrary colorizations and the nonrequirement for an additional reference image, scribble-based strategy has performed better than the example-based one in recent years [12]. This trend has been observed especially due to the simplicity of scribble-based methods which basically relies on an interactive interface in order to operate. The classical work by Levin et al. [7] is a good representative of scribblebased approach. Levin’s method aims at optimizing the color of all image pixels using the scribbles as constraints. Although it presents satisfactory results for various types of images, Levin’s approach tends to convey colors beyond the texture boundaries, thus resulting in unpleasant colorizations. The technique proposed by Yi-Chin Huang et al. [8] employs adaptive edge detection so as to prevent colors from going beyond region boundaries. Further improvements of this technique have been proposed by Yatziv and Sapiro [9], who present a faster scribble-based color optimization method that relies on chrominance blending to perform the colorization. In [10] and [11], the authors employ texture continuity to colorize manga-cartoons and natural images, respectively. In a more recent work, Casaca et al. [6] has introduced an innovative user-based interface namely ProjColor that relies on a simple drag-and-drop manipulation of badly colorized pixels using multidimensional data projection as an recursive tool. Despite good results, most existing scribble-based methods require intensive user involvement, especially when the image contains complex structures or has different texture patterns, which can demand lots of scribbles until acceptable outcomes are reached. In this work we propose a new framework for colorizing grayscale images that makes use of a scribble-driven interface to replace the excessive provision of user strokes typically employed by existing colorization methods. Moreover, the proposed approach holds the good segmentation properties derived from the Laplacian Coordinates (LC) methodology [13, 14]. Since Laplacian Coordinates is used to precompute a prior segmentation of the monochromatic image, our framework leads to pleasant results and requires just a few user interventions to fully colorize the image. As we shall show, by one-shot stroking the grayscale image, the user can colorize complex textured areas quite easily, preserving region boundaries and preventing the addition of new scribbles. 1.1

Contribution

In summary, the main contributions of this work are: – A novel interactive colorization technique that combines the accuracy of the Laplacian Coordinates approach with a fast color propagation scheme to colorize images. – An easy-to-implement and efficient framework that allows for recursively colorizing the image by reintroducing new seeds in an intuitive and nonlaborious manner. – A comprehensive set of practical applications typically performed by professional photo editors which shows the effectiveness of the proposed approach.

Fig. 1. Pipeline of our colorization framework.

2

Laplacian Coordinates-based Colorization

As illustrated in Figure 1, the proposed colorization pipeline comprises three main steps, namely, prior segmentation, color assignment and progressive colorization. First, color scribbles given by the user are taken as constraints to the Laplacian Coordinates approach aiming at producing a prior segmentation. The partitioning obtained is then used to promote color data transfer between input scribbles and image segments. Badly colored regions can be modified by interacting with the Laplacian Coordinates segmentation interface and our colorization apparatus in order to produce better outcomes. Details of each stage of the pipeline are presented bellow. 2.1

Prior Segmentation

In our approach, we use the Laplacian Coordinates methodology [13] to assist the colorization process by fragmenting the image into disjoint regions. Color labels are manually chosen by the user and freely spread inside representative image regions, as illustrated in Figure 2(b). Those labels are then designed to condition the linear system of equations obtained from the Laplacian Coordinates energy function: F(x) =



i∈B

xi 22 +

 i∈F

xi − 122 +

1 wij (xi − xj )22 , 2 i∼j

(1)

(a)

(b)

(c)

(d)

Fig. 2. Illustration of the segmentation procedure. (a) Grayscale image, (b) marked image with color scribbles, (c) prior segmentation, and (d) the resulting colorization after applying color transfer.

where x = (x1 , x2 , ..., xn ) is the sought solution, that is, a saliency map which assigns a scalar to each pixel i of the image, wij denotes the weight of pixel pair (i, j) locally determined by a regular 9-point stencil, and B and F represent the sets of labeled pixels (we assume a binary segmentation to simplify the notation). In less mathematical terms, the unitary components in F enforce fidelity of brushed pixels to scalars 0 (background ) and 1 (foreground ), and the pairwise term imposes spatial smoothness within image regions while promoting sharp jumps across image boundaries. Energy (1) is efficiently computed by solving a sparse linear system of equations as detailed in [13, 14]. Weights wij are calculated from an image gradientbased function such as [13, 15]. Finally, the segmentation can be achieved by specifying an image cutting criterium. For instance, one can obtain partitions by trivially assigning foreground and background labels as follows: Si =



1, if xi ≥ 12 . 0, otherwise

(2)

The partitions generated (see Figure 2(c)) are then used to support the next step of our pipeline, color assignment, which is described below.

2.2

Color Assignment

This stage is responsible for propagating the colors initially chosen by the user to the partitions generated by the LC segmentation. The propagation mechanism is accomplished as follows: given the set of color labels provided during the segmentation stage, we first convert those labels to Lαβ coloring system by employing basic matrix transformations as outlined in [16]. The L channel in Lαβ space determines the luminance of the labels and it does not carry any color information as the remaining components. Moreover, no correlation is established between L, α and β. As a result, if a grayscale pixel p is labeled as “background”, that is, Sp = 0 in Equation (2), its color coordinates α and β are obtained by taking the corresponding components in the specified color label. Similar procedure is performed until colorizing the remaining partitions (see the middle step in Figure 1). 2.3

Progressive Colorization

One of the main contributions of the proposed framework is to exploit the flexibility provided by the Laplacian Coordinates methodology to interactively promote progressive colorization. Similar to [17], Laplacian Coordinates enables an interactive tool that allows for repartitioning data by inserting new seeded pixels. In fact, if the result is not satisfactory, the user can select badly colored pixels, turning them into a different color label that can be reintroduced into the Laplacian Coordinates system of equations to partition the image and, thereby, improve the resulting colorization.

(a)

(b)

(d)

(c)

(e)

Fig. 3. The use of our colorization framework when allowing for user recolorization. (a) Input image, (b) initial scribbles, (c) 1st colorization, (d) improvement performed by the user, and (e) colorization after user interaction.

Figure 3 illustrates the need for user intervention. Notice that the group of pixels located on the upper right corner of the boy’s tray was not colored suitably (see Fig. 3(c)). User can then provide an additional color scribble to the region with badly colorized pixels as highlighted in Fig. 3(d), creating new constraints for the Laplacian Coordinates and, thus generating a better result as shown in Fig. 3(e).

3

Experimental Results

In this section we illustrate the use of the proposed approach in practical scenarios such as multiple colorization and portraiture. We also provide experimental comparisons against traditional scribble-based methods [6, 7, 9, 12]. 3.1

Multiple Color Substitution and Portraiture

Our first experiment shows the capability of the proposed framework in producing different visual effects by just keeping the initial strokes (Fig. 4(a)) and modifying user-selected colors. Changing the colors that guide the colorization gives rise to multiple representations of the image, as shown in Fig. 4(b)-(d). Selective colorization problem (portraiture) is investigated in Fig. 4(e)-(k). This application aims at accentuating certain features on a photography so that the vintage aspect of the image is preserved. Notice from Fig. 4(e)-(f) that no excessive image annotations were required to reach a pleasant result. Another example of portraiture is presented in Fig. 4(g)-(k), where the eyes and lips were successfully elucidated. 3.2

Qualitative Comparisons

Figure 5 compares the proposed approach against Levin’s and Projcolor methods. In contrast to Fig. 5(b), Projcolor and our technique have produced similar results, however, our approach does not make use of any data exploratory interface as the one used by Projcolor algorithm. Figure 6 brings another comparison between Levin’s and our framework, but now taking into account the effectiveness of both colorization scribble interfaces. The seeding mechanism provided by Laplacian Coordinates is simpler than the traditional scribble-based employed in [7], as the user does not need to spread an excessive number of scribbles in the image to reach a reasonable result. The experiment presented in Figure 7 establishes comparisons between the proposed approach and the popular scribble-based methods [7, 9, 12]. Colorizations produced from [7] and [9] smoothed the images considerably almost all cases while the outcomes obtained from [12] and our technique have produced more refined results. By reintroducing just a small amount of seeds in the marked images in Fig. 7(g),(o) and Fig. 7(w), one can see that our approach is quite flexible in capturing intrinsic details of the image such as pieces surrounded by image segments, a characteristic not present in the technique [12].

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(j)

(i)

(k)

Fig. 4. Practical applications supported by the proposed framework. (a) Input scribbles used by our method, (b)-(d) the resulting colorizations taking as input the scribbles provided in (a) and varying multiple colors, (e)-(f) the scribbled image and its selective colorization, (g)-(k) portraiture example performed by our framework.

(a)

(b)

(c)

(d)

Fig. 5. Comparison between Levin’s technique [7], Projcolor [6], and our approach. (a) Initial scribbles, (b) Levin’s colorization, (c) Projcolor colorization after some user involvement, and (d) result obtained by our framework in a one-shot colorization.

(a)

(b)

(c)

(d)

Fig. 6. Comparison between [7] and our technique. (a) Scribbles used by Levin’s method, (b) Levin’s colorization, (c) scribbles used by our method, and (d) our result.

(a) Original

(b) Marked

(e) Yao et al.

(f) Our result

(i) Original

(j) Marked

(m) Yao et al.

(n) Our result

(q) Original

(r) Marked

(u) Yao et al.

(v) Our result

(c) Levin et al.

(d) Yatziv & Sapiro

(g) New scribbles (h) Updated result

(k) Levin et al.

(l) Yatziv & Sapiro

(o) New scribbles (p) Updated result

(s) Levin et al.

(t) Yatziv & Sapiro

(w) New scribbles (x) Updated result

Fig. 7. Comparison between colorization techniques [7, 9, 12] and our approach.

3.3

Quantitative Comparisons

Finally, in this section we make use of multiple image quality metrics traditionally employed by the computer vision community to quantitatively evaluate the effectiveness of the proposed approach against the well-established colorization techniques [7, 9, 12]. Tables 1 and 2 summarize the quantitative measurements of Mean Absolute Error (MAE), Peak-to-Noise-Ratio (PSNR), Structural Similarity Index (SSIM) [18], and Universal Image Quality Index (UIQI) [19] between ground-truth color images and the colorizations produced by the algorithms from Fig.7. One notices that our method outperforms others (in average) for all evaluated metrics, being also superior for most of the measurements (bold values indicate better results). Table 1. Quantitative comparison against [7], [9] and [12] when computing MAE (↓) and PSNR (↑) measures for images from Fig. 7. Bold values indicate the best score. Levin MAE Church 20.24 Horse 99.75 River 74.16 Average 64.71 Image

et al. PSNR 35.06 28.14 29.42 30.87

Yatziv-Sapiro MAE PSNR 23.70 34.38 89.28 28.62 72.02 29.55 61.66 30.85

Yao et al. MAE PSNR 18.13 35.54 57.58 30.53 72.94 29.50 49.55 31.85

Proposed MAE PSNR 12.95 37.00 42.39 31.85 18.26 35.51 24.53 34.78

Updated MAE PSNR 11.73 37.43 42.27 31.87 18.19 35.53 24.06 34.94

Table 2. Quantitative comparison against [7], [9] and [12] when computing SSIM (↑) and UIQI (↑) measures for images from Fig. 7. Bold values indicate the best score. Levin SSIM Church 0.8373 Horse 0.2042 0.8461 River Average 0.6292 Image

4

et al. UIQI 0.9065 0.7828 0.9392 0.8762

Yatziv-Sapiro SSIM UIQI 0.8566 0.9153 0.7243 0.9878 0.8601 0.9438 0.8137 0.9490

Yao et al. SSIM UIQI 0.8404 0.9097 0.8228 0.9925 0.8500 0.9421 0.8377 0.9481

Proposed SSIM UIQI 0.8440 0.9379 0.8259 0.9920 0.8555 0.9407 0.8418 0.9569

Updated SSIM UIQI 0.8443 0.9392 0.8258 0.9916 0.8640 0.9551 0.8447 0.9620

Conclusion

In this work we address the fundamental problem of image colorization as an interactive framework that unifies scribble-based image partition and recursive colorization. Besides enabling a local modification of badly colored regions, the combination of Laplacian Coordinates approach, fragment-based colorization and scribble-driven mechanism turns out to be effective in popular practical

applications such as progressive retouching, multiple colorization and portraiture. The experimental results we provided shows that the proposed framework outperforms existing representative techniques in terms of accuracy, flexibility and quantitative measurement, rendering it a very attractive interactive tool in the context of image colorization.

Acknowledgments The authors would like to thank the anonymous reviewers for their constructive comments. This research has been funded by FAPESP (the State of S˜ao Paulo Research Funding Agency, grants #2014/16857-0 and #2011/22749-8), and CNPq (the Brazilian Federal Research Funding Agency).

References 1. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David Salesin. Image analogies. In ACM Transactions on Graphics (TOG), pages 327– 340, 2001. 2. Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer Graphics and Applications, 21(5):34–41, 2001. 3. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. Transferring color to greyscale images. ACM Transactions on Graphics (TOG), 21(3):277–280, 2002. 4. Revital Irony, Daniel Cohen-Or, and Dani Lischinski. Colorization by example. In Proc. of the Eurographics Symposium on Rendering, pages 201–210, 2005. 5. Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong, Stephen Lin, Chi-Sing Leung, and Pheng-Ann Heng. Intrinsic colorization. ACM Transactions on Graphics (TOG), 27(5):152:1–152:9, 2008. 6. Wallace Casaca, Erick Gomez-Nieto, Cinthya de O.L. Ferreira, Geovan Tavares, Paulo Pagliosa, Fernando Paulovich, Luis Gustavo Nonato, and Afonso Paiva. Colorization by multidimensional projection. In 25th Conference on Graphics, Patterns and Images (SIBGRAPI), pages 32–38. IEEE Computer Society, 2012. 7. Anat Levin, Dani Lischinski, and Yair Weiss. Colorization using optimization. ACM Transactions on Graphics (TOG), 23(3):689–694, 2004. 8. Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. An adaptive edge detection based colorization algorithm and its applications. In Proc. of the 13th ACM International Conference on Multimedia, pages 351–354, 2005. 9. Liron Yatziv and Guillermo Sapiro. Fast image and video colorization using chrominance blending. IEEE Transactions on Image Processing, 15(5):1120–1129, 2006. 10. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. Manga colorization. ACM Transactions on Graphics (TOG), 25(3):1214–1220, 2006. 11. Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and HeungYeung Shum. Natural image colorization. In Proc. of the Eurographics Symposium on Rendering, pages 309–320, 2007. 12. Chen Yao, Xiaokang Yang, Li Chan, and Yi Xu. Image colorization using bayesian nonlocal inference. Journal of Electronic Imaing, 20(2):023008–023008–6, 2011.

13. Wallace Casaca, Luis Gustavo Nonato, and Gabriel Taubin. Laplacian coordinates for seeded image segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 384–391. IEEE Computer Society, 2014. 14. Wallace Casaca. Graph Laplacian for Spectral Clustering and Seeded Image Segmentation. Ph.d. thesis, University of S˜ ao Paulo (ICMC-USP), Brazil, 2014. 15. Wallace Casaca, Afonso Paiva, Erick Gomez-Nieto, Paulo Joia, and Luis Gustavo Nonato. Spectral image segmentation using image decomposition and inner product-based metric. Journal of Mathematical Imaging and Vision, 45(3):227– 238, 2013. 16. L.A. Torres Mendez, C.A. Ramirez-Bejarano, G. Ortiz-Alvarado, and C.A. de AlbaPadilla. A fast color synthesis algorithm using the l-alpha-beta color space and a non-parametric mrf model. In 8th Mexican International Conference on Artificial Intelligence (MICAI), pages 53–58, 2009. 17. Wallace Casaca, Danilo Motta, Gabriel Taubin, and Luis Gustavo Nonato. A userfriendly interactive image inpainting framework using laplacian coordinates. In IEEE International Conference on Image Processing (ICIP), pages 1–5, 2015. 18. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, pages 600–612. 19. Zhou Wang and A.C. Bovik. A universal image quality index. IEEE Signal Processing Letters, 9(3):81–84, 2002.