Segmentation of Mosaic Images based on Deformable Models using Genetic Algorithms Alberto Bartoli Gianfranco Fenu Eric Medvet Felice Andrea Pellegrino Nicola Timeus Department of Engineering and Architecture University of Trieste Italy
Goodtechs, 30/11–1/12 2016, Venice (Italy) http://machinelearning.inginf.units.it
Motivation
Table of Contents
1
Motivation
2
Our solution
3
Experimental evaluation
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
2 / 20
Motivation
Mosaics
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
3 / 20
Motivation
Mosaic preservation and restoration Currently, based on photographic or manual acquisition. Photographic
1:1 manual acquisition
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
4 / 20
Motivation
Mosaic preservation and restoration Currently, based on photographic or manual acquisition. Photographic + cheap and quick + born digital, can be easily stored + can acquire mosaics in hardly reachable or dangerous locations
1:1 manual acquisition
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
4 / 20
Motivation
Mosaic preservation and restoration Currently, based on photographic or manual acquisition. Photographic + + + −
cheap and quick born digital, can be easily stored can acquire mosaics in hardly reachable or dangerous locations outcome: tessellae details are missing
1:1 manual acquisition
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
4 / 20
Motivation
Mosaic preservation and restoration Currently, based on photographic or manual acquisition. Photographic + + + −
cheap and quick born digital, can be easily stored can acquire mosaics in hardly reachable or dangerous locations outcome: tessellae details are missing
1:1 manual acquisition + outcome: tessellae details are included
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
4 / 20
Motivation
Mosaic preservation and restoration Currently, based on photographic or manual acquisition. Photographic + + + −
cheap and quick born digital, can be easily stored can acquire mosaics in hardly reachable or dangerous locations outcome: tessellae details are missing
1:1 manual acquisition + − − −
outcome: tessellae details are included costly and long not digital cannot be applied in dangerous locations
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
4 / 20
Motivation
1:1 manual acquisition
Figure: Mosaic“del Buon Pastore”, Aquileia, manual acquisition dated 1992
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
5 / 20
Motivation
Digital acquisition: problem statement Input: an image I of the mosaic Output: a segmentation S(I ) where regions R ∈ S(I ) should correspond to tessellae
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
6 / 20
Motivation
Digital acquisition: problem statement Input: an image I of the mosaic Output: a segmentation S(I ) where regions R ∈ S(I ) should correspond to tessellae
Input I : photographic acquisition
Bartoli et al. (UniTs)
Output S(I ): mosaic segmentation
Mosaic Segmentation with GA
6 / 20
Motivation
Effectiveness indexes
“Regions should correspond to tessellae”. . . W.r.t. a manual segmentation (ground truth) TI of I : P maxR∈S(I ) |R∩T | Precision: Prec(S(I ), TI ) = |T1I | T ∈TI |R| maxR∈S(I ) |R∩T | 1 P Recall: Rec(S(I ), TI ) = |TI | T ∈TI |T | (together F-measure) Count error: Count(S(I ), TI ) =
abs(|TI |−|S(I )|) |TI |
(previously proposed1 for the specific problem)
1
Fenu et al. 2015. Bartoli et al. (UniTs)
Mosaic Segmentation with GA
7 / 20
Motivation
Precision and Recall
T
T R
T
R
T R
Precision
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
R
Recall
8 / 20
Our solution
Table of Contents
1
Motivation
2
Our solution
3
Experimental evaluation
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
9 / 20
Our solution
Genetic Algorithm
A population of candidate solutions (individuals) is evolved through mutation and recombination of the fittest individuals Individual ≡ segmentation, represented as a fixed-length bit array Multiobjective fitness
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
10 / 20
Our solution
Individual representation
Individual ≡ segmentation, represented as a fixed-length bit array 001010101110101010100101111010111101010101010110. . . 1001101110
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
11 / 20
Our solution
Individual representation
Individual ≡ segmentation, represented as a fixed-length bit array 001010101110101010100101111010111101010101010110. . . 1001101110 One gene gi for each region Ri Number of genes has to be set in advance (by estimation)
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
11 / 20
Our solution
Individual representation
Individual ≡ segmentation, represented as a fixed-length bit array 001010101110101010100101111010111101010101010110. . . 1001101110 One gene gi for each region Ri Number of genes has to be set in advance (by estimation)
Gene encodes position and shape Position w.r.t. reference point (on a grid) Two variants for the shape
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
11 / 20
Our solution
Gene Rotated and deformed square
Rotated rectangle gi = (xi , yi , φi , l1i , l2i )
l2i
gi = (xi , yi , φi , li , dx1i , dy1i , . . . , dx4i , dy4i )
dx1i Φi
Φi
dy1i
(xi,yi)
(xi,yi)
l1i
li
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
12 / 20
Our solution
Fitness
Precision and recall cannot be used, since a manual segmentation is not available Instead, two other indexes: In-tile color dissimilarity (to minimize) how uniform is the color of the region? region = tessella =⇒ uniform
Out-tile color dissimilarity (to maximize) how uniform is the color upon the border of the region? region = tessella =⇒ not uniform
2
Fenu et al. 2015. Bartoli et al. (UniTs)
Mosaic Segmentation with GA
13 / 20
Our solution
Fitness
Precision and recall cannot be used, since a manual segmentation is not available Instead, two other indexes: In-tile color dissimilarity (to minimize) how uniform is the color of the region? region = tessella =⇒ uniform
Out-tile color dissimilarity (to maximize) how uniform is the color upon the border of the region? region = tessella =⇒ not uniform
Conflicting → multiobjective Shown2 to be correlated with difficulty of segmentation
2
Fenu et al. 2015. Bartoli et al. (UniTs)
Mosaic Segmentation with GA
13 / 20
Our solution
Fitness: intuition In-tile color dissimilarity (to minimize)
High
Low
Out-tile color dissimilarity (to maximize)
High Bartoli et al. (UniTs)
Low
Mosaic Segmentation with GA
14 / 20
Experimental evaluation
Table of Contents
1
Motivation
2
Our solution
3
Experimental evaluation
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
15 / 20
Experimental evaluation
Dataset Five mosaics, manually annotated Different age and style Already used in previous work3
Bird
3
Church
Flower
Museum
University
Fenu et al. 2015. Bartoli et al. (UniTs)
Mosaic Segmentation with GA
16 / 20
Experimental evaluation
Reference
Lamia Benyoussef and St´ephane Derrode (2008). “Tessella-oriented segmentation and guidelines estimation of ancient mosaic images”. In: Journal of Electronic Imaging 17.4, pp. 043014–043014 the only method specifically designed for mosaic segmentation radically different approach
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
17 / 20
Experimental evaluation
Results With “rotated and deformed square” variant: Mosaic
Count
Our method Prec Rec
Fm
Count
TOS Prec Rec
Fm
Bird Church Flower Museum University
0.01 0.03 0.07 0.03 0.03
0.411 0.418 0.503 0.503 0.459
0.659 0.629 0.626 0.760 0.674
0.506 0.502 0.558 0.605 0.546
0.03 0.54 0.06 0.14 0.90
0.528 0.564 0.494 0.644 0.632
0.817 0.719 0.678 0.873 0.785
0.642 0.632 0.572 0.741 0.701
Average
0.03
0.459
0.669
0.544
0.33
0.572
0.774
0.658
W.r.t. TOS:
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
18 / 20
Experimental evaluation
Results With “rotated and deformed square” variant: Mosaic
Count
Our method Prec Rec
Fm
Count
TOS Prec Rec
Fm
Bird Church Flower Museum University
0.01 0.03 0.07 0.03 0.03
0.411 0.418 0.503 0.503 0.459
0.659 0.629 0.626 0.760 0.674
0.506 0.502 0.558 0.605 0.546
0.03 0.54 0.06 0.14 0.90
0.528 0.564 0.494 0.644 0.632
0.817 0.719 0.678 0.873 0.785
0.642 0.632 0.572 0.741 0.701
Average
0.03
0.459
0.669
0.544
0.33
0.572
0.774
0.658
W.r.t. TOS: smaller count error implicitly chosen by user in our method
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
18 / 20
Experimental evaluation
Results With “rotated and deformed square” variant: Mosaic
Count
Our method Prec Rec
Fm
Count
TOS Prec Rec
Fm
Bird Church Flower Museum University
0.01 0.03 0.07 0.03 0.03
0.411 0.418 0.503 0.503 0.459
0.659 0.629 0.626 0.760 0.674
0.506 0.502 0.558 0.605 0.546
0.03 0.54 0.06 0.14 0.90
0.528 0.564 0.494 0.644 0.632
0.817 0.719 0.678 0.873 0.785
0.642 0.632 0.572 0.741 0.701
Average
0.03
0.459
0.669
0.544
0.33
0.572
0.774
0.658
W.r.t. TOS: smaller count error implicitly chosen by user in our method
lower Fm Bartoli et al. (UniTs)
Mosaic Segmentation with GA
18 / 20
Experimental evaluation
Segmentation example
Our method
TOS
Our method detects the filler
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
19 / 20
Experimental evaluation
Future work
Finding a better starting point for reference points Consider and manage overlapping Image pre-processing
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
20 / 20
Experimental evaluation
Thanks!
Bartoli et al. (UniTs)
Mosaic Segmentation with GA
20 / 20
