Scientific Image Processing

Weighting Estimation for TextureBased Face Recognition Using the Fisher Discriminant Texture-based automatic face recognition (AFR) methods find global similarities between two faces by computing their local regional similarities. A novel method based on Fisher discriminant analysis is proposed to estimate each region’s contribution to the global similarity score. Experimental results show that the method considerably improves recognition performance for texture-based AFR.

F

ace recognition technology is the preferred biometric method in many applications mainly because of its low intrusiveness and high accuracy.1 In some large-scale applications, such as passport or driver’s license identification, the database consists of a single frontal view for each person. In recent years, texture-based approaches are proving increasingly interesting for such applications. Many studies demonstrate the methods’ superiority over alternative face recognition approaches in applications dealing predominantly with frontal views with little variation of facial expression.2–7 In these approaches, a texture image is generated by replacing each image pixel with a binary code that represents the texture within the pixel’s neighborhood. For recognition, the texture image is partitioned into non-overlapping blocks 1521-9615/11/$26.00 © 2011 IEEE Copublished by the IEEE CS and the AIP

Raul Queiroz Feitosa, Dário Augusto Borges Oliveira, and Álvaro de Lima Veiga Filho Pontifical Catholic University of Rio de Janeiro

Raphael Pithan Brito, José Luiz Buonomo de Pinho, and Antonio Carlos Censi Montreal Informática

2



CISE-13-3-Feitindd.indd 2

and a weight is assigned to each block according to its importance in the recognition process. Following the common psychophysical findings, which indicate that some facial features (such as eyes) are more important in human face recognition than other features, Timo Ahonen and his colleagues adopted an empirical procedure to obtain a fixed set of weights based on the recognition rate.2 They recognize that the values aren’t optimal, but improve the recognition results when compared to uniform weight values. Since then, most texturebased automatic face recognition (AFR) methods8 have used fixed weight values. To our knowledge, only one previous approach has proposed adjusting the weighting to the application’s specific perturbation.9 In that approach, the optimum weighting derives from a homogeneous, nonlinear equation system solved by a least-squares technique. Experiments show that this least-squares-based method can outperform fixed, application-independent weightings proposed in the literature.9 However, the approach involves a fairly complex equation system and entails numerous training images for each estimated weight. Here, we propose a new method for weighting estimation that derives from the Fisher linear discriminant.10 In comparison to the leastsquares-based approach, our method demands Computing in Science & Engineering

15/03/11 4:57 PM

m = 8, R = 1

m = 16, R = 2

m = 24, R = 3

Figure 1. Neighboring pixels for different values of m and R in local binary patterns’ (LBPs’) image coding: (a) m = 8 and R = 1, (b) m = 16 and R = 2, and (c) m = 24 and R = 3. Database

Original images

LBP representation

LPQ representation

FERET

FEI

Figure 2. Sample images and their local binary patterns (LBP) and local phase quantization (LPQ) representations. (a) Images from the Facial Recognition Technology (Feret) database. (b) Images from Brazil’s University Center’s Faculty of Industrial Engineering (FEI) database.

fewer training samples per weight and is more efficient computationally. In tests we conducted for two texture-coding techniques on two face image databases, our method consistently outperformed the alternative strategies proposed in the literature.

Texture-Based Face Recognition

Texture-based AFR involves two main steps: face description and face matching. Here, we describe two widely used texture-coding methods and present the matching procedure that can be applied for both representation approaches. Texture-Based Face Description

The texture description is a matrix containing codes that identify one out of a predefined set of texture types. We now describe the two most widely texture coding schemes used for AFR. Face description with LBP. Local binary patterns’ (LBPs’) image representation5 is computed by assigning a code to a pixel at location x = (x, y). As Figure 1 shows, this code is related to the signs of the differences between the central pixel intensity and the intensity of its m equally spaced

May/June 2011 

CISE-13-3-Feitindd.indd 3

neighbors at a distance R. The binary values 0 and 1 are assigned to a negative and a positive difference, respectively. A bilinear interpolation is used whenever the sampling point doesn’t fall in a pixel’s center. The LBP code results from the concatenation of m 0’s and 1’s in an arbitrary but fixed order. Figure 2 shows the LBP representation of four image samples where the intensities are related to the LBP code at each pixel location. Face description with LPQ. Local phase quantiza-

tion (LPQ) is a method for textures description conceived to outperform LBP in applications where images are affected by blur and uniform illumination changes. Like LBP, in LPQ, for each pixel at location x = (x, y), a code is computed to represent the texture in the M × M neighborhood Nx centered at x (see Figure 3). Phase quantization is performed by looking at the sign of the real and imaginary values of the Fourier transform Fx(u), u = (u, v), of Nx at four low frequencies (see the white circles in Figure 3b). This generates 8 bits, which are 0 or 1 depending on whether each value is negative or positive. These bits are concatenated in an arbitrary but fixed order, forming an 8-bit integer 3

15/03/11 4:57 PM

n (0,a)

(a,a) u

(a,0)

(a,–a)

Dirjt = dirjtwT.

(b)

(a)

Figure 3. Face description using local phase quantization. (a) The neighborhood Nx. (b) The corresponding Fourier spectrum.

value that represents the texture in Nx. This procedure is carried out for all pixels in the image, bringing about the corresponding LPQ image representation. The method includes a simple procedure that decorrelates the Fourier coefficients before the quantization step; the goal is to maximize the information preserved in the texture code.6,7 Figure 2 shows LPQ coded image samples, where intensities are given by the LPQ code at each pixel location. Matching Procedure

Let’s assume hereafter that the face database consists exclusively of well-framed images with a constant interocular distance and the eyes are imaged at the same pixel coordinates. We denote with Sir the rth image of the ith subject in a database. In the recognition step the texture image is divided in equal-sized nonoverlapping blocks numbered from 1 to B. The histogram bHir of the texture codes inside the bth block is computed for b = 1, 2, …, B. The dissimilarity between histograms bHir and bHjt of the bth block respectively of images Sir and Sjt is computed by a proper distance function bdirjt(bHir, bHjt), for simplicity denoted henceforth bdirjt. To decide whether two faces are from the same subject, we use a global dissimilarity measure given by a linear combination of the computed histogram distances B

Dirjt = ∑ wb b d irjt , b=1



(1)

where the coefficients wb are weights that represent the relative importance of the region corresponding to the bth block in the recognition process.

Optimal Weighting Estimation

Our proposed solution is to estimate the wb coefficients derived from a simple reinterpretation of Equation 1. Let’s group the histogram distances 4

CISE-13-3-Feitindd.indd 4

between faces Sir and Sjt of all B blocks in a vec1 2 tor dirjt = [ dirjt, dirjt, …, Bdirjt] of a B-dimensional similarity space. Let’s further group the set of coefficients wi in the vector w = [w1, w2 , …, wB]. Equation 1 might take the following form: (2)

Therefore, the global dissimilarity measure shown in Equation 2 is the projection of the distance vector dirjt over a direction in the similarity space defined by the coefficient vector w. We assume that the optimum weighting corresponds to the similarity space’s direction, along which pairs of images of the same subject achieve maximum separation from pairs of images of different subjects. If we can plausibly assume that the covariance matrices of both classes of image pairs are equal, the problem of finding the optimum weighting boils down to a direct application of the Fisher discriminant method, whose solution is given by

w = ( d other − d same )∑ pooled , −1

(3)



where d other and d same are the mean distance vectors for pairs of images from the same and different subjects, respectively, and ∑ pooled is the pooled covariance matrix. Most books on multivariate statistical analysis, such as Applied Multivariate Statistical Analysis by Richard A. Johnson and Dean W. Wichern,10 report that Fischer’s approach usually works fine even when the equal covariance assumption doesn’t hold exactly. Equation 3’s weights don’t follow the expected left-to-right face symmetry. Nevertheless, it’s generally interesting to enforce weight symmetry to reduce the problem’s complexity by halving the number of coefficients to estimate. Assuming that blocks b and b + B/2, for B even, correspond to symmetric face regions, weight symmetry might be imposed by making bw = b+B/2w. Thus, we can write Equation 1 as B/2

Dirjt = ∑Wb b=1

(

b

)

d irjt +b+B / 2 d irjt .

(4)

Experiments

We used two databases for performance assessment. The first consists of 1,640 frontal images of 820 subjects from the Facial Recognition Technology (Feret) database’s fa and fb sets,11 with two images per subject that have slight variations in facial expression. We built the second database Computing in Science & Engineering

15/03/11 4:57 PM

LBP Feret

0.95 Uniform Fisher Ahonen Least squares

0.9

0.85

1

2

3

4

5

(a)

6

LPQ Feret

1

Recognition rate

Recognition rate

1

7

8

9

0.95

0.85

10

Uniform Fisher Ahonen Least squares

0.9

1

2

3

4

(b)

Rank

0.95

0.95

0.9 0.85 0.8

Uniform Fisher Ahonen Least squares

0.75 0.7 1

2

3

4

5

6

7

8

9

10

LPQ FEI 1

Recognition rate

Recognition rate

LBP FEI

(c)

6

Rank

1

0.65

5

7

8

Rank

9

0.9 0.85 0.75 0.7 0.65

10

Uniform Fisher Ahonen Least squares

0.8

1

2

3

(d)

4

5

6

7

8

9

10

Rank

Figure 4. Cumulative rank rates for different weightings. Weightings measured on the Feret database working with (a) LBPs and (b) LPQ, and on the FEI database working with (c) LBP and (d) LPQ.

from Brazil’s University Center’s Faculty of Industrial Engineering database (FEI; www.fei. edu.br/~cet/facedatabase.html). It consists of two frontal faces of 50 subjects with neutral and smiling expressions. In all cases, we framed the images at 80 × 64 pixel resolution,9 putting the right and left eyes at (20, 14) and (20, 51) pixel coordinates, respectively. Figure 2 shows image samples from both databases used in our experiments. We used the uniform LBP variant, with eight sampling points (m = 8) over a circle of radius equal to 2 pixels (R = 2). Following other research,6,7 we computed the Fourier transform for LPQ over each 7 × 7 pixel neighborhood (M = 7) and performed phase quantization at frequencies corresponding to a = 1/7. In both databases, we computed histograms over 10 × 8 non-overlapping blocks of size 8 × 8 pixels. In all our experiments, we randomly separated the individuals in the databases—using half for training and half for the test. The rates were measured by picking up one test image, whose matching face should be identified from all the other images of the test set. After repeating this procedure for all the test images, we computed the average rate. The following results for each configuration of database and texture coding are averages over five runs, each run with a May/June 2011 

CISE-13-3-Feitindd.indd 5

different random distribution of training and test set individuals. Recognition Performance

In our first experiment sequence, our goal was to compare our method’s recognition performance with that of other approaches. The plots in Figure 4 refer to rank recognition rates measured on the Feret and FEI databases using LBP and LPQ texture coding. Each plot contains four curves relative to four different weightings: uniform, computed by our method, proposed by Timo Ahonen and his colleagues,4 and computed according to the least-squares method.9 We resampled the weight matrix proposed by Ahonen and his colleagues so as to fit the 10 × 8 grid. As in previous work,9 with the least-squares method, we imposed the constraint that certain groups of blocks have equal weights. This was mandatory because the method didn’t generalize well for more weights with the available training faces. In Fisher’s method, we assumed face symmetry, so that only 40 coefficients were estimated and then mirrored to form a 10 × 8 weighting matrix. As Figure 4 shows, our method consistently delivered the highest performance for both databases and coding techniques. The weightings computed by the least-squares method was the second best performing method. In relation to the 5

15/03/11 4:57 PM

Table 1. Weightings obtained using the Fisher discriminant. Texture coding Local binary patterns (LBPs)

Database Feret*

FEI*

Local phase quantization (LPQ)

19

0

31

16

16

31

0

19

18

-10

15

-6

-6

15

-10

18

32

14

48

40

40

48

14

32

16

53

13

70

70

13

53

16

5

25

-7

39

39

-7

25

5

-5

18

-12

14

14

-12

18

-5

9

18

32

48

48

32

18

9

6

5

31

55

55

31

5

6

-17

-19

-12

13

13

-12

-19

-17

-11

-9

-31

22

22

-31

-9

-11

-19

-11

39

42

42

39

-11

-19

-17

-7

30

26

26

30

-7

-17

6

16

7

-7

-7

7

16

6

-9

7

2

-3

-3

2

7

-9

19

3

22

23

23

22

3

19

36

0

23

12

12

23

0

36

14

12

11

14

14

11

12

14

7

-2

2

19

19

2

-2

7

-11

14

2

4

4

2

14

-11

-11

14

-6

5

5

-6

14

-11

30

36

2

12

12

2

36

30

12

45

-5

-3

-3

-5

45

12

26

19

35

39

39

35

19

26

12

26

44

44

44

44

26

12

7

-4

6

13

13

6

-4

7

20

-15

-20

0

0

-20

-15

20

-6

1

-7

63

63

-7

1

-6

-20

-15

3

37

37

3

-15

-20

1

4

-13

55

55

-13

4

1

7

-10

-24

68

68

-24

-10

7

-12

2

-8

37

37

-8

2

-12

-22

20

-29

23

23

-29

20

-22

-2

6

-20

-15

-15

-20

6

-2

17

-6

11

-18

-18

11

-6

17

41

-3

-22

19

19

-22

-3

41

10

-6

-3

14

14

-3

-6

10

1

9

13

16

16

13

9

1

11

-1

15

10

10

15

-1

11

25

19

5

3

3

5

19

25

18

24

2

0

0

2

24

18

*Feret is the Facial Recognition Technology database and FEI is Brazil’s University Center’s Faculty of Industrial Engineering database.

Table 2. Correlation between weightings and coding techniques. Feret Feret FEI

FEI

LBP

LPQ

LBP

LPQ

LBP

1

0.80

0.59

0.42

LPQ

0.80

1

0.60

0.49

LBP

0.59

0.60

1

0.80

LPQ

0.42

0.49

0.80

1

uniform and Ahonen’s weightings, our method was clearly superior. The recognition rates for the least squares shown in Figure 4 start close to 100 percent. Any possible improvement is restricted to what’s missing to achieve 100 percent. As Figure 4 shows, with this perspective, our method’s improvement over the least-squares approach was substantial in most cases. Weights Estimates

We also analyzed the weightings’ variability that our method produced in distinct configurations. 6

CISE-13-3-Feitindd.indd 6

Table 1 shows the results for four different combinations of database and coding techniques. To improve visualization, the values were scaled so that all weighting vectors have the same magnitude; we then rounded them to the closest integer. Notwithstanding the evident differences, a certain common structure among the four weightings of Table 1 is perceptible. To measure the level of agreement among these results, we computed the correlation between each pair of weightings in Table 1. Table 2 shows the results. For clarity, we shadowed the cells containing redundant (the table is symmetric) or unimportant values (the diagonal is all 1). As expected, the weightings are indeed highly correlated, but the difference brought by changing the database, the texture coding, or both might be considerable. In the next section, we investigate how meaningful those differences are in terms of recognition performance. As Table 2 shows, the correlation is high for the same database, independent of the texture-coding method. This could imply that the weights might be more or less the same for different texture-coding Computing in Science & Engineering

15/03/11 4:57 PM

LBP Feret

LBP Feret LPQ Feret LBP FEI LPQ FEI

0.9

1

2

(a)

3 Rank

4

LBP Feret LPQ Feret LBP FEI LPQ FEI

(c)

2

3 Rank

1

2

4

(d)

4

5

0.95 LBP Feret LPQ Feret LBP FEI LPQ FEI

0.9

0.85

5

3 Rank LPQ FEI

1

0.9

1

LBP Feret LPQ Feret LBP FEI LPQ FEI

0.9

(b)

0.95

0.85

0.95

0.85

5

LBP FEI

1

Recognition rate

Recognition rate

0.95

0.85

LPQ Feret

1

Recognition rate

Recognition rate

1

1

2

3 Rank

4

5

Figure 5. Rank recognition rates for different configurations of database and coding: the plot title indicates the configuration used for rates measurement, while the legend indicates the configuration for weighting estimation.

methods. On the other hand, it’s clear that one mask can’t be generally used for any database, and should be computed for each specific database.

weighting to the kind of image variation present in the target application.

Weighting Variation and Recognition Performance

ur experiments assume that weights were symmetric following the leftto-right face symmetry. This assumption simplifies the problem by halving the number of coefficients to estimate. The results let us presume that our method can capture systematic asymmetries that might appear in face images within a given application, such as those caused by a nonsymmetric illumination pattern. Real applications may have to deal with much larger databases consisting of poor quality images. We have conducted experiments on images from a non public database provided by a Brazilian security agency that contains millions of samples. Such tests indicated that in more realistic conditions the recognition rates tend to be much lower than in this paper and the absolute gain of the proposed method over the alternative ones is expected to become even higher. Although this article focuses on texture-based approaches, we believe our proposed technique can be successful in other approaches that measure the global similarity between two faces as a linear combination of the similarity computed upon each face region.

In our final experiment, we assessed the impact of weighting variation on recognition rates. For each weighting estimated in the first experiment, we computed the rank recognition rates on the test images for all possible combinations of database and coding technique. Figure 5 shows the results. Each plot title specifies the configuration of coding technique and database upon which the rates have been measured. The curves in each plot correspond to the training sets used for weighting estimation. In every case, the highest performance was achieved when training and tests were performed on the same database and for the same texture coding. The change from LBP to LPQ or vice-versa brought slight performance changes when both training and the test were conducted on the same database. In contrast, the plots show a substantial performance loss when the weighting was estimated on one database and tested on the other. This result is consistent with the correlation analysis of weights we presented earlier. As Figure 5’s results show, significant performance gains are possible when we tune the May/June 2011 

CISE-13-3-Feitindd.indd 7

O

7

15/03/11 4:57 PM

References 1. S.Z. Li and A.K. Jain, eds., Handbook of Face

Recognition, Springer Verlag, 2005. 2. T. Ahonen, A. Hadid, and M. Pietikäinen, “Face

3.

4.

5.

6.

7.

8.

9.

10. 11.

Recognition with Local Binary Patterns,” Proc European Conf. Computer Visions (ECCV), LNCS 3024, Springer, 2004, pp. 469–481. G. Heusch, Y. Rodriguez, and S. Marcel, “Local Binary Patterns as an Image Preprocessing for Face Authentication,” Proc. 7th Int’l Conf. Automatic Face and Gesture Recognition, IEEE Press, 2006, pp. 9–14. T. Ahonen, A. Hadid, and M. Pietikäinen, “Face Description with Local Binary Patterns: Application to Face Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28 no. 12, 2006, pp. 2037–2041. T. Ahonen, A. Hadid, and M. Pietikäinen, “Face Recognition with Local Binary Patterns,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 12, 2006; doi:10.1109/TPAMI.2006.244. V. Ojansivu and J. Heikkilä, “Blur Insensitive Texture Classification Using Local Phase Quantization,” Proc. Int’l Conf. Image and Signal Processing, LNCS 5099, Springer Verlag, 2008, pp. 236–243. T. Ahonen et al., “Recognition of Blurred Faces Using Local Phase Quantization,” Proc. Int’l Conf. Pattern Recognition, IEEE Press, 2008, pp. 1–4. S. Liao et al., “Facial Expression Recognition Using Advanced Local Binary Patterns, Tsallis Entropies, and Global Appearance Features,” Proc. Int’l Conf. Image Processing, IEEE Press, 2006, pp. 665–668. R.Q. Feitosa et al., “Optimal Region Weighting for Local Binary Pattern-Based Face Recognition,” Proc. Biosignals and Biorobotics Conf (ISSNIP), IGI Publishers, 2010, pp. 204–208. R.A. Johnson and D.W. Wichern, Applied Multivariate Statistical Analysis, Prentice Hall, 1998. W. Zhao et al., “Face Recognition: A Literature Survey,” ACM Computing Surveys, vol. 35, no. 3, 2003, pp. 399–458.

Raul Queiroz Feitosa is an associate professor in the Department of Electrical Engineering at the Pontifical Catholic University of Rio de Janeiro, Brazil. His research interests include image analysis, biometrics, remote sensing, computer vision, and pattern recognition. Feitosa has a Dr-Ing in computer science from the University of Erlangen Nuremberg in Bavaria, Germany. Contact him at [email protected].

8

CISE-13-3-Feitindd.indd 8

Dário Augusto Borges Oliveira is doctoral student at the Pontifical Catholic University of Rio de Janeiro. His research interests include image analysis, medical imaging, biometrics, remote sensing, and pattern recognition. Oliveira has an MS in electrical engineering from the Pontifical Catholic University of Rio de Janeiro. Contact him at [email protected]. Álvaro de Lima Veiga Filho is an associate professor in the Department of Electrical Engineering at the Pontifical Catholic University of Rio de Janeiro. His research interests include nonlinear temporal series, multivariate analysis, empirical finance, risk management and modeling, and stochastic statistics. Filho has a doctoral degree in signal processing from École Nationale Supérieure des Télécommunications (Telecom ParisTech). Contact him at [email protected]. Raphael Pithan Brito is a software analyst for Montreal Informática in Rio de Janeiro. His research interests include biometrics, computer vision, digital image processing, and face recognition. Brito has a computer engineer’s degree from the Pontifical Catholic University of Rio de Janeiro. Contact him at [email protected]. José Luiz Buonomo de Pinho is a senior consultant at Montreal Informática and a post-graduate student in the Department of Electrical Engineering at Pontifical Catholic University of Rio de Janeiro. His research interests include biometrics, image analysis, and large databases. Pinho has a civil engineer’s degree from Veiga de Almeida University in Rio de Janeiro. Contact him at [email protected]. Antonio Carlos Censi is the director of technology at Montreal Informática. His research interests include biometrics and its use for identification, image analysis, and large databases. Censi has an electrical engineer’s degree from Pontifical Catholic University of Rio de Janeiro. Contact him at accensi@montreal. com.br.

Selected articles and columns from IEEE Computer Society publications are also available for free at http://ComputingNow.computer.org.

Computing in Science & Engineering

15/03/11 4:57 PM