Eye Localization via Eye Blinking Detection Jerry Chi-Ling Lam (994873428) Department of Electrical and Computer Engineering, University of Toronto, 10 Kings College Road Toronto, Canada M5S 3G4 [email protected]

Abstract In this paper, the goal is to localize eyes by determining the eye blinking motions using optical flow happened in a sequence of images. Frame differencing is used to find all possible motion regions that satisfy the eye blinking motion constraints. Furthermore, using Optical flow’s direction and its magnitude, it is possible to determine whether a blink has occurred. Experiments are performed under a controlled environment and the proposed algorithm achieves a detection rate of 89% with a false alarm rate of 17%. Compared with other two approaches used in the literatures [1] and [2], the proposed algorithm provides a better control of tradeoff between detection rate and false alarm rate.

1

Introduction

Eye localization is important for many areas. For example, it could be useful for face normalization [3], gaze-based human-computer interfaces [4], and security systems using the human iris for identification [5]. Eye localization via eye blinking detection is a sensible approach because eye blink is a natural physiological response which occurs frequently. Using motion analysis to detect eye blink is not new either: in Yano et al. [1], frame differencing is used for blink detection. In Tracy et al. [2], optical flow is used to detect blink. This is due to the fact that an eye blink is very distinct from all other motions on the face; it takes only about 0.35 second for a full blink to occur. In this study, the goal is to combine frame differencing technique and optical flow technique, but instead of using a simple iterative Lucas-Kanade algorithm to compute optical flow as in [2], here multi-resolution Lucas Kanade algorithm described in [6] will be used to improve the accuracy of the optical flow.

The proposed algorithm is more robust because frame differencing is used to localize motion regions. Then optical flow is used to infer the magnitude and the direction of the motion. By using the magnitude and the direction of each motion region, we can classify the motion area is due to eye blinking motion or other facial motions. Since the classification depends on the optical flow, it is very important that the pattern of the eye blinking optical flow is consistent all the time. It is experimentally proven that the multi-resolution Lucas-Kanade algorithm captures the motion of the eye blinking better than the iterative LucasKanade. With a consistent optical flow, the classification task is straightforward. It is expected that the system would allow the subject relative freedom of motion. The paper starts with a description of the environment that the algorithm has been tested. In section 3, a short introduction to the multi-resolution Lucas-Kanade will be given. In section 4, the eye localization algorithm will be presented and lastly, in section 5, the experimental results will be discussed in details and a side-by-side comparison between the frame differencing only technique and optical flow only technique using Lucas-Kanade will be given followed by a summary of their weaknesses and strengths for each method.

2

The environment

The sequences of grayscale images (192 x 256 pixels) for training and testing are obtained in a controlled environment with 2 infrared light sources and a camera of 30 frames/sec which bypasses only the infrared wavelength as seen in figure 1. The intensity of the light was adjusted to ensure that the subject face is well-illuminated. Partially occluded eyes and eyes with glasses are excluded from the experiment. The algorithm will be tested on sequence of images that both eyes are in the field of view of the camera and there are no drastic movements in consecutive frames. The subject is allowed to move freely as long as the full face is always presented in the images. Experimental results focus mainly on the problem of clas-

sification between eye blink motion and a fixed set of noneye blinking motions such as head movements.

neighbors have the same velocity. With a window size of 5 x 5, we will have 25 equations per pixel as illustrated below.

Minimum least squares solution given by solution (in d) of: Figure 1: images used for testing

3

Optical Flow Overview

Optical flow has the following assumptions. 1. Image intensity in a small region remains the same although their location may change. 2. Neighboring points in the scene typically belong to the same surface and hence typically have similar motions and since they also project to nearly points in the image, we expect spatial coherence in image flow. 3. The image motion of a surface patch changes gradually over time. Under these 3 basic assumptions, the optical flow for 1-D case can be computed easily using the following pseudocode. For each pixel p of interest: Compute local image derivative at p: Ix Initialize velocity vector: vp = 0 Repeat until convergence: Compensate for current velocity vector: I’(x,t+1) = I(x+ vp,t+1) Compute temporal derivative: It = I’(p,t+1) – I(p,t) Update velocity vector: vp = vp - It / Ix end

The problem becomes slightly more complicated in 2-D case. This is due to the aperture problem which states that with one point, we can only detect movement perpendicular to the brightness gradient. Therefore, we get at most “Normal Flow”. Lucas-Kanade [7] came up with an algorithm which will be presented next to solve this problem.

a. Iterative Lucas-Kanade Optical Flow The basic idea to solve the aperture problem is to impose additional constraints by assuming that the flow field is smooth locally. Therefore, we can pretend the pixel’s

The problem of Lucas–Kanade optical flow is that it requires to have a very large window size and iterate multiple times on this scheme to get a robust solution for large motions. However, large window size decreases the accuracy of optical flow. Intuitively, for eye blinking motion, only the eyelid motion is of interest and therefore, using a large window is doomed to fail because it “smooth out” the details. To solve this problem, multi-resolution LucasKanade algorithm [6] will be used instead.

b. Multi-resolution Lucas Kanade Algorithm This algorithm solves the fast motion problem by reducing the resolution of the image and compute the optical flow in different scale. The central motivation behind this is to be able to handle large pixel motions (larger than the window size). In each scale, starting from the coarse level as in figure 2, the simple iterative Lucas-Kanade algorithm will be used to provide sufficient local optical flow, then the image is upsample and the next level optical flow will be estimated again using iterative Lucas-Kanade algorithm which is used as a correction factor to the previous calculated optical flow. It is proven in [6], using this algorithm, we can achieve a robust estimation of the optical flow without trading off the accuracy by increasing the window size. This simple extension to the iterative LucasKanade algorithm will be used in the proposed algorithm to calculate the optical flow of the eye blinking motion.

only technique and step 5 to step 6 can be seen as optical flow only technique.

Figure 2: Coarse-to-fine optical flow estimation The pseudocode is presented here for reference. Compute Lucas-Kanade at coarse level For each level i Take flow ui-1, v i-1 from level i-1 Bilinear interpolate it to create ui*,vi* matrices of twice for level i Multiply ui*,vi* by 2 Compute ft from a block displaced by ui*(x,y), vi*(x,y) Apply iterative Lucas-Kanade to get ui’(x,y),vi’(x,y) Add corrections ui’ vi’: ui=ui*+ui’, vi=vi*+vi’ end

4

For frame differencing only technique, it is determined experimentally that the optimal ratio of the width and height of the blob is set between 1.5 and 2. For optical flow only technique, the window size is determined to be 11 x 11 pixels to achieve a reasonable result. For the proposed algorithm, the width and height ratio has been relaxed to within 0.9 and 2.5. This is due to the fact that the main features for classification are the direction and the magnitude of the optical flow as opposed to the frame differencing technique which uses the ratio of the blob as the main feature for classification. Also, the window size used in the proposed algorithm has been experimentally determined to be 5 x 5 pixels to achieve optimal performance in the experiment using 3 level of pyramid.

resolution

Eye Localization

The algorithm used to localize eyes in a grayscale image is the following. 1. Obtain locations of possible motion using frame differencing. 2. Suitably threshold the motion regions and obtain blobs using morphological operations and connected components. 3. Remove unsuitable blobs i.e. blobs which are either too big or too small or have incorrect width to height ratios to be considered as eyes. 4. If no possible candidate blobs are found, repeat 1 to 3 on subsequent frames until at least 2 suitable blobs are found, and mark their positions. 5. Compute optical flow field in the blob regions. 6. Ascertain dominant direction of motion in the blobs. If the dominant motion is downwards in a pair of blobs, their positions are noted. If the motion is not downwards then steps 1 to 4 are repeated. 7. The bounding boxes of the blobs where blink is deemed to have occurred are taken as eye locations. The above algorithm will be compared with two other different approaches namely the frame differencing technique and the optical flow only technique using Iterative Lucas-Kanade Algorithm. Notice that step 1 to step 4 of the above algorithm can be seen as frame differencing

For the optical flow only algorithm and the proposed algorithm, as indicated in the algorithm above, the classification problem consists of finding dominate flows in the motion area. If the direction of the optical flow is downward in the motion area, it is classified as possible eye candidates. Since the eye blinking motion is generally very fast, the magnitude of the optical flow can be used as a detection threshold to differentiate between slow vertical movements (i.e. head movements) and involuntary eye blinking movements. This detection threshold is needed to be adjusted properly to improve the accuracy of the detection with a minimum false alarm. For the optical flow only approach, the detection threshold is set to 5. The small threshold value is used because the inconsistency of the optical flow of the simple iterative Lucas-Kanade method. On the other hand, the proposed algorithm uses a detection threshold of 8 for best result.

5 Experimental Results and Discussion The level of performance of the algorithms is evaluated using 5 test sequences of images that are collected independently from the training sequence on the test day. The illumination and the relative position from the subject to the camera are expected to be different. The first sequence consist 20 involuntary eye blinking motions only which will be used to calculate the detection rate of the various algorithms (figure 3). The 2nd, 3rd and 4th sequences consist of nose, mouth and head motions respectively. These sequences will be used to calculate the false alarm rate of the algorithms. The last test sequence consists of a mixed of motions; it is used to evaluate the Optical flow only approach using Iterative Lucas-Kanade

Algorithm and the proposed algorithm by varying the detection threshold. The ROC curve will be used to illustrate their characteristics. Using those sequences, the weaknesses and the strengths of different algorithms can be revealed. A detected eye is considered as valid if the eye window contained the full eye. The detection rate is the ratio between the number of successful detections and the number of eye blinking motion in the test sequence. The false alarm rate is the ratio between the number of false positive detection and the number of non eye blinking motion.

a.

b.

a. 1st Sequence The result of the 1st sequence is summarized in table 1. Frame differencing technique works very well (see figure 4) when only eye blinking motion is in the field of view of the camera. It works 95% all the time if the ratio threshold is set properly. Different individual might require a different range of ratio threshold which can be very cumbersome.

c. Figure 4: a) Frame Differencing, b) Connected Component, c) Successful Detection

Using iterative Lucas-Kanade optical flow, the direction of the flow is used to detect eye blinking motion. The magnitude of the flow is used as a detection threshold. When the downward direction with magnitude higher than the threshold value, it is classified as eye blinking motion. As illustrated in figure 5, the optical flow using the iterative Lucas-Kanade algorithm can’t reliably reflect the motion of the eye. The largest magnitude of the flow is pointing to the right instead of to the downward; it fails in areas of large motion. The detection threshold is set to be 5 to get the optimal result in the 1st sequence and it achieves 74%. Using the proposed algorithm, an accuracy of 91% can be achieved. As illustrated in figure 6, the optical flow generated using multi-resolution Lucas-Kanade algorithm on the same pair of frames as figure 3 captures the direction of the eye blinking motion much better than the iterative Lucas-Kanade approach. It couldn’t achieve 95% of accuracy as the frame differencing technique because motion blurred affect the accuracy of the optical flow in some of the frame pairs.

Figure 5: Lucas-Kanade without Pyramid Optical Flow

Figure 6: Multi-Resolution Lucas-Kanade Optical Flow b a Figure 3: a) Eye Before Blinking and b) Eye During Blinking

Table 1: Detection Rate of Various Algorithms Algorithm Total # DetecDetection Rate tion (%) Frame Differenc- 100 95 95 ing LK Optical Flow 100 74 74 Proposed 100 91 91 a.

b.

b. 2nd, 3rd and 4th Sequences The 2nd, 3rd and 4th sequences collect statistics of the false alarm rate of each algorithm. The results are summarized in table 2. As noticed, the frame differencing technique is very sensitive to any motion in the scene. In order to decrease its false alarm rate, geometric constraints must be accurately enforced in the algorithm such as the distance between the eyes, the horizontal alignment of the eyes and the ratio between the eye’s width and height. It is expected that this information must be tuned for each individual to achieve optimal performance which can be very cumbersome and impractical for clinical use. Especially, when the distance between the camera and the subject might vary every time, this can be very tedious process. As illustrated in figure 7, the blobs created using frame differencing technique will classify the corners of the mouth as two eye candidates when the mouth corners are moving upward (smiling expression). On the other hand, the Lucas-Kanade Optical Flow only technique and the proposed algorithm have a much lower false alarm rate. It proves that the direction and the magnitude of the optical flow is a very good feature to use to discriminate the eye blinking motion with other motions. This is due to the fact that under normal situation, the eye blinking motion is much faster than other motions happened in the scene and therefore by setting the detection threshold high enough the false alarm rate falls rapidly. The optical flow of one corner of the mouth in figure 7 is illustrated in figure 8. Notice that the magnitude of the flow vectors are much smaller compared to eye blinking motion in figure 5. The proposed algorithm works better than the iterative Lucas-Kanade only approach because the optical flow of the eye blinking motion can be accurately measured and therefore a higher threshold value can be used to discriminate from other motions. For iterative Lucas-Kanade, it is observed that the false alarms occur more frequently when head moves up and down rapidly. However, they both are very good to handle horizontal motion as illustrated in figure 10 and 11. Again, it is observed that the multi-resolution Lucas-Kanade algorithm produces a very dense optical flow for the true horizontal motion of the head (figure 11).

c. d. Figure 7: a) Before Smiling b) During Smiling c) Frame Differencing of a and b d) connected component of c

Figure 8: Mouth Corner Motion using Iterative LucasKanade

Figure 9: Mouth Corner Motion using Multi-Resolution Iterative Lucas-Kanade

the proposed algorithm initially uses frame differencing to filter out noneye motion regions. Since this depends on the ratio of the blob initially set in the algorithm, therefore the proposed algorithm might also miss the possible eye candidates. It is noticed that motion blur does affect the accuracy of the optical flow and therefore even the eye is detected, the optical flow of the eye blinking motion does not always pointing downward even multi-resolution iterative Lucas-Kanade algorithm is used. That have been said, it is much less severe than the iterative LucasKanade algorithm.

Figure 11: Horizontal Motions using Multi-Resolution Iterative Lucas-Kanade Table 2: False Alarm Rate of Various Algorithm Algorithm Total # False False Alarm Rate Alarm (%) Frame Differencing Optical Flow Proposed

126

59

47

126 126

26 21

21 17

c. 5th sequence The 5th sequence is used to observe the behavior of the optical flow only technique and proposed technique by varying the detection threshold value. It consists of a mixed of 50 eye blinking motions and 50 other motions performed naturally in front of the camera. Figure 12 shows the detection rates versus detection threshold. Notice that both methods can’t achieve 100% of the detection rate even the threshold is set to zero due to various reasons. For optical flow only technique, the optical flow of the eye blinking motion does not always pointing downward as noted in figure 5. Therefore, the detection rate can never achieve 100%. On the other hand,

Figure 14 illustrates the ROC curves of both algorithms. Each point on the curve corresponds to a given threshold applied to the speed of the downward motion in order to classify a motion area into eye blinking motion or noneye blinking motion. For both algorithms, high detection and false alarm rate are obtained for small threshold values and false alarm rate goes down to zero for high threshold values as in figure 14. However, as the threshold value increases (toward the left), the detection rate decreases slowly while the false alarm rate decreases more quickly as the threshold value increases. It is observed that the ROC curve of the proposed algorithm is a left-shift version of the optical flow only technique which is an indication that the proposed algorithm is more robust than the optical flow only technique. Detection Rate vs. Detection Threshold 100 Detection Rate (%)

Figure 10: Horizontal Motions using Iterative LucasKanade

Figure 13 shows the false alarm rate vs. detection threshold. Using Lucas-Kanade only technique, when the detection threshold is set to zero, the direction of the flow becomes the main feature for the classification problem. Therefore, all motions that have dominant optical flow pointing downward are considered eye blinking motions. Therefore, it has higher false alarm rate. On the other hand, the proposed algorithm uses frame differencing and the direction of the flow as features for classification when detection threshold is set to zero. Therefore, it has less false alarm rate than the optical flow only technique. In fact, the gap between the curves in figure 12 and 13 indicates the net gain of using frame differencing and the multi-resolution iterative Lucas-Kanade algorithm.

80 Optical Flow Only

60

Proposed

40 20 0 0

5

10

15

Detection Threshold (pixel)

Figure 12: Detection Rate vs. Detection Threshold

20

6

Conclusion

False Alarm Rate (%)

False Alarm Rate vs. Detection Threshold 60 50 40 Optical Flow Only

30

Proposed

20 10 0 0

5

10

15

20

Detection Threshold (pixel)

Figure 13: False Alarm Rate vs. Detection Threshold

100 80 60

Optical Flow Only Proposed

40 20 0 0

10

20

30

40

The frame differencing technique takes much less computational time compared to the other two approaches, however it has the highest false alarm rate among them and it requires to adjust the algorithm parameters (blob widthheight ratio, distance between the eye blobs, etc) for each individual to achieve optimal result which is unrealistic. Iterative Lucas-Kanade Optical flow approach has fewer false alarms than the frame differencing technique when the detection threshold is set sensibly but its detection rate is too low for practical use.

Detection Rate vs. False Alarm Rate

Detection Rate (%)

From the experimental result, it is concluded that the proposed algorithm is more robust and accurate then the frame differencing technique and the optical flow technique.

50

60

False Alarm Rate (%)

Figure 14: ROC curve. Each point on a curve correspond to a given detection threshold value

d. Discussion From the experimental results, it is noticed that the current implementation of the proposed algorithm suffers from the relatively high false alarm rate largely due to motion blur. Motion blur affect the accuracy of the optical flow because the brightness constancy assumption for optical flow does not hold anymore. To fix this problem in the future, we can try to set a faster shutter speed for the camera. However, a faster shutter speed will reduce the amount of light going into the CMOS sensor and therefore, we need to open up the aperture of the camera to compensate for this effect. Also, the frame differencing technique used in the current implementation is not perfect; it has a false negative rate of about 5%. To improve this, we can take into account of the face geometric constraint such as the distance between the eyes with respect to the eye blob ratio.

The proposed algorithm achieves a detection rate of 91% with much fewer false alarm than the two other approaches. However, a high cost has to be paid for its extra accuracy and robustness; the multi-resolution LucasKanade algorithm requires approximately 3 times of the computation time of an iterative Lucas-Kanade algorithm despite the fact that frame differencing technique has been used to reduce the search space. We can however further reduce the computational time by reducing the size of the blob window or returns immediately when several consecutive downward flow vectors has a magnitude higher than the threshold value. However, the performance of the accuracy of the detection might be reduced due to the noise. Nevertheless, both optical flow approaches have problem when motion blur occurs between frames. In this paper, we focus on the eye detection. However, to localize eyes in a video, we should make use of the temporal information of sequence. Therefore, after having initialized the eye windows using the eye detection algorithm, an eye tracking algorithm can be used to track the eye windows for the subsequent images. It is reported in [7], [8] and [9] that KLT tracker performs very well; It can be used to track feature points in the eye windows. The eye detection algorithm will only be initiated when the eye tracker lose track of the eye windows. Using this combined approach, the robustness of the eye localization algorithm should be improved.

References [1] K.Yano, K.lshihara, M. Makikawa, H. Kusuoka. “Detection of eye blinking from video camera with dynamic

[2]

[3] [4]

[5]

[6]

[7]

[8] [9]

ROI fixation”, IEEE Trans. SMC, Vol 6, pp 335-339, 1999 Tracy Westeyn, Peter Pesti, Kwang-Hyun Park & Thad Starner. Biometric Identification using Song-Based Blink Patterns. unpublished, 2005. Website: http://wwwstatic.cc.gatech.edu/~pesti/pubs/biometric_ID_HCII05.pd f http://www.math.tau.ac.il/~arielt/normalization.html Kawato, S., and Tetsutani, N. 2002. Detection and tracking of eyes for gaze-camera control. In Proeedings of VI2002, 348–353. K. Grauman, M. Betke, J. Lombardi, J. Gips, and G. Bradski. Communication via eye blinks and eyebrow raises: Videobased human-computer interfaces. Universal Access in the Information Society, 2(4):359–373, November 2003. J.Y. Bouguet, .Pyramidal Implementation of the Lucas Kanade Feature Tracker., OpenCV Documentation, Microprocessor Research Labs, Intel Corp., 2000. Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In I jvai81 (p.674-679). S. Birchfield, “Derivation of Kanade-Lucas-Tomasi Tracking Equation”, unpublished, May 1996. C. Tomasi and T. Kanade, “Detection and Tracking of Point Features”, Carnegie Mellon University Technicul Report CMU-CS-91-132, April 1991.