User Domain Knowledge and Eye Movement Patterns During Search Michael J. Cole, Jacek Gwizdka, Nicholas J. Belkin, Chang Liu Rutgers, The State University of New Jersey School of Communication and Information New Brunswick, New Jersey 08901

belkin,[email protected], [email protected], [email protected] Cognitive effort measures inferred from eye movement patterns during textural information search have been correlated with subjective task difficulty and objective measures of task effort. Analysis of the same cognitive effort measurements in an independent user study (n=40) of recall-oriented search in the genomics domain reveals strong correlations with self-assessed domain knowledge. A simple regression model based on these measures was successful in predicting participant domain knowledge. These measurements of cognitive effort during search can be calculated on-the-fly and require only recent eye fixation location and duration. We discuss the potential for real time detection of domain knowledge during information search using this eye movement analysis technique.

Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval—relevance feedback, search process

General Terms Performance, Design, Human Factors.

Keywords knowledge detection, cognitive effort, user study, personalization, information search behavior

1.

INTRODUCTION AND RELATED WORK

Domain knowledge or expertise has been shown to affect search behaviors [23, 25]. If one could infer the user’s knowledge of their task there could be significant opportunity to improve search system performance [21, 7]. This work reports on a relationship between eye movement patterns and user domain knowledge during (textual) information search based on eye tracking logs from a user study. Measures derived from eye movements are well-suited to represent information search processes and can provide a basis for inference of important user mental states. In (textual) interactive information retrieval (IIR), information acquisition is mediated by eye movement patterns in service of the reading process. Eye movements are known to be cognitively-controlled [8] and research into the reading process has identified several observable indicators of cognitive Copyright is held by the author/owner(s). HCIR’11, October 20, 2011, Mountain View, CA. ACM X-XXXXX-XX-X/XX/XX.

effort associated with reading eye movements. In previous work we have demonstrated relationships between eye movement patterns and task and page types [6, 4]. We have developed several measures of cognitive effort due to information acquisition based on units of reading, i.e. sequences of eye fixations, which are identified using eye tracking logs of user studies. These measures include the length of the reading unit, the speed of reading, the duration of fixations, the spacing of fixations, and fixation location patterns within the reading unit. These measures have been shown to correlate well with user assessment of task difficulty and with objective task effort measured in user behaviors, such as number of documents examined, and document use [5]. Our cognitive effort measures are closely associated with semantic processing, such as acquisition of word meaning, which in turn is correlated with a user’s level of knowledge. It has long been known that familiarity and conceptual complexity of the text processed is positively correlated with the fixation duration (e.g.[17]). Vocabulary knowledge can be directly related to knowledge of concepts through an internal representation as concept features or as the mechanism for accessing these concepts. This recognition of the essential link via meaningfulness of words and concept formation and use is a core aspect of research into the nature of concepts (cf [11, 9, 1, 13, 12]). Despite the difficulty of fixing the precise relationship between psycholinguistics and concept access and use, knowledge of vocabulary is well-accepted as an indicator of concept knowledge. In this paper we present evidence for correlations between user levels of domain knowledge and objective measurements of eye movement patterns. We also build a simple regression model to infer user domain knowledge from eye tracking logs. The methodology to acquire and calculate the cognitive measures has very low computational demands and calculations of cognitive effort due to reading can be made while the person is engaged in their task. If effective predictive models based on the eye movement measurements can be identified, our methodology could enable real-time detection of the user’s level of domain knowledge and allow for dynamic personalization of an information system to improve its effectiveness.

2.

METHODOLOGY

Undergraduate and graduate students (n=40) in biologyrelated programs rated their knowledge of 409 genomicsrelated MeSH terms (1–’No knowledge’, 5–’Can explain to others’). They then performed four recall-oriented search tasks from the 2004 TREC Genomics Track using the In-

dri search system with Medline abstracts (n=1.85 million). Search interactions were recorded using a multi-source logging system [2]. Technical reasons prevented analysis of two participants. The participant domain knowledge (PDK) was calculated as:

P DK =

Pm

i=1 (ki ∗ti )

5∗m

where ki is the term knowledge rating and i ranges over the terms. m is the total number of terms rated (409) and ti is 1 if rated or 0 if not. The sum is normalized by a hypothetical expert who rated all terms as ’can explain to others’. Hierarchical clustering identified high domain knowledge (n=6), intermediate domain knowledge (n=24), and low domain knowledge (n=8) groups.

2.1

Cognitive effort measures

Several cognitive effort measurements were calculated based on the fixation properties and the properties of reading sequences. One is the reading speed, i.e. the ratio of the amount of text processed (the reading length) to the processing time. Reading speed is greater for easy text [18] and is affected by word familiarity [24], experience with word senses [20], and when additional reflection is required to comprehend the concepts involved [14]. Sentence parsing can also impact reading speed. Fixation duration is an indicator of the cognitive processing required to establish the meaning of the word, and the meaning of the word in context. [16] show that text passages of greater conceptual difficulty resulted in more fixations and slightly longer mean fixation duration. The amount of cognitive processing to acquire the meaning of a word or phrase is related to the duration of an eye fixation beyond that required for lexical access (˜113 ms). We call this the lexical fixation duration excess (LFDE). Fixation spacing is associated with cognitive processing constraints. Perceptual span is the amount of text one takes in at a time. Studies of reading in different orthographic systems provide evidence that perceptual span reflects a concept throughput bottleneck [10, 22, 15]. Perceptual span is operationalized as the average separation of fixations in display pixels. A regression fixation is a fixation that returns to a portion of the text already processed. The number of regressions in a reading sequence, and the fixation durations of the regression fixation have been associated with the difficulty of reading passages, resolution of ambiguous (sense) words, conceptual complexity of text, parsing difficulties and the reading goal [18, 16]. It is a common feature of reading eye movement sequences and can have an incidence of 1015% of the total fixations in the reading sequence [3]. We operationalize a regression measurement as a count of the 1

3.

RESULTS

Eye movement analysis

Models of the reading process have been developed that explain observed fixation duration and word skipping behaviors. We implemented the E-Z Reader model1 [19]. It was used to process the user study eye tracking logs to classify which fixations were lexical, i.e. which fixations resulted in acquisition of the meaning of words, and then to group fixations into sequences of reading.

2.2

regression fixations in a single reading sequence consisting of at least four fixations. The details of our operationalization of these measures are given in [5]. For the present work we considered each task session as a contiguous collection of reading sequences. This represents the user’s experience of information acquisition due to reading during the search. We calculated these cognitive effort measures for two levels of reading sequences during the session. Effort measures were calculated for all reading sequences and for long reading sequences (number of fixations > 3). It seems plausible the longest reading sequences may be particularly interesting for exploration of domain knowledge effects because of the extended attention to that text by the user.

[5] provides implementation details.

3.1

Domain knowledge and cognitive effort

We first looked at correlations between the cognitive effort measures and domain knowledge when all reading sequences were considered. Perceptual span was not normally distributed, but was correlated with domain knowledge (Kruskal-Wallis χ2 = 4734.254, p-value < 2.2e-16), likewise for the median LFDE (Kruskal-Wallis χ2 = 5570.103, p-value < 2.2e-16). Reading speed was also not normal but correlated (Kruskal-Wallis χ2 = 105.094, p-value < 6.3e-09). Long reading sequences may better reflect concept use by participants during information acquisition. The number of regressions is likely meaningful only in reading sequences of four or more fixations. Applying this somewhat arbitrary threshold we selected the longer sequences which were 7.5% (19477/258586) of the total number of reading sequences in the study. The mean number of regressions was nearly normal, but not correlated significantly with domain knowledge. Using the long reading sequences, we found perceptual span was normally distributed and correlated with domain knowledge (ANOVA F-value 29.144 p-value < 6.8e-08). The same was true of reading speed (ANOVA F-value=5.2342, p-value=0.024). The median LFDE was not normally distributed but was correlated with domain knowledge (KruskalWallis χ2 = 4724.891, p-value < 2.2e-16).

3.2

Modeling domain knowledge with cognitive effort measurements

As an initial modeling effort, we treated each reading sequence in the study as an observation of the respective participant’s domain knowledge. Using the long reading sequences, a Gaussian family general linear model (glm) was constructed using all of the cognitive effort measures (Table 1). The reading sequences for each task session were used as the model input to predict the participant’s domain knowledge. We then calculated the mean domain knowledge prediction over the four task sessions for each participant. The glm predictions and the participants’ PDK were correlated (ANOVA F-value=4.78, p-value=0.035). The glm prediction standard deviation was an order of magnitude lower as compared to the PDK value (0.011 vs. 0.132). To examine the model performance, we grouped the participants by predicted domain knowledge using hierarchical clustering to compare them with the MeSH-term rating groups (Table 2). The glm predictions were reasonable in discriminating

the higher and low domain knowledge groups. After removing non-native English speakers, the model performance improved for the high knowledge group and deteriorated somewhat for the low knowledge group (Table 3). (Intercept) numRegressions perceptualSpan readingLength readingSpeed maxDur medianDur totalDur

Estimate 0.3859 0.0050 -0.0001 0.0001 0.0877 < 0.0001 -0.0002 < -0.0001

Std. Error 0.0082 0.0012 < 0.0001 < 0.0000 0.0156 < 0.0001 < 0.0001 < 0.0001

t value 47.08 4.26 -3.14 8.29 5.62 4.54 -10.06 -0.19

Pr(>|t|) < 2e-16 2.1e-05 0.0017 < 2e-16 < 2e-08 < 6e-06 < 2e-16 0.8517

Table 1: Cognitive effort domain knowledge model

PDK groups low intermediate high

Predicted low inter high 2 4 2 5 12 7 0 2 4

Table 2: Classification errors: All participants

PDK groups low intermediate high

Predicted low inter high 0 2 1 3 9 5 0 0 4

Table 3: Classification errors: Native English only

4.

DISCUSSION

The results show correlations between user domain knowledge and measurements of eye movement patterns that reflect cognitive effort in the reading process. Reading speed, the fixation duration in excess of that minimally needed to acquire word meaning (LFDE), and fixation spacing (perceptual span) were all significantly correlated with the participant’s genomics domain knowledge. The glm results are gratifying, given the naive construction of the model and use of reading sequences over entire task sessions without regard to their position in the task session or sequence. There are obvious shortcomings as well. For example, the range of predicted domain knowledge was much smaller that the actual range. In domains where knowledge differences are not so clearly distinguished, the discrimination of the model may be so minimal as to be useless. Practical application of the model may also be difficult because of individual differences in cognitive effort measures. In our user study, we found wide variations in mean LFDEs within a domain knowledge group that can be explained to a significant degree by the native language of the participant. The improvement in model domain knowledge predictions when only native English speaking participants were considered underlines this modeling challenge. More generally, individual differences in cognitive effort measures have some basic ambiguity because a user’s level

of knowledge is an individual difference that is manifested broadly in observable behaviors. For example, one cause of fixation duration is word familiarity. Presumably, words indicative of domain knowledge will be more familiar to those with high domain knowledge and will require lower fixation durations as compared to those with low domain knowledge. Yet, insofar as these words are relevant/useful to the task at hand, such words are more likely to be selected and processed by high domain knowledge users. It is plausible users will tend to fixate on significant words with which they have some level of familiarity because these passages are more accessible conceptually for the task at hand. This word selection issue is a significant complication for inferring level of domain knowledge from cognitive effort measures. The foregoing shows how the observed eye pattern-based cognitive effort measures reflect at least two general aspects of domain knowledge use: first in content selection and then in the cognitive processing of the content. Our cognitive effort measurement methodology addresses only the cognitive processing dimension of the problem. The small standard deviation observed in the model’s domain knowledge predictions might be partially explained by this dependency of an individual’s process of acquiring information on their domain knowledge. In the face of this basic challenge, the success of a naive glm in predicting the level of domain knowledge is unexpected and it suggests that domain knowledge inference from cognitive effort measures may be robust. To address the selection problem, one could also capture and analyze semantic properties of the selected words. Comparing the probability of word selection in a document with its frequency in a domain language model is just one way to extend this work to better detect domain knowledge. Our technique requires only input of the location and duration of recent eye fixations and simple processing. These measures of the user’s current processing of text meaning can be generated in near real-time and can be available for every interaction segment in the task session. Personalization of search interaction would benefit most from early prediction of the user’s domain knowledge, so one direction for further analysis is to identify when a domain knowledge prediction can be made with reasonable confidence in a task session. A number of limitations exist, some of which can be addressed in future research. The analysis concerns a sequence of reading interactions. It does not take account of page boundaries and it is plausible domain knowledge effects on search results pages may be different than on the link (content) pages. Such page level analysis is an obvious next step since it correlates with the page-oriented observational units that are typical for interaction analysis. Page-oriented measures are also easier to coordinate with analysis of higherlevel behaviors that may be partially reflected in search logs, such as document use and dwell time. Domain knowledge representation is difficult and there are a number of shortcomings in the MeSH-based representation we used. The coverage of rated MeSH terms (n=409) in the entire MeSH space (n=25,186) is an important limitation.

5.

CONCLUSIONS

Eye movement patterns are useful for detection of user domain knowledge because they have a direct relationship with cognitive processing that connects document content with the user’s knowledge of the meaning of the text. Patterns of spatial-temporal processing of regions of pages, or

transitions from processing information objects to system interactions are also promising features for modeling the user’s cognitive engagement in the search process. There is a wealth of research in cognitive psychology and related fields that provide a solid empirical foundation for analysis of eye movement patterns in information search interactions. One contribution of this paper is to show that cognitive effort measures derived from eye movement patterns are associated with the level of self-assessed domain knowledge. Another contribution is a simple regression model that suggests prediction of user domain knowledge based on these measures is possible. A person’s use of knowledge during search drives the interaction process and the acquisition of information by reading. These results connecting eye movement patterns and domain knowledge raise the possibility of robust and direct detection of a user’s knowledge level during information search.

6.

ACKNOWLEDGMENTS

[11] [12]

[13]

[14]

[15]

IMLS grant LG-06-07-0105-07 supported this work.

7.

REFERENCES

[1] Armstrong, S., Gleitman, L., and Gleitman, H. What some concepts might not be. Cognition 13, 3 (1983), 263–308. [2] Bierig, R., Cole, M. J., Gwizdka, J., and Belkin, N. J. A data analysis and modelling framework for the evaluation of interactive information retrieval. In Proceedings of ECIR 2010, (Milton Keynes, UK 2010) (2010), C. Gurrin et al., Eds., vol. 5993 of Lecture Notes in Computer Science, Springer, pp. 673–674. [3] Boland, J. E. Linking eye movements to sentence comprehension in reading and listening. Taylor & Francis, 2004, pp. 51–76. [4] Cole, M. J., Gwizdka, J., Bierig, R., Belkin, N. J., Liu, J., Liu, C., Zhang, J., and Zhang, X. Linking search tasks with low-level eye movement patterns. In Proceedings of the 17th European Conference on Cognitive Ergonomics (June 2010). [5] Cole, M. J., Gwizdka, J., Liu, C., and Belkin, N. J. Dynamic assessment of information acquisition effort during interactive search. In Proceedings of the American Society for Information Science and Technology Conference (2011) (New York, October 2011), ACM. [6] Cole, M. J., Gwizdka, J., Liu, C., Belkin, N. J., Bierig, R., and Zhang, X. Task and user effects on reading patterns in information search. Interacting with Computers 23, 4 (July 2011), 346–362. [7] Cole, M. J., Zhang, X., Belkin, N. J., Liu, C., and Gwizdka, J. Knowledge effects on document selection in search results pages. In Proceedings of SIGIR 2011 (July 2011), ACM. [8] Findlay, J., and Gilchrist, I. Active vision: The psychology of looking and seeing. Oxford University Press New York, 2003. [9] Fodor, J. A. The language of thought. Harvard Univ Pr, 1975. [10] Inhoff, A. W., and Liu, W. The perceptual span and oculomotor activity during the reading of Chinese sentences. Journal of Experimental Psychology.

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

Human Perception and Performance 24, 1 (1998), 20–34. Katz, J. J. Semantic theory. Harper & Row, New York, 1972. Landauer, T., Laham, D., and Foltz, P. Automated scoring and annotation of essays with the intelligent essay assessor. In Automated essay scoring: A cross-disciplinary perspective, M. Shermis and J. Burstein, Eds. Lawrence Erlbaum Associates, Mawah, NJ, 2003, pp. 87–112. Landauer, T. K. On the computational basis of learning and cognition: Arguments from LSA. In The Psychology of Learning and Motivation, N. Ross, Ed., vol. 41. ?, 2002, pp. 43–84. Morris, R. Lexical and message-level sentence context effects on fixation times in reading. Journal of Experimental Psychology: Learning, Memory, and Cognition 20, 1 (1994), 92. Pollatsek, A., Rayner, K., and Balota, D. A. Inferences about eye movement control from the perceptual span in reading. Perception & Psychophysics 40, 2 (1986), 123–130. Rayner, K., Chace, K. H., Slattery, T. J., and Ashby, J. Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading 10, 3 (2006), 241–255. Rayner, K., and Duffy, S. Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition 14, 3 (1986), 191–201. Rayner, K., and Pollatsek, A. The psychology of reading. Lawrence Erlbaum Associates, Mahwah, New Jersey, 1989. Reichle, E. D., Pollatsek, A., and Rayner, K. E–Z Reader: A cognitive-control, serial-attention model of eye-movement behavior during reading. Cognitive Systems Research 7, 1 (2006), 4–22. Sereno, S., O’Donnell, P., and Rayner, K. Eye movements and lexical ambiguity resolution: Investigating the subordinate-bias effect. Journal of Experimental Psychology: Human Perception and Performance 32, 2 (2006), 335. Teevan, J., Dumais, S. T., and Horvitz, E. Potential for personalization. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 1 (2010), 1–31. Tsai, C., and McConkie, G. The perceptual span in reading Chinese text: A moving window study. In Seventh International Conference on the Cognitive Processing of Chinese and Other Asian Languages, Hong Kong (1995). White, R. W., Dumais, S. T., and Teevan, J. Characterizing the influence of domain expertise on web search behavior. In Proceedings of WSDM (2009), ACM New York, NY, USA, pp. 132–141. Williams, R., and Morris, R. Eye movements, word familiarity, and vocabulary acquisition. Journal of Cognitive Psychology 16, 1 (2004), 312–339. Zhang, X., Anghelescu, H., and Yuan, X. Domain knowledge, search behavior, and search effectiveness of engineering and science students: An exploratory study. Information Research 10, 2 (2005), 217.