Modelling User Search Behaviour Based on Process

Typically, interactive information retrieval (IIR) system evaluations assess search processes and outcomes using a combination of two types of measures: 1. user perception (e.g. users? attitudes of the search experience and outcome); 2. user behaviour (e.g. time and counts of various actions including mouse and keyboard clicks). In general, we assume that they are indicative of the search outcomes (e.g. performance, opinion). However, search is a dynamic process with changing outcomes. Therefore, neither measure solely provides a holistic way of evaluating search. On one hand, user behaviour measures are only descriptive of the outcome, and are not interpretive of the process. That is to say, they lack the rationale behind why those behaviours occurred. Another problem is that some mental activities may not reflect on user behaviour [1]. The challenge with logfiles, which contain behaviour data, is the voluminous number of data points and the need to find a reliable approach to define groups or sets based on behavioural patterns. Not all users are alike and nor do they all take the same approach to search for the same things, as evidenced by the TREC, INEX and CLEF interactive tracks. On the other hand, user perception measures are acquired in such small samples that do not scale to large participant populations, and are rarely measured constantly due to the laborious and time consuming data collection methods (e.g. questionnaire, interview). Moreover, not enough emphasis is put on assessing the reliability of individual perception measures, and the wide usage of likert-type scale limits the interpretation of answers. For a holistic understanding of the search process, we need both perception and behaviour measures. I speculate that user behaviour may predict user perception, and thus we should be able to analyse large-scale files for a greater understanding of the likely human responses.