Share your own Notes and Web Clippings with Memonic
Start now
Search results are often listed in an order specified by a relevance metric. Alternatively, results are ordered according to a metadata attribute, such as reverse chronological order for news search and email search, or number of citing papers for journal article search. It is also common to group results by well-defined metadata fields, such as grouping email by sender name or journal article by author name.
Studies and query logs show that searchers rarely look beyond the first page of search results. If the searcher does not find what they want in the first page, they usually either give up or reformulate their query (Chapter 6 discusses query reformulation in detail). Furthermore, several studies suggest that web searchers expect the best answer to be among the top one or two hits in the results listing, and this expectation influences whether or not they will click on a result. Several of these studies are discussed below.
An eye-tracking study by Granka et al., 2004 on 26 participants and 397 queries found that on average, participants took 7.78 seconds to select a document, but the time varied significantly among the 10 pre-defined search tasks, from 5-6 seconds up to 11 seconds for the most difficult questions. The first result was selected approximately 85% of the time, and the second link about 10% of the time. Furthermore, the first and second results were by far the most viewed, with a sharp dropoff starting with the third result. A followup eye-tracking study by Joachims et al., 2005 with 29 participants showed that the percentage of times a search result in the top ten listings was looked at dropped off sharply from first search result to sixth, and then flattened at around 5% of the time for results 7--10. Likelihood of clicking on the result, however, dropped much more dramatically, falling from 43% of the time for the first hit to 15% of the time for the second hit, 10% of the time for the third hit, and 5% or less for the rest. This result held despite the fact that 5 of the 22 participants were shown the results in reverse order of their original ranking, and another 5 of the participants were shown the top two hits in swapped position.
Joachims et al., 2005 also found that participants tended to view the first and second-ranked results right away, with a large gap before viewing the third-ranked abstract. They also found that while participants did not necessarily view all abstracts above a click, they view substantially more abstracts above than below the click. More surprisingly, they also found that the abstract right below a click is viewed roughly 50% of the time.
Joachims et al., 2005 also found bias in relevance judgements based on placement location. They did a followup experiment focusing on the top two results, since these are scanned equally frequently. They compared how often a participant clicked on either result 1 or result 2 depending on the manually judged relevance of the abstract. They found that participants were influenced in their relevance assessment by the order of presentation, since the number of clicks on link 1 was significantly higher than its relevance would merit.
Despite these results, Joachims et al., 2005 found that participants were not blindly following link order. In a condition in which they complete reversed the rank order of the top 10 results, they found that in the reversed condition, participants viewed lower ranked links more frequently, scanning significantly more results than in the normal condition. Those who saw the reversed condition were also much less likely to click on the first link and were more likely to click on a lower-ranked link. The average rank of a click in the normal condition was 2.66 and 4.03 in the reversed condition. However, the average relevance of the selected documents in the reversed condition was lower than in the normal condition.
Guan and Cutrell, 2007 performed an eye-tracking study in which they pre-determined the queries and results, and controlled which of the top 10 positions the one relevant result appeared in. They also contrasted navigational and informational query types. In a study with 18 participants, they found a significant main effect of target position on total task time and on query type. Participants spent more time when the target was farther down the result list, but this extra time did not result in more success at making the correct choice. The click accuracy rates dropped from about 84% when the target result was in position 1 or 2, to about 11% when the correct response was in position 8. For navigational queries, when participants did not find the result below position 3, they either selected the first hit (40% of the time) or reformulated their query. For informational search, participants rarely reformulated the query without first trying to click on the first hit (about 50% of the time) or clicking on the other links at random. One might infer from this that participants are more confident in the search engine ranking for the relatively easier navigational queries than for the more general informational ones. Guan and Cutrell, 2007 examined the results of the eye-tracking data and found that participants did look at the lower ranked results. They concluded that the participants' behavior was caused by their expectation that the relevant results would be at or near the top.
A somewhat different kind of behavior was seen in an eye-tracking study by Aula et al., 2005b. They analyzed the eye movements of 42 students on 10 pre-defined queries, and found two distinct styles of scanning results list. 46% of the participants were “economic,” scanning at most half of the 6-7 visible search results in 50% of the tasks. The remaining 54% were “exhaustive” evaluators, who for most queries viewed more than half of the visible results, and in some cases scrolled down to see the full list of 10 hits. The difference in time before first action was significantly shorter for the economic searchers, especially when good results were available. The authors find a marginal difference between the evaluation style and computer experience, with more experienced searchers tending to use the economic style, and speculate that the Granka et al., 2004 study may have employed only expert searchers, thus explaining the different results.
(a)
(b)