Analyzing User Knowledge Gain in Informational Search Sessions on the Web

1

Ujwal Gadiraju, Ran Yu,

Stefan Dietze and Peter Holtz

Serving Learning Needs

2

• Web search is frequently used to acquire new knowledge & satisfy learning-related objectives
• Little is known & understood about how the knowledge of a user evolves through the course of informational search sessions

Web Search Queries : Navigational, transactional or informational intents [Broder, 2002]

​

​

.. in informational web search sessions, the intent of a user is to acquire some information assumed to be present on one or more web pages ..

Serving Learning Needs - Prior Works

1. Eickhoff et al. investigated the correlation between several query and search mission-related metrics with learning progress [2014]
2. Wu et al. predicted the difficulty of search tasks from query and mission-related features [2012]
3. Collins-Thompson et al. investigated the aspects of search interaction which are effective for supporting superior learning outcomes [2016]

​

​

3

Main focus has been to improve the learning experience and efficiency during search sessions.

There is a lack of understanding on the impact of web search on a user’s knowledge state.

Research Questions

• How does a user’s knowledge evolve through the course of an informational search session on the web?
• How does the topic/ information need in a search session influence a user’s knowledge gain?
• What is the impact of information need on the search behavior of users in a search session?

4

Study Design

5

1

2

3

4

5

Topics & Information Needs

6

High Cronbach’s α indicating a reliable internal consistency of the knowledge tests

A Few More Details . . .

• RELIABILITY: Participation of only Level-3 CrowdFlower workers from primarily English-speaking countries
• 50 workers per topic, filtered out workers who entered no queries, workers who selected the same option ‘TRUE/FALSE’ for all items ⇒ 420 workers
• KNOWLEDGE TESTS: Using 100 workers per topic, and a larger pool of items (~30); filter items that were too easy (>80% of the workers got it right) or too hard/ambiguous (<20% got it right).

​

7

Measuring Knowledge Gain

8

Knowledge Gain

• 70% of users exhibited a KG

​

• Negative relationship between KG of users and topic popularity (R= -.87)

​

• Positive relationship between KG and fraction of IDK responses in calibration test (R=.72)

​

9

Main Findings (1/2)

• Users depicted a higher knowledge gain in sessions corresponding to topics that are generally less popular.
• The information need in a search session influences the number of queries entered, the number of pages consumed, and the number of different PLDs accessed by users.ain.

10

Main Findings (2/2)

• Last queries entered in search sessions are significantly longer than the first queries, with more unique terms in the last queries than in the first.
• Avg. complexity of queries entered is positively correlated to their knowledge gain. The amount of active time that users spent on web pages also correlated positively with their knowledge gain.

11

What’s in Store?

12

Collaborative Search

13

Searching in Pairs

14

Teammate’s Screen

Experimental Setup (1/2)

• Participants first complete the Big-Five personality traits test
• Pre-calibration test on a given topic, randomly assign participants into pairs
• Brief participants on the information need(s), final test, rewarding strategy
• Users in pair will be able to see each others’ screens once the main task begins.

15

16

Experimental Setup (2/2)

• Log all activity, communication
• Incentivize participants to “learn well AND collaborate” by using different payment strategies
• minACC(usr1, usr2)
• avgACC(usr1, usr2)

17