If You Build it, Will They Come?

Natural Experiments with Springshare's Proactive Chat Reference

Gabriel Gardner - CSU Long Beach

Joanna Kimmitt - CSU Dominguez Hills

Mike DeMars - CSU Fullerton

Sarah Baker - CSU Los Angeles

​

​

Background

​

• 23 campuses
• 480.000 students
• Largest 4 year university in the US
• 800,000 visitors per week
• Over 17 million Volumes

​

Background

2015

• CSU System Adopts Alma
• CSU Chooses Primo
• Go Live June 2017

​

Why Primo?

• Alma integration
• Primo Central Index
• Cost
• Customization

​

To Chat or not to Chat?

• Some incorporated chat
• Others stuck with older communication methods

​

LibChat Pop-Up Chat Demonstration

Our Campuses

CSU Long Beach campus profile

• Location: Long Beach, CA - Los Angeles County
• FTE: 37,000
• Migrated from: III Millennium
• Chat widget placements:
• At time of study: pop-up on Primo results pages, passive on some LibGuides, all LibAnswers FAQs, our LibCal hours page
• Now: only passive widgets on same platforms

CSU Fullerton campus profile

• Location: Fullerton, CA - Orange County
• FTE: 40,000
• Migrated from: III Millennium
• Chat widget placements:
• At time of study: None
• Now: Bottom floating

CSU Dominguez Hills campus profile

• Location: Carson, CA - Los Angeles County
• FTE: 11,325 (Spring 2018) [headcount: 14,635]
• Migrated from: Millennium
• Chat widget placements:
• At time of study: Bottom right, floating:
• Located in Primo, library website, LibGuides, most library databases
• Now: Same

​

​

From CSUDH Campus Profile: https://www.csudh.edu/ir/campusprofile/

CSU Los Angeles campus profile

• Location: Los Angeles, CA - Los Angeles County
• FTE: 28,000
• Migrated from: Millenium
• Chat widget placements:
• At time of study: None
• 24/7 chat, e-mail form on Ask Us page
• Now: None
• Report a Problem form in Primo
• 24/7 chat, e-mail form on Research Help page

​

Our Unwitting Natural Experiment With Pop-Up Chat

Digression on the Neyman–Rubin causal model

• Proposed by Jerzy Neyman in 1923, popularized by Donald Rubin beginning in 1974
• Focused on potential outcomes - counterfactual conditional results
• i.e. What would happen to the control group if they were given treatment?
• Necessary Assumptions:
• Randomization of treatment assignment
• Or “as-if” randomization - this requires a conditional independence assumption
• i.e. treatment assignment is unconfounded or not selected on observables
• Potential outcomes for a unit should be unaffected by the treatment assignment status or response to treatment of other units in the study group
• i.e. no interaction effects / network effects / spillovers / etc.
• Simple analysis: comparison of treatment & control means
• Check for statistical significance using either: t-test, z-test, or Fisher’s exact test

​

Neyman–Rubin Potential Outcomes Framework

• Let t=treatment, c=control, u=unit, Y=outcome
• We only ever observe either Y_t(u) or Y_c(u). Causal inference is a missing data problem

​

​

​

​

• Random assignment allows us to use the control observations to fill in the missing outcomes for the treated observations (on average)

Why should we believe “as-if” random?

• Qualitative evidence:
• Information
• Do units have information that will be/are being exposed to a treatment?
• Did students/faculty know that only some campuses were using pop-up chat on search result pages? Did they care?
• Incentives
• Do units have incentives to self-select into treatment or control groups?
• Would using the catalog and pop-up chat at a different campus benefit students/faculty?
• Capacities
• Do units have the capacity to self-select into treatment or control groups?
• Could students/faculty have used the catalog at a different campus just to get reference help via pop-up chat?

Why should we believe “as-if” random?

Methods

• Looked back at two years of daily chat traffic for each institution
• Retrospective data (data prior to “as-if” randomization) is required to check for the conditional independence assumption
• From 2016/07/01 through 2018/06/30
• Our sample performed acceptably on qualitative and quantitative checks of “as-if” randomization so we could then apply the Neyman-Rubin model
• Network effects / interaction effects seemed implausible
• Two comparisons of means
• (Average Treatment Effect = mean of treatment group - mean of control group)
• 1st: absolute basis
• 2nd: percentage basis relative to prior year
• t-tests for statistical significance
• Supplemented by graphical checks

LB changed from 180 second delay to 360

Results

• Average Treatment Effect: 13.49 chats per day
• Post-treatment only (this is standard use of Neyman-Rubin model for non-relative calculations)
• Standard error: 1.26
• p = 0.00
• Average Treatment Effect in percentage terms: 267% increase
• Compared against previous year
• Sample size smaller since previous year values were not available for each date due to Long Beach and Dominguez Hills beginning treatment on different dates; also null values filled for days when previous year value was zero (can’t divide by zero)
• Standard error: 0.26
• p = 0.00

Limitations

• Did not measure where chats originated - our Average Treatment Effects err towards overstatement
• Did not take timing of the pop-up (which clearly has some effect) into account
• Only applicable to LibChat’s pop-up chat on Primo search result pages with anywhere from a 120 second to 360 second delay
• Not generalizable to:
• Placement of a non-pop-up chat widget on search result pages
• Long Beach removed pop-up chat and went to passive chat and saw a ~50% decrease
• Placement of pop-up chat widget onto other locations

Other Research on Pop-Up/Proactive Chat

• Epstein (2018) found that placing Springshare’s LibChat pop-up chat on homepage with 60-sec. delay increased chat traffic by 600%
• Kemp, Ellis, and Maloney (2015) found, using Zopim chat with pop-up proactive and user initiated passive widgets, that:
• 56% of chat traffic came from pop-up widgets
• Their mixed deployment increased traffic by 340%
• 81% of their pop-up chat traffic was “complex” - READ level 3 or greater

Other Research on Pop-Up/Proactive Chat

• Among their findings, Rich and Lux (2018) noted:
• Their discovery system (Summon) generated the highest number of proactive chat engagements, followed by the library catalog
• 132% increase in chat traffic over previous year
• Altered their staffing model to an on-call system to accommodate increase in traffic level and complexity
• Pyburn (2019) found that adding proactive chat to Summon pages and ProQuest databases doubled traffic
• Staffing adjustments required

Other Research on Pop-Up/Proactive Chat

• Through an analysis of chat transcripts, Meert-Williston and Sandieson (2019) found that:
• 19% of questions analyzed (475 out of n=2,529) required a high level of library expertise to answer, and even fewer - 9% - if citation/copyright questions were excluded
• Change in staffing model to have library assistants as first responders
• Fan, Fought, and Gahn (2017) recommend installation of proactive chat on strategically selected pages and services.
• 18% of chat traffic came from pop-up widget on a single webpage in a 2-year period, compared to traffic from over 100 pages with static widget

Workflow/staffing configurations to manage traffic

Dominguez Hills:

• On-call staffing model for Research Help Desk
• Frontline response: student assistant on desk, 1 librarian on chat and on call for in-person queries
• Creating backup staffing workflow for peak hours
• Created second queue for circulation/course reserves/space support
• Created third queue for asynchronous electronic resources error reporting through Primo
• Triage missed chat attempts for later response
• Pop-up chat also in databases

​

Workflow/staffing configurations to manage traffic

Long Beach:

• One chat queue monitored by librarian also working the Research Help Desk
• Tremendous stress, many complaints
• Increased pop-up delay timer from 180 seconds to 360
• Created volunteer backup staffing workflow for peak 4 hours mid-day
• Previous day’s missed or dropped chats followed up on by librarian working the opening morning shift
• Eventual removal of pop-up chat
• Currently chat widget appears on all Primo pages but is only user-initiated
• Opens a new window, no dropped chats

​

Workflow/staffing configurations to manage traffic

Fullerton:

• Two librarians now monitor chat and error reports, up from one
• Created queue of experts to deal with issues specific to Primo
• Most issues still reported by link -including for reference questions

​

​

Find / “Conduct” Your Own Natural Experiments

Neither natural nor experimental

• Observational studies with convincing/plausible randomization procedure
• Randomization is what allows simple statistical analysis
• An alternative to A/B testing
• Published/public natural experiment results allow for real-world reliable measurements that other libraries can rely on

Figure from Dunning, 2012, p. 31

Natural Experiments Flowchart

As-if random?

How was “treatment” assigned?

Information / Incentives / Capacities

Formal statistical tests

Pre-treatment tests on important variables

Outcomes under potential randomizations

No

Not a natural experiment

Use alternative research design

Neyman model applies?

Noninterference assumption

Simple data analysis

Difference of averages

Statistical hypothesis testing

Conservative t-test

Interpretation

What *is* the “treatment”?

Limitations

Idiosyncrasy of treatment

Yes

Y

Y

Yes

No

Are extensions or adjustments to Neyman model possible?

Consult Dunning (2012) for details and alternatives

Figure adapted from Dunning, 2012, p. 329

Yes?

Contact Information / Questions?

• Gabriel Gardner - CSU Long Beach gabriel.gardner@csulb.edu
• Joanna Kimmitt - CSU Dominguez Hills jkimmitt@csudh.edu
• Mike DeMars - CSU Fullerton mdemars@fullerton.edu
• Sarah Baker - CSU Los Angeles Sarah.Baker6@calstatela.edu

​

​

​

​

​

References

Dunning, Thad. Natural experiments in the social sciences: a design-based approach. Cambridge University Press, 2012.

Epstein, Michael. "That thing is so annoying: How proactive chat helps us reach more users." College & Research Libraries News 79, no. 8 (2018): 436. https://doi.org/10.5860/crln.79.8.436

Fan, Suhua Caroline, Rick L. Fought, and Paul C. Gahn. "Adding a Feature: Can a Pop-Up Chat Box Enhance Virtual Reference Services?." Medical reference services quarterly 36, no. 3 (2017): 220-228. https://doi.org/10.1080/02763869.2017.1332143

Kemp, Jan H., Carolyn L. Ellis, and Krisellen Maloney. "Standing by to help: Transforming online reference with a proactive chat system." The Journal of Academic Librarianship 41, no. 6 (2015): 764-770. https://doi.org/10.1016/j.acalib.2015.08.018

Meert-Williston, Debbie, and Rachel Sandieson. "Online Chat Reference: Question Type and the Implication for Staffing in a Large Academic Library." The Reference Librarian 60, no. 1 (2019): 51-61. https://doi.org/10.1080/02763877.2018.1515688

Pyburn, Lydia L. "Implementing a Proactive Chat Widget in an Academic Library." Journal of Library & Information Services in Distance Learning 13, no. 1-2 (2019): 115-128. https://doi.org/10.1080/1533290X.2018.1499245

Rich, Linda, and Vera Lux. "Reaching Additional Users with Proactive Chat." The Reference Librarian 59, no. 1 (2018): 23-34. https://doi.org/10.1080/02763877.2017.1352556

​

​

​