Smarter Searches, Deeper Discovery: Integrating Natural Language Search in Primo

Matthew Hartman, Timothy Kohn

October 17th, 2025

Who We Are

Matt Hartman

Senior Lead, Delivery Services & Library Applications

​

​

Tim Kohn

Resource Sharing Manager

2

Stony Brook Libraries, AI

3

Stony Brook Libraries, AI Team

4

Stony Brook Libraries, AI Team

5

6

7

Primo Analytics

8

7.5% of searches no results

​

Around half of sessions have more than one search

​

Facet usage is startlingly low

​

Growing number of natural language-style questions

​

​

Does the World Cup promote world unity or increase nationalistic tensions?

​

How does expenditure affect the percentage of votes for a candidate?

​

During the American Revolutionary War did local militia service records show that families with more property were more likely to enlist?

​

How did the penalties of witchcraft get worse over time?

​

9

10

11

12

13

14

15

16

17

(From ENUG 2024) What do users need from our systems?

18

Quick, reliable discovery of resources

Quick, reliable delivery of resources

Support for information literacy

Illustration of and support for skills that can be used in other platforms

Compelling incentives to use library tools to discover & access materials purchased/licensed for their research

Low stress, high reward, navigation through the research process

Webb, H. & Babb, N. (2024, October 18) Artificial Intelligence Across the Discovery Landscape. [Conference session]. Ex Libris Northeast User Group (ENUG) 2024, , New Brunswick, NJ, United States.

19

Varnum, K., Kessler, R., Zhu, J., Hazen, T., Patham, B., & Holloway, J. (2025). Generative Artificial Intelligence and Web-Scale Discovery: A report from the NISO Open Discovery Initiative Standing Committee

20

Varnum, K., Kessler, R., Zhu, J., Hazen, T., Patham, B., & Holloway, J. (2025). Generative Artificial Intelligence and Web-Scale Discovery: A report from the NISO Open Discovery Initiative Standing Committee

21

Varnum, K., Kessler, R., Zhu, J., Hazen, T., Patham, B., & Holloway, J. (2025). Generative Artificial Intelligence and Web-Scale Discovery: A report from the NISO Open Discovery Initiative Standing Committee

Painting the picture

Focus more on…

• Quick delivery of resources
• Enhanced discovery of related materials
• Support research skills & literacy
• Incentivize using library tools
• Librarian oversight

Focus less on…

• Summarization
• Circumventing the research experience
• Separating AI from the Discovery system
• Convoluted search process

22

We can do that!

23

Asks Question

Rewrites in Boolean string

Top 30 Search Results

Top 5 for user query

Knowledge

Base

Summarization

4o mini

4o mini

24

Output from Primo RA = 925,163 results, uses top 5 usable CDI results

(factors contributing to global warming) OR (causes of global warming) OR (global warming contributors) OR ((global warming) OR (climate change causes)) OR ((human activities) AND (global warming factors)) OR ("factors contributing to global warming") OR (greenhouse gases climate change global warming causes) OR (((anthropogenic) OR (human-induced) AND (global warming) OR (climate change) AND factors)) OR (carbon emissions deforestation industrialization global warming) OR (climate drivers)

OR (What factors contribute to global warming?)

25

26

27

28

29

30

31

library.edu/discovery/search?query=any,contains, [your query here] & [your default settings here]

Issues

• Material information was always added to the search string
• Material type (books, articles, etc.)
• Creation dates (... OR “1942” OR “1943” OR …)
• “Peer-reviewed”
• “Held in our library”
• “Available online”

32

33

&facet=tlevel,include,peer_reviewed

&mfacet=tlevel,include,online_resources,1

&mfacet=tlevel,include,available_p,1

&mfacet=searchcreationdate,include,2020%7C,%7C2025,1

34

Query

Topic

Date

Facets

35

Topic

(factor* OR cause* OR contributor* OR driver*) AND ("global warming" OR "climate change" OR "greenhouse effect")

36

Date

2020-2025

Recent

Format

Book

Local

37

library.edu /

search = BOOLEAN� & date = RECENT� & format = LOCAL BOOKS

Benefits of Agentic Model

• Narrower focus makes the process faster and less expensive
• Reduced guardrails on controversial topics
• Librarian control
• Flexibility

38

User Interface Considerations

• Integrated into the existing search experience
• Working in Primo takes patience
• Some elements are easier to manipulate than others
• Overlaying of search bar
• Toggle replaces the entire search bar

39

40

User Interface Considerations

41

User Interface Considerations

Homepage Widget

​

42

43

Codebase

​

44

Primo

OpenAI

Web Server

Timing

45

Top 10% (p90) Latency - 90% of all queries load in under 5.5s, indicating predictable system speed for nearly all users.

OpenAI vs AI Server

​

OpenAI version (GPT 4.1-mini)

• Trained model on boolean creation.
• High success rate in natural language queries correctly identify facets, topics, time periods, and correctly format them as boolean expressions.
• 1 cent per ~90 searches

​

AI Server (llama 3.1)

• Untrained model.
• Reliability & repeatability isn’t at a level suitable for research use.
• No API Costs.
• Local power costs.

​

46

Require Sign In?

Capture more data at the cost of limiting usage.

47

Data

• What do we have access to?
• JSON capture | Processing speed | Comparative Primo analytics
• What do we want access to / How do we evaluate this tool?
• Based on feedback…

48

• How did you feel about the process?
• Do you want to use AI in this context?
• Did you get results you needed?
• Would you use this again?

• Number of results (potentially misleading)
• Session data, ie How many searches before reviewing results?

We are retaining no personal information. No gathered data is used in training.

API Costs

​

49

.04 cents per search

​

4.5 cents per 100 uses

​

~ a dollar a month

Listed Pricing

Actuals

Usage (10/1 - 10/14)

​

50

1245 Advanced

​

421 SEARCH AI

Usage (10/1 - 10/14)

​

51

Informational

topic-based or reference-seeking queries like:

​

“AI bias and facial recognition” or

​

“Hamlet by Shakespeare”

​

Exploratory

multiconcept or relational searches like:

​

“give me peer reviewed articles about the future of creative writing and its relationship with AI”

​

Creative

open-ended or speculative queries:

​

“How does a college student's mental health affect their grades. Has it changed since the pandemic?”

​

​

Revisiting the goal

Focus more on…

• Quick delivery of resources
• Enhanced discovery of related materials
• Support research skills & literacy
• Incentivize using library tools
• Librarian oversight

Focus less on…

• Summarization
• Circumventing the research experience
• Separating AI from the Discovery system
• Convoluted search process

52

Limitations and User Feedback

• No discussion or memory
• No instruction of what prompt to use or what is possible
• “... appears to rely on knowledge that novice researchers might not have”
• Limited training data (in development)

• Exact title matching
• Some may not adjust the Boolean string to fit their needs
• “... concerned that users would think that this is an exhaustive search and not delve deeper”
• Time to completion could be faster

53

What did we learn

Future Directions

• Underutilized search techniques
• Sensitive relationship between training data and system prompting
• Building systems, not just “solutions”
• Benefits of full control over this system

• More facets, more agents
• Confidence scores and reinforcement
• Chat-like instructive interface
• Updating the UI
• Using this system elsewhere
• Bento search, librarian contacts, Interlibrary Loan

54

search.library.stonybrook.edu

https://github.com/Stony-Brook-Libraries-Digital-Services

Learn more about this project and more!

THANK YOU!�ANY QUESTIONS?

We’re looking forward to hearing from you!

matthew.hartman@stonybrook.edu

timothy.kohn@stonybrook.edu