1 of 22

Title Lorem Ipsum

Sit Dolor Amet

Comparing Primo Research Assistant to Other Academic AI Search Tools

​

John Wenzler (Cal State East Bay)

eCAUG 2025, SFSU, October 31

2 of 22

Agenda

  • Motivation
    • Challenges with GenAI for research
    • Options (Primo Research Assistant & other stuff)
  • Methodology
  • Results
  • Discussion

3 of 22

Challenge -- Hallucinations

4 of 22

Challenge – Grounding (lack of)

5 of 22

Challenge – Academic Paywalls

6 of 22

Challenge – Bias

7 of 22

Three ways of addressing challenges

  • Frontier models adding stuff (reasoning, research, etc)

​

  • Specialized Academic AI

​

​

  • AI in traditional library interfaces

​

8 of 22

Improvements on the Frontier

Iterative reasoning and research

9 of 22

Academic AI

10 of 22

AI in Library Tools

11 of 22

Aaron Tay – Does Relevance Matter Again?

https://aarontay.substack.com/p/were-good-at-search-just-not-the

12 of 22

Methodology

  • Get actual Cal State East Bay reference questions from LibAnswers

​

  • Hack together a rubric for evaluating the relevance of search results for PRA, Elicit, Consensus, & Claude

13 of 22

Questions 1-6

  1. I have a project to do for a management course. I need information on how workers' poor reading and writing skills negatively affects business productivity. I need info on background/introduction,, major causes, and different perspectives.
  2. Looking for articles of how mergers in healthcare happened due to covid.
  3. hi can you help me find articles on women of color dealing with postpartum and mental health issues
  4. Hi, I need articles, journals or books for this topic Topic: "Are the UN Sustainable Development Goals (SDG) Transforming the World?
  5. I need help finding a popular and scholarly source for my research paper. My question is What are the most effective teaching methods for promoting language development in elementary students? I would love my sources to focus on students learning a second language, being English and effective methods to do so. I also am looking for my sources to be recent as well!
  6. hello i am trying to find articles on the use of aspirin to prevent preeclampsia in pregnancy of black women. i am coming up with nothing when i search

​

​

14 of 22

Questions 7-11

  1. I'm having a really hard time finding articles about Carl Jung and Daniel Levinson and the theory of finding balance
  2. Hi, I'm looking for articles about Ethical Dilemmas Nurses have to deal with. The main topic our class is focusing on is Patient Autonomy Vs Medical Advice. Are there any articles than may correlate with this topic?
  3. im doing research on why DuBois makes the case that the failure of Reconstruction was both a ‘counter-revolution’ and a return to slavery. but I do not know how to start looking up scholarly references for this subject, please help me
  4. Hello, im trying to find a scientific primary source about soy bean farmers and how that relates to the deforestation of the amazon
  5. Hey! Can you find me 8 sources related to cancel culture

​

15 of 22

Relevancy Rubric …

Aboutness

Is the article about the topic of the query, does it partially address the query or is it unrelated. -- Score range 1-5, weighted double

Quality

My assessment of the abstract. Does it have a strong argument and good evidence. -- Score range 1-5

Crediibility

Is it an academic source; is it a highly rated journal; does it have a high citation count --- Score range 1-5

Recency

Over 10 years old = 1; 5-10 years = 2; less than 5 = 3

Diversity (per query)

Are the sources repetitive or do they reflect a variety of views and perspectives

16 of 22

Tools Compared

  • Primo Research Assistant
  • Consensus
  • Elicit
  • Scite_
  • Claude (default version)

17 of 22

Results Per Query

18 of 22

Average and Range

  1. Claude – 17.2
  2. Consensus – 16.7
  3. Elicit – 16.0
  4. PRA – 15.5

​

19 of 22

Claude’s DOI Problem …

20 of 22

Grounding Still a Challenge

21 of 22

Discussion

  • Competition or integration? [MCP (Model Context Protocol) - like a Wiley AI Gateway ]
  • Use cases for different types of models
  • Measuring Relevance???
  • Would I turn on Primo Research Assistant?

22 of 22

Questions/Contact Info

John Wenzler

Cal State East Bay

john.wenzler@csueastbay.edu