1 of 34

MEDLINE search retrieval issues: �A longitudinal query analysis of five vendor platforms

Sean Burns�Associate Professor, College of Communication and Information, University of Kentucky

Tyler Nix�Interim Asst. Director, Research and Informatics�Taubman Health Sciences Library, University of Michigan

2 of 34

Where can I find…

Available on the event site https://guides.lib.umich.edu/TaubmanTalks/MedlinePlatforms

Slides (now)
Padlet (now)
Link to article (now)
Search queries file (now)
Recording (~1 week)
Recording transcript (with recording)
Chat and Q&A files (with recording)

2

3 of 34

Today's Talk

What it is:

Presentation of research on search retrieval across Medline platforms
Hear from the authors!
Ask questions!
What do the the results mean for information professionals/expert searchers?

3

4 of 34

Today's Talk

What it isn't:

"How to search Medline" tutorial
Comparison of Medline platform tools and features
"What's wrong with my search?"

4

5 of 34

"What it isn't" resources

"How to search" tutorial(s) / Comparison of Medline platform tools and features

Lots of sources (platform documentation, MLA courses, workshops, Carrie Price's Youtube)

"What's wrong with my search?"

MLA UX Caucus Blog - https://uxcaucustips.blogspot.com/
"What's going on with my search?" Taubman Talk - https://guides.lib.umich.edu/TaubmanTalks/UXDatabaseTips

5

6 of 34

Questions!

We'll pause at 3 points during today's session to answer questions:

Methods
Results
Discussion

Please put your questions for the presenters into the "Q&A" option, NOT the chat!

For questions related to specific platforms that you'd like to crowdsource, please use the Padlet on the event page

6

7 of 34

MEDLINE search retrieval issues: �A longitudinal query analysis of five vendor platforms

Sean Burns�Associate Professor, College of Communication and Information, University of Kentucky

Tyler Nix�Interim Asst. Director, Research and Informatics�Taubman Health Sciences Library, University of Michigan

8 of 34

Research question

When controlling for search query syntax, are search results generally consistent across MEDLINE platforms?

8

9 of 34

Motivation

To investigate how search result counts vary when searching MEDLINE records across five platforms and controlling for search query syntax.
To document longitudinal consistencies and/or changes in these results

9

10 of 34

Methods

Developed 29 sets of search queries
A "set" is a group of equivalent queries for each of 5 MEDLINE platforms (PubMed, ProQuest, EBSCOhost, Web of Science, and Ovid)
145 total queries, run monthly Oct 2018-�Sept 2019
Searches are logically and semantically equivalent (and platform-specific)

10

11 of 34

Methods

Search result counts were compared to PubMed’s MEDLINE as the baseline.
Results were compared using modified �z-scores

11

12 of 34

Acronyms

PMML: PubMed MEDLINE
PQML: ProQuest MEDLINE
EHML: EBSCOhost MEDLINE
WSML: Web of Science MEDLINE
OML: Ovid MEDLINE

12

13 of 34

Query example: Case 02�Single Mesh Term, No Explode

13

PMML	"neoplasms"[MH:NOEXP]
PQML	MESH.EXACT("neoplasms")
EHML	MH("neoplasms")
WSML	MH=("neoplasms")
OML	neoplasms.SH

29 sets of searches using various permutations of field tags, Boolean operators, MeSH terms, from simple to complex

Here is a ‘simple’ example, S02 of 29 (characteristics above with 5 platform-specific iterations below, starting with PubMed/MEDLINE)

Queries were designed to be short and logically clear to enhance the comparability of the results. ��The strategies developed for this study were purposive, but not comprehensive. Intended to reflect many of the basic building blocks of searches that users may employ when conducting research in literature databases. But number of permutations a search strategy can have prevents a comprehensive approach.

Searches are not typical of what SR / systematic searches will likely look like. More like traditional search queries.��For instance, each term, field, limiter, and combination offers a new chain of outcomes to examine.

14 of 34

Query example: Case 11�Single Mesh term, ≥2 branches, exploded, keyword, and date: 1950-2015

14

PMML	"dementia"[MH] AND "immune"[ALL] AND 1950:2015[DP]
PQML	MESH.EXACT.EXPLODE( "dementia") AND NOFT("immune") AND YR(1950-2015)
EHML	MH("dementia+") AND TX("immune") AND YR 1950-2015
WSML	MH:exp=("dementia") AND TS=("immune") AND PY=(1950-2015)
OML	1. EXP dementia/ AND immune.AF 2. limit 1 to YR=1950-2015

15 of 34

Methods Questions

with Mark MacEachern

15

16 of 34

Results

Results indicate there is substantial variance in search result counts across 5 MEDLINE platforms.

16

17 of 34

Results

Walk-through of figures from paper:

Burns CS, Nix T, Shapiro RM 2nd, Huber JT. MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLoS One. 2021 May 6;16(5):e0234221. doi: 10.1371/journal.pone.0234221. PMID: 33956834; PMCID: PMC8101950.

17

18 of 34

Results, Fig. 3

Viewing results at macro (group) scale shows broad differences between the platforms…� �but does not capture fluctuation within each platform.

18

19 of 34

Fig 3. Macro view of cases #03–#11.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

20 of 34

Results, Fig. 4-6

Viewing results at micro (individual) scale shows significant fluctuation within platforms throughout the year.

20

21 of 34

Fig 4. Micro view of cases #03–#11.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

22 of 34

Fig 5. Micro view of cases #03–#11.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

23 of 34

Fig 6. Micro view of cases #03–#11.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

24 of 34

Results, Fig. 7-8

Variance and deviation in search result counts compared to PubMed, using modified z-score

24

25 of 34

Fig 7. Deviations per case and platform.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

– each case/platform combination is either not present (no deviation), OR present fig.7 (deviation), OR present fig.8 (outlier)

Fig 7. Deviations per case and platform, not including outliers

Deviations per case and platform with deviations centered around PubMed MEDLINE. Cases exclude outliers.��We also define our search result count outliers as any modified z-score that deviates more than ±3.5 from PubMed, as recommended by Iglewicz and Hoaglin [53]. However, statistical outliers and practical or clinical outliers are different issues. The modified z-score will only highlight the former, but any difference in counts will help answer how MEDLINE platforms deviate from each other.

typically, Z Scores highlight deviations from the statistical center of the distribution of ranges; here they highlight deviations from the annual range of search counts from PubMed

26 of 34

Fig 8. Deviations per case and platform with outliers.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

27 of 34

Results, Fig. 9

MEDLINE update frequency across platforms

27

28 of 34

Fig 9. Search result count differences for COVID-19 related searches across PubMed based platforms.

Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLOS ONE 16(5): e0234221. https://doi.org/10.1371/journal.pone.0234221

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234221

Fig 9. Search result count differences for COVID-19 related searches across PubMed based platforms as of January 2020

At the time, January 2020, PubMed sort options only included PubMed Legacy Most Recent Sort (PMLMR), PubMed Legacy Best Match Sort (PMLBM), PubMed New [Best Match default] (PMN)

ProQuest PubMed (PrQPM), EBSCOhost PubMed (EPM), Web of Science PubMed (WSPM), Ovid PubMed (OPM)

PubMed Medline updates daily; less frequently in the other platforms (Ovid is close; WoS, Proquest, Ebscohost slower)

"Some differences might be attributed to delays in transferring the MEDLINE file to the vendors, since PubMed updates MEDLINE daily but the other vendors update at different times. To test this, we ran a COVID-19 search based on a query provided by the NLM in January 2020 and found that there were uneven search result hits for new literature on the COVID-19 pandemic across platforms. This suggests that if researchers need urgent access to the timely publications, they should be concerned about which version of PubMed they use to collect data."

29 of 34

Results Questions

with Mark MacEachern

29

30 of 34

Discussion

Differences in metadata functionality across platforms (e.g. date and MeSH fields)
Online-first versus issue-based publications impact bibliographic control
Variable timing of MEDLINE update across platforms
Reproducibility appears to improve when combining multiple fields in query

30

We found that queries were more likely to return comparable search result counts when they included multiple and specific field terms, such as queries combining keywords appearing in a journal title or an article title, with an author name, and a relevant MeSH term (e.g., cases #13 and #17).��We show other database-based issues that hamper the reproduction of search strategies across MEDLINE platforms. These include:�

lack of comparable search fields in the studied MEDLINE platforms
inconsistent pattern of results from some MEDLINE platforms that indicate a break in functionality
difference in lag time between updates from PubMed/MEDLINE and other platforms in updating the database

External to these systems, we show how digital publishing complicates the bibliographic record, which was primarily designed in a print era

31 of 34

Open Q&A Time

31

32 of 34

Thanks

Sean Burns – sean.burns@uky.edu�Associate Professor, College of Communication and Information, University of Kentucky

Tyler Nix - tnix@umich.edu�Interim Asst. Director, Research and Informatics�Taubman Health Sciences Library, University of Michigan

32

33 of 34

Wrap up

Available on the event site https://guides.lib.umich.edu/TaubmanTalks/MedlinePlatforms

Slides (now)
Padlet (now)
Link to article (now)
Search queries file (now)
Recording (~1 week) - sign up for notification!
Recording transcript (with recording)
Chat and Q&A files (with recording)

33

34 of 34

Coming soon to Taubman Library Talks…

March/April-ish:

34

https://guides.lib.umich.edu/TaubmanTalks