1 of 28

Expanding the ICPSR Bibliography of Data-related Literature to Study Communities of Data Reuse

Sara Lafia (Research Fellow, University of Michigan)

Elizabeth Moss (Librarian, ICPSR)

ICPSR Data Fair

September 19-23, 2022

2 of 28

Agenda

Context: The existing ICPSR Bibliography
Activity: Audience Poll - Data Citation Best Practices (Zoom)�
Visualizing and exploring the ICPSR Bibliography
Activity: Audience Discussion - Feedback on Visualization (Zoom)�
Resources�
Q&A

2

3 of 28

Context: The existing ICPSR Bibliography

3

4 of 28

Database of data-literature links

Over 100,000 linked citations

Meant to help you discover data via the citing literature

Curated by four ICPSR Bibliography staff

4

Citation

Full

text

ICPSR Study(s)

5 of 28

5

You see the contents of the database whenever you conduct a search.

As you probably know, all searches on the ICPSR websites provide results from several databases. The Bibliography’s search results show up in the Data-related Publications tab that the orange arrow points to.

In this case, the search on the word, Maternal, brought back 816 citations from the Bibliography’s database. Note the grey box containing a LEGEND labeling the icons that stand for the types of publications we collect.

There are 11 types and they cover both peer reviewed and grey literature. Everything from journal articles to dissertations, books, reports, conference presentations, and more.

Note the filters on the left are a bit different than those for a study search. Any search conducted in the Bibliography is not of the full text publication, but rather the citation elements, themselves.

This includes the publication title, source title, author. Basically the things you see there on the page in the first two search results hits. You can see clearly where the word maternal was matched in the citation metadata of these two hits. Stemming on matern was used in the first hit to match with maternity, and the other uses the word maternal in the title.

Because the database does not contain full text, it means you won’t have as many hits in your search, but the ones you do have are meaningful.

Before we leave this slide, I wanted to point out two more things.

One is that I’ve sorted on recent additions. The Bibliography team adds publications as we find them, no matter the year. For example, we recently added a 1989 and a 2022 article relevant to this search, each using a different dataset. Right now, the Bibliography team is in the midst of adding many publications that cite a huge variety of studies in the large general archive that we’ve found by doing a large API query. I’ll talk about that a bit later. But if you check back on recent additions occasionally in the next few months, you may discover publications about studies you hadn’t know existed in our vast Membership-funded collection.

Finally, the main list of 816 citations with links out to the full text and links to the related studies are here in the middle. But in the filters, you can sort on study and get a full list of all the studies. It’s a nice consolidated view of all of the studies out there that may have some relevance to your topic of interest.

Okay, so another place you see the contents of the Bibliography database is on each study home page.

Click

For example, on this study home page you see the number of publications we’ve collected for that study on the right. In the case of the Maternal Lifestyle Study we have 120.

Clicking it takes you to that Data-related Publications tab, which then lists all of the publications we have found for that study.

In the citation list down the middle, each citation is linked to that study. Or, if more that one study were used in that publication, they all would appear here.

6 of 28

6

htps://www.icpsr.umich.edu/web/pages/ICPSR/citations/

There is another place you can find the Bibliography besides search results and study home pages.

Click

Try going to the Find Data pull-down, and click on Data-related Publications. This gets you to the Bibliography portal page, where you can learn much more than I’m covering today. It contains a search bar and search tips. And in these black boxes, I’ll point out just a couple of resources that you or people you work with may find helpful.

Click

Each week, I post a new Current Events in the Bib. These are article summaries highlighting ICPSR study data that were used by the authors in their analysis. These publications taken directly from the Bibliography. They show how ICPSR data are being used in current research in recently published literature.

Click

A second resource are the Research Spotlights, written by the Bibliography staff using the Bibliography as their source. These short reports synthesize the findings about one or several related topics. They can give students and others a quick lit review of publications using our data about areas of interest.

7 of 28

We LINK studies to publications that use data from those studies.

We attempt to COMPILE the universe of use and reuse for each study by capturing and linking the “citing” publications.

We DESCRIBE these linkages in resources we create based on how the data are discussed in the literature.

7

8 of 28

EXTRACT and VISUALIZE relationships

8

9 of 28

Basics

Main sources
Collection criteria--data analyzed--a collection based on one main type of data use

Manual Searching

Scale of collection
Customized queries and human verification

Automated Searching

API querying
Non-analysis usage
Data citation implications

9

10 of 28

Main sources

People

Depositors

Public and restricted data reusers

Research teams

New Sources

Overton Interdisciplinary

Policy Commons

DimensionsPlus

Platforms

Google Scholar

PM/PMC

ProQuest

Scopus/ScienceDirect

EBSCOhost

Individual Publishers/Journals

Wiley

NCJRS

Journal ToCs

10

I wanted to say a word about the main sources of publications that are added to the Bibliography.
On the left of the purple line, you will see that some publications are submitted by people who comply with our terms of use and send us citations after they publish using data from ICPSR. Others, like ongoing research teams, send us publications that they themselves are tracking.
But the vast majority of publications in the Bibliography are added and linked to studies by the Bibliography staff, who search in various sources. Here are some of the main ones, though they can vary depending on the topic of a study. I just wanted to add that two new policy databases are out there that seem very interesting and that we will be harvesting from. And finally, although we had used it a bit in our manual searches, Sara has been utilizing the Dimensions API to bring back results. She and I will both talk about that in a minute.

11 of 28

Collection criteria

11

https://www.icpsr.umich.edu/web/pages/ICPSR/citations/collection-criteria.html

12 of 28

12

1. Is this already in the Bibliography? If no . . .

3. Where does query string appear in text? (tables, footnotes, methods, supplement, acknowledgements, grant number?)

4. If a good “hit” (data analyzed) – 5. If a bad “hit” (just mentioned or pub cited)— 5a.

If 5a, find cited pubs & check for #1--#5.

If 5. Does ICPSR catalog have same years, waves, panel, sample?

2. Were the data formally cited in the references? If no . . .

6. Collect citation and associate with study numbers from ICPSR catalog.

7. Was any year/wave /phase clearly analyzed but not cited in the references? Add those.

8. Did author use something in a series, but didn’t say which specific study? Collect for series.

https://www.icpsr.umich.edu/web/pages/ICPSR/citations/biblio-ref.html

Heuristic to Evaluate Data Use in Publications

This is a diagram of our decision-making within a paper as the ICPSR Bibliography team member evaluates it for collection. I mentioned that we try to try to link and compile the citing literature. But citing is not done very well or consistently. So a lot of human judgment and verification must take place. And we must make customized queries to find informally cited data.

We have to look for our queries within the full text, to determine if that paper is collectible, and if so, what study number or numbers should be linked to it. I won’t read the full heuristic, you can see it on the About the Bibliography page from the Bibliography portal.

This is to show you that even though the Bibliography collection may not reflect perfectly the universe of citing literature for each study, we can say that in the collection we have found, the relationship between the publication and the data reflect true usage, not, say, mere mentions.

13 of 28

Study DOI citation practice

We evaluated:

Dimensions API query results
DOIs for 11,000+ ICPSR studies
Search results examined for collectibility

We found:

We could collect 626 publications that met our collection criteria
We had to reject 1,031 that did not?
Why were study citations used if not to cite data analysis?

13

So, given the human intervention needed, it is hard to collect full, up to date Bibliographies of the citing literature for over 11k studies, especially in an environment, spanning 60 years, where authors have acknowledged data usage data inconsistently. Sifting through the hits to distinguish between analysis and other types of use is revealing. I’ll now describe a project that is a great illustration of that.

In an effort to make the collection more complete, Sara took advantage of the API for the Dimensions platform. On the left of this slide, you’ll see she sent out a DOI query for 11k plus studies. All of the studies in the ICPSR collection have been registered with a unique DOI since 2008. So her DOI query went that far back in the publications. (As an aside, and as I alluded to earlier when I showed you the recently added sort, Sara also has queried the API on the full study names and study numbers, run on the full collection, and that is why we are still verifying and adding a lot of older publications using older studies.)

The right side of this slide shows that as we evaluated the publications that came back in the DOI query, we found many that had not yet been collected. So we added them, thereby expanding the Bibliography.

We often reject papers for that don’t meet our collection criteria. But we really hadn’t noted how many times we had rejected publications that cite the study DOI until this project.

We collected a little over 600 publications that we hadn’t found before. But there were almost twice as many that we rejected.

14 of 28

14

https://dx.doi.org/10.7302/4809

When and what

should you cite?

15 of 28

Archive-provided study citation/DOI may be convenient, but . . .

More clarity is needed from various authorities about when to do so:

Data/object producer
Distributors of record
Style guides
Journals’ author instructions
Instructors assigning works containing use of data

What deserves a citation? In what form?

Methodological concepts, inspiration
Items from survey instruments
Mentions of similar sources
Mentions of brief data points
Constructed measures
Codebooks

15

16 of 28

Activity: Audience Poll

In my publications, I cite, or recommend that others cite, data when I

(select all that apply):

Describe a study or a feature in a study (e.g., a variable)
Refer to study documentation (e.g., a codebook)
Use questions from a study in my work
Credit a study design or methodology that inspired my approach
Reuse the study's data in my analysis
Publish data I derived from analyzing an existing study
Other

16

17 of 28

Visualizing and exploring ICPSR’s citation network

17

18 of 28

Using the ICPSR Bibliography to study scholarly communication and impact

18

What are researchers data search needs?

What impact does data curation have on data reuse?

How should we prioritize curation to achieve impact and return on investment?

NSF-funded projects: MICA and RecSys

Source: ACRL (2021)

19 of 28

What can we learn from studying citation networks?

19

Example from 150 years of Nature: linked articles show the flow of ideas that inspire across disciplines

I am interested in studying citation networks to better understand how communities engage in research. Some of you may have seen this visualization published in the journal Nature a few years ago. This is an example of a citation network of articles (dots) that are color coded by field and linked if they are cited together. With visualization, we can start to see how ideas might flow across disciplines. For instance, you can see some earth and space papers (green) that are cited with physics (blue) and chemistry (red) papers. These disciplines are more closely aligned and researchers are more aware of research in these fields. Certain papers may also be prominent in the network because they are well-known and are cited across fields. Citation networks can tell us more about how knowledge is structured and the roles that scholarly works (papers in this example) play in connecting these academic communities.

20 of 28

Modeling data citations in the ICPSR Bibliography as a network

20

Teenage Attitudes and Practices Survey, 1989: [United States] (ICPSR 9786)

National Survey on Drug Use and Health (NSDUH) Series (1979-2014)

Crime, juvenile delinquency, and dysfunctional behavior (Sherman, 2003)

SERIES/STUDIES: Which data are used together?

PUBLICATIONS: Which disciplines are using data?

Central America and the international trade in drugs (Bunck et al., 2015)

I model data citations in the ICPSR Bibliography as a citation network. The connections between data and publications form the backbone of the network and they tell us about the research communities that engage with ICPSR data. For example, the Bibliography captures relationships between publications and the studies/series that they use. This lets me see which data are often used together. Here, we see two papers (pink) that use ICPSR data (blue). Because these ICPSR data are used together in a publication, they are co-cited. About 60% of ICPSR studies are part of a series, meaning that they are part of a recurring collection that has new data archived over time (e.g., repeated cross-sectional studies or longitudinal studies). If studies are part of a series, I group them into a single node. If they are not part of a series, they are modeled individually.

21 of 28

Overview of the ICPSR data co-citation network

21

Nodes: ICPSR studies or grouped series

Edges: Studies used together in 2+ publications

Edge weight: Number of co-citations

When we look at the entire co-citation network, we can start to see the relationship between studies and publications at scale. This is an overview of the ICPSR study co-citation network. Here, each node (dot) is one ICPSR study. The nodes are linked by an edge if they are used together two or more times across publications. Edges are heavier if the data have been used together more often and lighter if they are used together less often. The structure of the whole network has a strongly connected core with some co-cited studies on the periphery. This means that data in series or studies (in the core) tend to get used with more other studies (highly linked), while those along the edges are used with fewer studies (are less linked). Note that if an ICPSR study has very few or no citations in the ICPSR Bibliography, it is not included in this analysis. About 75% of ICPSR studies currently have at least one publication in the Bibliography.

[unhighlight the center nodes?]

22 of 28

Finding “hubs” in the ICPSR data co-citation network

22

General Social Survey Series

American National Election Study (ANES) Series

Current Population Survey Series

Census of Population and Housing, 1790-1950 [United States] Series

National Health Interview Survey Series

Uniform Crime Reporting Program Data Series

Studies

High-degree

High betweenness

Both

If we zoom in on the center of this network, we can see a few important studies and series jump out, which are highlighted in the inset. These important datasets are long-running series made up of multiple studies. Data in these studies/series function as hubs in the network, meaning that they are highly connected to many other studies. We can use some network-based measures to characterize how data in these series are used. For example, the series in red (National Health Interview Survey Series) is “high degree” meaning that it is highly connected with other studies and series in the network. It is prominent in the network because compared to all other studies/series, it is used frequently with other data. The series in blue (Census of Population and Housing, 1790-1950 United States Series) has “high betweenness”, meaning that while is is used less frequently with other data, it plays an important role in connecting different parts of the network that would otherwise be disconnected. This can be interpreted to mean that this study has broader disciplinary utility. Finally, the four studies in purple have a high degree and a high betweenness centrality, meaning that they are heavily used with other ICPSR data and are integrative. Of these, the Uniform Crime Reporting Program Data Series has the highest degree and betweenness. It has been used with 115 other datasets from studies or series and has strong ties to many other datasets.

23 of 28

Detecting communities of data use

23

41 communities (minimum size of three studies/series) labeled with most common subject terms

To find collections of datasets that are often used together in publications, I used a technique called community detection to see if the data in the co-citation network can be easily grouped into communities of use. I used clique detection, which allows each node to belong to more than one community and discovers communities algorithmically by grouping nodes into (potentially overlapping) clusters, which are densely connected. I found 41 communities, which are summarized in the legend on the right. Each community is labeled based on the three most common ICPSR subject terms assigned to studies during curation. The largest clique has 461 dataset members and is topically broad (e.g., “demographic characteristics, employment, income”) while smaller

cliques tend to have narrower focuses (e.g., “terrorism, terrorists, radicalism”). For example, the small community in green on the right portion of the visualization has three studies, which are used together exclusively, and are labeled with the terms “crime rates, citizen participation, and citizen patrols”. Only a fraction of datasets in the analysis (N = 632; 63%) belonged to cliques of size three or larger; these datasets are often analyzed with at least two additional ICPSR datasets. The datasets that fell out of our analysis were used independently and were not combined with other datasets.

24 of 28

User stories: data “subdivisions”

24

terrorism, terrorists, radicalism

e.g., Lone Wolf Terrorism in America: Using Knowledge of Radicalization Pathways to Forge Prevention Strategies, 1940-2013 (ICPSR 36107)

police citizen interactions, police effectiveness, police response

e.g., Survey of Victimization and Attitudes Towards Crime and Law Enforcement in Boston and Chicago, 1966 (ICPSR 9085)

inmates, correctional facilities, United States

e.g., Improving Correctional Classification, New York, 1981-1983 (ICPSR 8437)

Disciplinary resources

Scenario: finding compatible, topical datasets for instruction

Now, I would like to share two main takeaways from my network analysis and community detection analysis of the current ICPSR Bibliography: The first insight is that some ICPSR studies and series function as disciplinary resources. On the edge of the visualization, we see smaller, more narrow communities of data use. For example, the pink community (labeled “terrorism, terrorists, radicalism”) is made up of three studies/series that have been used together. These exclusive communities function as “subdivisions” in the network and are marked by more exclusive patterns of use. The fact that data from these studies have been used together multiple times might indicate that data from these studies support specific kinds of research questions. Data from these studies might be a good starting point for instructors seeking compatible, topical datasets for instruction.

25 of 28

User stories: data “crossroads”

25

multiple (substance abuse treatment, program evaluation)

e.g., Evaluation of the Los Angeles County Juvenile Drug Treatment Boot Camp, 1992-1998 (ICPSR 3157)

multiple (demographic characteristics, education)

e.g., India Human Development Survey (IHDS), 2005 (ICPSR 22626)

multiple (terrorism - demographic characteristics)

e.g., Patterns of Behavior in Police and Citizen Transactions: Boston, Chicago, and Washington, DC, 1966 (ICPSR 9086)

Connective resources

Scenario: finding interoperable studies with broad utility for starting a project

26 of 28

Activity: Audience Discussion

Explore the interactive visualization (https://tinyurl.com/icpsr-datasets) with the following questions in mind:

Who do you envision using a visualization like this?
What other information would you like this visualization to show?

Please type your feedback into the Zoom chat (3 min.)

26

27 of 28

Resources

Banaeefar, H., Burchart, S., Moss, E., & Palvolgyi-Polyak, E. (2022). Best practice may not be enough: Variation in data citation using DOIs. [Poster]. Presented at IASSIST 2022. https://doi.org/10.7302/4809

Lafia, S., Fan, L., Thomer, A., & Hemphill, L. (accepted). Subdivisions

and Crossroads: Identifying Hidden Community Structures in a

Data Archive's Citation Network. Quantitative Science Studies

(QSS). https://doi.org/10.48550/arXiv.2205.08395

Lafia, S. (2022). ICPSR Bibliography Citation Network (February

2022) [Data set]. Inter-university Consortium for Political and Social

Research (ICPSR). https://doi.org/10.3886/E174361V1

Lafia, S. (2022). ICPSR/data-communities (Version v1.0.0) [Code].

https://doi.org/10.5281/zenodo.6799127

27

28 of 28

Contact us

Sara Lafia

slafia@umich.edu

Elizabeth Moss

eammoss@umich.edu

Support 60 more years of data: https://myumi.ch/ICPSR-60MoreYears