1 of 54

Data + Innovation in one corner of the Library of Congress

19 May 2023

Laurie Allen, Chief, Digital Innovation Division, Library of Congress

1

2 of 54

Agenda

Situating the Digital Innovation Division
Sharing data for exponential impact

Documenting the API
Sharing data in bulk
Careful enrichment

Call for data users!

3 of 54

About the Library of Congress

https://loc.gov

4 of 54

Our Mission

Engage, inspire, and inform Congress and the American people with a universal and enduring source of knowledge and creativity.

5 of 54

The Office of the Chief Information Officer (OCIO) is the centralized IT organization working to transform the Library of Congress into a data-driven and digitally enabled institution.

OCIO builds and sustains the technology necessary for:

Library Collections and Services
Congressional Research Service
United States Copyright Office

More specifically, I work within the central IT organization of the Library, which we call OCIO. OCIO is responsible for building and sustaining the technology for all three of the big units within the agency… It is a very complex job, but I’ll mention that we maintain over 174 Total PB storage. It’s a lot.

210 Library IT systems maintained

2,700 Servers (physical & virtual) maintained

76 New or significantly enhanced digital collections

Continuously innovate to digitally transform and expand the Library’s ability to meet its mission to Congress and all Americans. �Enable, safeguard, innovate.�We empower the Library’s mission through effective, efficient use of technology. �We provide cybersecurity protections of the Library’s IT systems and data as a national treasure.�We continuously strive….

6 of 54

Digital Strategy Directorate

6

LC Labs

Connecting Communities Digital Initiative

7 of 54

8 of 54

I’m not going to talk about data + innovation in the…

9 of 54

Massive quantities of data produced & processed in the operation of a federal agency

10 of 54

Selected datasets added to the collections

11 of 54

Secure, long term data storage

12 of 54

Congress.gov API

13 of 54

Any of this (sorry… I know this is the metadata research center)

14 of 54

But I would like to talk about users

14

15 of 54

Users

Congress
Creators
Learners
Connectors

16 of 54

Users and scale at the Library of Congress

30,000 (typically 1M) onsite visitors to Capitol Hill campus & 178+ million unique visitors to the Library's web properties

Collection of more than 173 million items includes 51 million cataloged books and other print materials

3,200 permanent staff members

And another reason we need connectors is because we’re really big. Starting from the right, our collections include more than, and in a typical, non-pandemic yar, we have a collection of more than 173 million items includes 51 million cataloged books and other print materials in 470 languages; more than 75 million manuscripts; the largest rare book collection in North America; and the world's largest collection of legal materials, films, maps, sheet music and sound recordings.

Main research arm of the U.S. Congress and home of the U.S. Copyright Office.

And we serve and maintain that with staff that’s a little more than 10x the number of people who worked at Penn Libraries last time I worked there… so we are a LEAN organization.

618M Library website page views

So in addition to the 535 or so people in Congress plus all of their staff, we serve about 1million people on site in a typical year, as well as over 178 million unique visitors to the Libraries websites.

17 of 54

Innovations around access for re-use

18 of 54

History of LC Labs:

Founded in 2016 in National Digital Initiatives (NDI)�

Publicly launched a series of rotating experiments on labs.loc.gov starting 2017�

Joined the Digital Strategy Directorate in (OCIO) in 2019�

Now ending the FY18-23 strategy and moving to the next strategic plan!

19 of 54

Methods

19

Experiments

Convenings

Residencies

And roughly, as a digital innovation division, you can break down our work into experiments – relatively low-risk ways of learning things, testing things, trying new approaches to see what works. And we’ve had lots of those, And then residencies – we know that good innovations can come though testing new approaches, or new technologies, but it also comes through listening to people whose voices or perspectives you can learn from, or from turning the reigns over to someone creative. And then convenings, a good old fashioned meeting should not be ignored. These things are valuable!

Basically, we’re trying to have exponential impact, and that’s how we got into the data game. Because we think that getting our materials out into the world in formats that people can transform is a way to have that exponential impact. We’ve got 173 million unique visitors to our websites, but what about everyone else? How can we get more of the precious stories and materials we steward into the contemporary world. Hint, we want you to help. Well, we keep finding that, in order to have those big impacts, we have to get into the weeds of making our data usefully available. So I’ll offer one example, because I’m not going to talk about any more cool experiments really after this, so I’ll just start with one. I love this one for lots of reasons, but partially because its an example that one way to have that exponential impact that I think we can forget is by leaning into people’s particular visions.

20 of 54

21 of 54

What is the loc.gov api?

Loc.gov API’s was built to make the loc.gov website.

Collections info moves from metadata 🡪 ETL 🡪 API 🡪 loc.gov

Library of Congress metadata and data is *incredibly heterogenous*

22 of 54

API documentation & support

23 of 54

API documentation & support continued!

24 of 54

*Updated* documentation at loc.gov/apis

https://www.loc.gov/apis/json-and-yaml/

25 of 54

Computing Cultural Heritage in the Cloud

Supported by $1M grant from the Mellon Foundation

25

26 of 54

Computing Cultural Heritage in the Cloud

“This effort will help produce models for supporting cloud-based research computing,

and will make the costs and possibilities of this work more transparent to the broader cultural heritage community.”

Supported by $1M grant from the Mellon Foundation

26

Here’s the purpose. Produce models and explore the costs and possibilities for supporting cloud-based research computing. I think we pictured basically like researchers just want the raw data - we’ll set up a space where the data we can legally and ethically share will just be there, and people can just go in and mess with it in a safe way and then they’ll only pull what they need!

And the grant isn’t over yet, but I’m going to tell you that is not how it’s turning out – we’re learning a huge amount about cloud options, and about data – and about what models make sense, but this idea of like “people can just get the stuff” isn’t it. People really want more information, all the time, and they want it in the ways that work for their expectations and needs. They’re willing to do a lot, but they want a start. So, I’ll talk about a couple aspects of CCHC, now that you know what it is, because it’s really a constellation of experiments. But here’s about one convening. Last fall.

27 of 54

CCHC Data Jam (convening!)

Approximately 3 weeks, capped by one day workshop
7 experienced data wranglers:

Accessed Library data in cloud storage environment (AWS S3)
Tested computer program-computer program access methods
Provided in-depth, real-time, authentic feedback on:

current technical set up
computational methods of access
documentation

Lots of great lessons!

28 of 54

Datasets & Access Pathways

Datasets	Description	Access pathway	Participants
Stereograph Cards images	39,526 stereograph card images from the 1850s through 1924.	AWS Command Line Interface (CLI)	Zoe LeBlanc, Aaron Straup Cope
Selected Digitized Books text files	166,218 .txt and JSON files containing full text from 90,414 books in the Selected Digitized Books collection on loc.gov.	AWS REST Application Programming Interface (API)	Tim Sherratt, Quinn Dombrowski & Nichole Nomura
Austro-Hungarian map set	4,998 images in TIFF format representing non-georeferenced map sheets and corresponding GeoTIFF formatted images that are georeferenced.	Software Development Kit (SDK)	Daniel van Strien, Vikram Mohanty

29 of 54

CCHC Data Jam: Findings

Dr. Zoe LeBlanc “Subject Unknown” presentation slide at CCHC Data Jam, 25 October 2022

Dr. Tim Sherratt “American World Gazetteer” presentation slide at CCHC Data Jam, 25 October 2022

30 of 54

CCHC Data Jam Takeaways

Opportunity: They’d like more data translators -- people to answer questions about metadata, gaps in data, and interoperability with other data sources

Opportunity: Create space to support community interaction around the re-use of LC data, and the various trials and possibilities inherent in it

Integrate user feedback and historical/institutional context for ongoing improved access

31 of 54

Aspiration: replicable, useful data packaging

Going to take many attempts!

31

32 of 54

Access to bulk data through labs.loc.gov

32

33 of 54

What is a CCHC “data package”?

Ex: Stereograph Cards image data package

A folder containing 39,526 stereograph image (.JPG) files pulled from Prints & Photographs Division’ Stereograph Cards collection
metadata.json : a JSON file containing the API response for all 39,526 images in the data set
metadata.csv: a CSV transformation of the original JSON metadata
manifest.txt : a text file listing the id (combination of the image ID and image file name), MD5 hash, and location of the images in the data set
README.md : technical overview of how the data set was extracted and processed
Data cover sheet: a more substantive overview of the data and the collection from which it is derived
Sample data: 1,000 randomly selected items from the 39,526 set with their corresponding image files have been provided as sample data. Included with this are their own metadata.csv metadata.json, and manifest.txt.
Data Processing Plan describes transformations, any ML models used, and additional information

https://data.labs.loc.gov/stereographs/

34 of 54

Where might CCHC data packages live?

S3://data.labs.loc.gov

https://data.labs.loc.gov

35 of 54

CCHC Data Transformation Services experiment with contractor AVP

Goal of the experiment:

To understand the benefits, risks, and quality benchmarks for the various workflows, compilation methods, transformations, documentation practices and other tasks required to assemble datasets for public use in the cloud

36 of 54

AVP DTS approach

Extraction

Standardization

Enrichment

Packaging

37 of 54

Guiding questions

	Question	Target Users
General purpose schema	What does an “out-of-the-box” dataset generated from API data look like, beyond just packaging up an API response?	Already skilled in data work who want straightforward, consistent data Want to combine datasets from collections across the Library
Standardized Location and/or dates	How do we make locations and dates more comprehensible and machine actionable for users?	Want to explore spatial or temporal aspects of a collection or plot data on a map or timeline Want to explore/navigate/refine datasets by place or time
Special purpose / enriched dataset	How do we enrich datasets with additional data that will support special purposes, such as network analysis, combination with other datasets, or other functionality?	Users interested in applying techniques requiring datasets to be standardized in a particular way Users interested in extracting data from other sources to combine with their data (ex. Wikidata, Geonames)

38 of 54

Speaking of enriched data….

Machine Learning and AI

38

39 of 54

Ongoing ML experimentation and research

40 of 54

Current Experiments

Exploring Computational Description

Testing 5 machine learning models to generate full bibliographic records from the textual and/or visual elements from 1 10,000+ data set of ebooks.

Generative Data for congress.gov using Machine Learning

Testing 5 machine learning models to examine bill texts, and the U.S. Code or public law text that the bill text may impact, and create relationships between the bill texts and terms from the Geographic Entity and Organization Name vocabularies used by Congress.gov.

Experimenting with ML for Copyright historic record books

Experiment with machine generating structured data from historic record books that document copyright registration activities during parts of the 20^th century.

41 of 54

Lessons learned so far

41

42 of 54

Keep our values and our value at the center

Data users want human-readable context – the more the better

Every time we work with data users, we find they want/need more information about the data – what is it? What is it not? Why is it? How, technically, did it come to be...

Aligning AI potential to our actual needs and our requirements is a challenge

Data enrichment is important and valuable. The opportunities that AI provides are probably huge. The hard part is about matching automation to our values, and maintaining the value of the Library as a trusted authority.

Iteration, creativity, and experimentation WITH people is the key to data

The more we ask people to use our data, and pay attention to what they say while they’re trying, and how much we’re recognizing their visions, the more we learn.

43 of 54

Please help

43

44 of 54

Of the People: Widening the Path

44

Of the People: Widening the Path is a multi-year initiative that creates new opportunities for more Americans to engage with the Library of Congress and to add their perspectives to the Library’s collections, allowing the national library to share a more inclusive American story.

www.loc.gov/ccdi

45 of 54

Artist

or Scholar

in Residence

For artists or scholars whose work connects at the intersection of technology and cultural heritage.

Informational Webinar

The Artist/Scholar in Residence program supports artists or scholars in the creation of new scholarly and/or artists works that imaginatively study, experiment with, and/or critique the Library’s digital collections, materials, and/or services. Proposed projects should center on one or more of the following groups: Black, Indigenous, Hispanic or Latino, Asian American and Pacific Islander, and/or other communities of color in the 50 U.S. states, the District of Columbia, territories and commonwealths (Puerto Rico, Northern Mariana Islands, Guam, American Samoa, U.S. Virgin Islands).

The program is open to a broad array of artists and scholars, including (but not limited to): visual artists, conceptual artists, dancers, journalists, cultural critics, independent scholars, academic scholars, and cultural heritage practitioners, including librarians.

For 2023, the Library intends to award two 12-month grants of up to $90,000 each.

Applications are open May 17, 2023 through 2:00 PM Eastern Time on August 7, 2023.

46 of 54

Grant for Libraries, Archives, Museums

46

Application Deadline: September 7, 2023 by 2pm EDT

Up to $70,000 to support three local cultural heritage organizations by enabling storytelling across a range of platforms. The grants are for up to 12 months.

The funds will support the creation of projects that make significant and meaningful uses of the Library’s digital materials. Projects must also center on one or more of the following groups: Black, Indigenous, Hispanic or Latino, Asian American and Pacific Islander and/or other communities of color.

https://www.loc.gov/programs/of-the-people/represent/libraries-archives-museums-application/

This year, we'll award up to 3 grant awards of up to $70,000 each to support three local cultural heritage organizations by enabling storytelling across a range of platforms. The grants are for up to 12 months.

The funds will support the creation of projects that make significant and meaningful uses of the Library’s digital materials. Projects must also center on one or more of the following groups: Black, Indigenous, Hispanic or Latino, Asian American and Pacific Islander and/or other communities of color. Please note: Actual awards may be less.

If you are a LAM at a higher education institution, you should apply for a higher education grant.

�These LAM grants are intended for public libraries, non-profit libraries, archives and museums.

For more information on this opportunity, we encourage you to visit the OTP website at https://www.loc.gov/programs/of-the-people/represent/libraries-archives-museums-application/

47 of 54

Grant for Minority-Serving Higher Education Institutions

47

Application Deadline: September 7, 2023 by 2pm EDT

Up to $70,000 to support up to three higher education institutions by enabling storytelling across a range of platforms. The grants are for up to 12 months.

The funds will support the creation of projects that facilitate teaching & learning with the Library’s digital collections for students, staff and/or community members and make significant and meaningful use of those digital materials. Projects must also center on one or more of the following groups: Black, Indigenous, Hispanic or Latino, Asian American and Pacific Islander and/or other communities of color.

https://www.loc.gov/programs/of-the-people/represent/higher-education-application/

This year, we'll award up to 3 grant awards of up to $70,000 each to enable storytelling across a range of platforms. The grants are for up to 12 months. Please note actual awards may be less than $70,000.

The funds will support the creation of projects that facilitate teaching & learning with the Library’s digital collections for students, staff and/or community members and make significant and meaningful use of those digital materials. Projects must also center on one or more of the following groups: Black, Indigenous, Hispanic or Latino, Asian American and Pacific Islander and/or other communities of color.

For more information on the Higher Education opportunity, we encourage you to visit the OTP website at https://www.loc.gov/programs/of-the-people/represent/higher-education-application/

You can also view our Higher Education informational webinar.

48 of 54

Time for Discussion�

49 of 54

Thank you!

For More Information

Web: https://labs.loc.gov

Points of Contact

Laurie Allen, lallen@loc.gov

50 of 54

Examples

https://labs.loc.gov/work/citizen-dj

https://blogs.loc.gov/thesignal/2020/11/lc-for-robots-in-action-using-the-api-to-access-the-federal-theatre-project-collection/

Eileen

These uses vary in size and complexity. For the sake of time, pulled just two but included more in the full deck shared with all of you.

English professor Derek Miller essentially wanted to bulk download the records of the The Federal Theatre Project (1935–39), which was a New Deal program both to support theater artists and to bring art to the masses during the Great Depression. He worked with Betty Brown, reference librarian in the main reading room, who was able to assist him using LC for Robots resources.

Given how exciting these records are, I was eager to download them so that I could convert them to formats more amenable to my research, such as spreadsheets or more complex database structures that allow additional sorting and interpretation. However, I was having trouble downloading the documents because they were digitized as individual images of multi-page items. For instance, some weekly New York performance summaries contain 15 images in their sequence. Being able to download each document as a single file, such as a PDF, was the first hurdle in working more closely with the materials
The solution was as straightforward as one would expect from LOC’s posted examples. Each individual item can be accessed in a JSON format—a simple data structure—by adding a suffix. So the page //www.loc.gov/item/fpraf.09640014 can be viewed as JSON by going to this address: //www.loc.gov/item/fpraf.09640014?fo=json. Note that I’ve added “?fo=json” to the end of the URL. This tells the server to return data in the format (fo) JSON.
JSON information can look overwhelming and complicated; a detailed explanation can be found in the Library’s documentation of the API. But basically, we’re looking at a kind of dictionary, with a set of keys and values, separated by a colon. The images I need are keyed here as “resources” and, within those resources, as “files.” Each file comes in a number of different types and sizes. For my part, I wanted the TIFF image files, which are the largest offered size of the image.
By writing some Python code much like that in LOC’s examples, I was able to get the URL for each TIFF file and download it to a folder.

And then Citizen DJ—created by Brian Foo IIR and now technologist with the CCDI divison

API query to build initial list of potentially free-to-use audio and moving image collections

Performing queries to the LOC API to retrieve the audio files & metadata from 9 free-to-use LC collections

Notes/reference only:

https://www.loc.gov/collections/?fa=original-format:sound+recording -from Brian - “ I don't recall a download parameter... only a query for certain rights language, e.g. https://www.loc.gov/audio/?q=%22No+known+restrictions%22+OR+%22No+known+copyright+restriction%22+OR+%22Library+is+not+aware+of+any+copyrights%22.

Inventing Entertainment: The Early Motion Pictures and Sound Recordings of the Edison Companies

Variety Stage Sound Recordings and Motion Pictures
The Joe Smith Collection: man on the street style interviews
Free Music Archive: Free Music Archive is a website devoted to distribution and curation of rights-free music
MusicBox project: In January 2010 Dyann Arthur, independent folklorist, musicologist and Berklee College of Music alumni, founded MusicBox Project, a non-profit corporation in the state of Washington, whose mission it is to document and preserve our musical history while advancing music education and occupational avenues of creation and performance
Tony Schwartz Collection
Radio, Conversations, and Found sounds
American English Dialect Recordings
Interviews, Oral histories, and Speeches
The National Screening Room
Government film
The National Jukebox: Blues, Jazz, Folk, Opera, Musical theatre, classical, popular
Music