1 of 22

Case Study: Oregon Digital Migration from CONTENTdm to Hydra �for digital collections

Hydra Connect 2016, Boston Public Library �Julia Simic & Linda Sato, University of Oregon Libraries

Ryan Wick & Margaret Mellinger, Oregon State University Libraries

2 of 22

Welcome & Introductions

Oregon State University and University of Oregon Libraries have had a longstanding and successful partnership to provide access to unique digitized and born-digital materials through a shared instance of CONTENTdm. A few years ago, we started to have some problems with CONTENTdm and were experiencing some limitations with the software. We wanted to provide users with a better media-viewing experience, as well as to have more flexibility around metadata, RDF and linked data. With an OSU and UO collaboration in mind, Karen Estlund and Evviva Weinraub began to explore options for open source platforms with programmers Tom Johnson, Jeremy Echols and Ryan Wick. Hydra was chosen as the platform, an MOU was written, a shared governance board was created, and the work began in 2012.

3 of 22

From CONTENTdm to Hydra

4 of 22

Hydra Prototype

All descriptive metadata fields were in RDF, everything was a string
Collection (set) landing pages and additional information pages
Items could belong to multiple Collections (sets)
Zoomable image viewer using IIP server and OpenSeadragon UI
PDF/Document viewer using Internet Archive BookReader
At the time, most in the community were not building repositories with many different item types or linked data.

5 of 22

After the Prototype

Decided to use deep RDF (URIs) where possible
Needed to write RDF code that didn’t exist elsewhere, such as fetch and label handling, some became ActiveTriples gem
Set up a separate server for derivatives, ingest, fetch and index jobs
Needed to have CONTENTdm item short URLs and viewer URLs redirected

CONTENTdm URL/ID stored with each new item, Rake task can export all of them and generate mapping files
Had some issues with map size until we used nginx

Audio player - HTML5

6 of 22

Migration Workflows

7 of 22

Examine CDM desc.all files

Regularize predicates/ field names

Find or publish LD predicates

Create YAML mapping files

Examine object data

Object normalization

Find or publish LD object data

Create equivalents lists

cdm2bag ingest

230,000 assets in 68 collections

8 of 22

Examine CDM desc.all files

Regularize predicates/ field names

Find or publish LD predicates

Exported CONTENTdm metadata by collection
Examined field names and content to determine needed predicates
Regularized and mapped field names to existing LD predicates
Published new predicates in opaquenamespace.org

Originally JSON-LD files on GitHub
Now published using the Controlled Vocabulary Manager

Deleted fields that were used only internally or were specific to CDM
Determined if predicates should require string or URI objects within Oregon Digital
Created mapping files (YAML)
Documented in Metadata Dictionary

Create YAML mapping files

The first task in the migration was to export and examine the existing desc.all files from each collection in order to identify fields that would be replaced by predicates. The examination revealed many common fields, mostly Dublin Core fields or their equivalents, but many that could use other schema including MARC relators, the RDA Registry, Darwin Core, Schema.org, BIBFRAME, ArchivesHub, and VRA Core among others. It also revealed many local fields. By examining field contents, we were able to normalize these variant fields into single predicates. In the end we came up with a list of 155 common predicates from a dozen schema that could be used across all our collections.

Local fields where no predicate existed but that were determined to contain sufficiently important information for preservation were published in OpaqueNamespace.org, a linked data service powered by the Controlled Vocabulary Manager, a Ruby on Rails application connected to Blazegraph. Those fields where we could not tell the intent, because of the ambiguous field content and where the original cataloger was no longer with the institution, or fields that contained data specifically for use in CDM were deleted.

Next, we determined for each predicate whether the Object values could contain strings or would require URIs for validation. Finally, we created mappings for each collection. All this is documented in the Oregon Digital Metadata Dictionary.

9 of 22

Examine object data

Object normalization

Find or publish LD object data

Create equivalents lists

Examined all Object data per collection
Fixed typos, delimiter errors etc. using scripts and by hand

Non-UTF-8 diacritic problems

Looked for usable LD vocabularies

Used a combination of scraping and hand gathering

Defined predicates to validate only chosen vocabs
Published new vocabularies in opaquenamespace.org (JSON-LD→CVM)
Created lists of old entries and new URI values

10 of 22

cdm2bag ingest

Cdm2bag Ruby script processed desc.all files, mapped fields and output bags
Replace object data using a combination of mapping methods and lists

BagIt utilized to output one asset + metadata per bag

Bad mapping, predicates or invalid objects stopped ingest

Problems with deep RDF predicates
Problems with unresolvable object URIs

Some high resolution files were missing and/or never existed
CDM full resolution links corrupted when assets were moved on servers and links were not updated
Processing derivatives was very slow, large PDFs and TIFs

Considered pre-processing derivatives

Migration was accomplished using a Ruby script, Cdm2Bag, that processed the desc.all files, mapped the fields using our YAML files and output Bags. The object data was replaced using a combination of mapping methods and the lists that we produced. Mapping errors or invalid Object values could stop the ingest cold. This caused significant time loss.

Also, some archival files were gone or corrupted and we had to either reproduce them or replace them with the low quality derivatives from CDM. If the archival files had been moved for preservation and the CDM links had not been updated, we were forced to search our servers to find them and were mostly successful. In addition, not all collections had Tiffs in CDM, and not all source files were Tiffs to begin with.

One final issue that caused time loss was the processing of derivatives which was agonizingly slow for large PDFs and Tiffs. So slow in fact that we considered pre-processing derivatives and ingesting them along with the archival files. In the end it was not worth the development time.

It took us about 2 years to migrate everything into Oregon Digital. And the fun didn’t stop there...

11 of 22

Post-Migration Cleanup

12 of 22

Post Migration Cleanup

13 of 22

Post Migration Cleanup

14 of 22

Bad Assets

One problem that we didn’t necessarily see coming was the number of duplicate assets. Jeremy Echols, senior programmer at the UO LIbraries, was the person who discovered and worked on the problem on the UO side; he recalls the number being in the thousands. Most of them occurred during the early stages when we went through cycles of ingesting and deleting batches repeatedly in an effort to fix metadata problems. Many of the duplicates were in review, and these were simple to identify and delete. Another smaller number of duplicates existed primarily because the asset had failed during ingest; since we could see that the process had errored, we would ingest again, not realizing that a corrupt asset was already in the system. The remaining duplicates, roughly two hundred or so, were fixed by hand.

15 of 22

Final Review: Image filename problems

SQKH_00002_KV

16 of 22

Final Review: Image problems

17 of 22

Mountain West Digital Library

Correctly formatted dates

2015

2015-02

2015-02-26

any of the above as a range: 2015/2016

or as a series: [1801, 1926]

What we found:

ca. 1915

1877, c. 1885

c. 1900-1909

1913?

Decmeber 29, 1955

Summer 1957

1948, dismantled 1983

27-Aug-80

February 1975

1911-11 - 1912-08

1937 and later

1883, hospital opened

Kenneth Gunn

1925, 1928, completed 1948

18 of 22

Lessons Learned

Metadata cleanup and review require significant resources, have metadata people closely involved with developers from the beginning
Leadership needs to have all impacted staff onboard
Ensure that original media files are named correctly and accessible
Linked data encourages better quality but can be time consuming with poor metadata, tools have improved over years

19 of 22

Lessons Learned continued

Being involved in the Hydra community is very important, share work and solve problems together
Devote development resources to migration tooling and QA
Compound objects are complicated

20 of 22

Resources

Metadata Dictionary: https://docs.google.com/spreadsheets/d/1nnJz49oqstQLF2PFBn5ZvWfWdF-8PVl5TFL1EMotQrA/edit?usp=sharing

Oregon Digital Descriptive Metadata schema: https://github.com/OregonDigital/oregondigital/blob/master/app/models/datastream/oregon_rdf.rb

Cdm2bag: https://github.com/OregonDigital/cdm2bag

Collection field mapping: https://github.com/OregonDigital/cdm2bag/blob/master/mapping.yml

Csv2bag: https://github.com/OregonDigital/csv2bag

OpaqueNamespace.org, powered by https://github.com/OregonDigital/ControlledVocabularyManager/

21 of 22

Oregon Digital Hydra Team, Past & Present

Oregon State University

Evviva Weinraub, Trey Pendragon,�Tom Johnson, Ryan Wick, Mike Eaton, �Brandon Straley, Greg Luis Ramírez,�Josh Gum, Ryan Ordway, Maura Valentino, �Brian Davis, Erin Clark, Michael Boock, Margaret Mellinger, Chris Petersen, �Trevor Sandgathe, Hui Zhang, Susan McEvoy, Helena Bales

University of Oregon

Karen Estlund, Sheila Rabun, David McCallum, Julia Simic, Jeremy Echols, Duncan Barth, Sarah Seymore, Linda Sato, Kate Jones

22 of 22

Questions?��Title�Mika-in-a-bag�LC Subject�American shorthair cat�Photographer�Rabun, Sheila�Condition Of Source�adorable�Identifier�mika6�Type�Image�Format�image/jpeg�Has Version�Sheila's cat �Rights�Rights Reserved - Restricted Access��

oregondigital:df715547z