1 of 13

Supporting Software Preservation @ NYPL

LIS 698.02 Practicum Seminar

Anne Boissonnault

2 of 13

Digital Archives & Preservation

3 of 13

Billy Rose Theatre Division, The New York Public Library. (1980). Best Little Whorehouse in Texas, 1980 May 22; undated. Retrieved from: http://digitalcollections.nypl.org/items/a28d6e6b-b1b6-f008-e040-e00a1806635d

When I think about digital archiving—especially in the context of the performing arts—I tend to think about audiovisual recordings, images, and ephemera, and the challenges surrounding their associated file formats (MOV, WAV, TIFF, etc.). These formats have specific needs in terms of preservation and access, but because they are commonly associated with archival collections, the digital preservation community has established a set of processes and best practices for caring for them.

IMAGE: A JPG file depicting a theater performance. The lighting for this performance was designed by Jules Fisher and Peggy Eisenhauer. The Jules Fisher and Peggy Eisenhauer papers and designs is one of the collections at LPA that contains the proprietary file formats which were the focus of my semester’s work.

4 of 13

Proprietary File Formats

5 of 13

Lightwright

6 of 13

Lightwright @ NYPL

Tharon Musser designs and papers
Jules Fisher and Peggy Eisenhauer papers and designs
Merce Cunningham Foundation records

Jerome Robbins Dance Division, The New York Public Library. (1956). Merce Cunningham in "Lavish Escapade." Retrieved from: http://digitalcollections.nypl.org/items/9c4bca50-08a0-0133-2120-58d385a7bbd0

7 of 13

The first step in my work was to map out the current preservation environment for the Lightwright software, and identify any gaps present in resources developed by the digital preservation community. In particular, I focused on registries that aggregate technical information about software products, their support lifecycles and requirements, and their associated file formats. My research led me to examine three registries:

PRONOM
Archivematica’s Format Policy Registry (FPR)
Wikidata / Wikipedia

What I found was that, aside from the website maintained by the developer of Lightwright, there was very little information available on the software and its file formats. There were no entries in either PRONOM or the Archivematica FPR, and the information recorded in Wikidata and Wikipedia was incomplete.

The next step in my process was to fill these gaps through further research and contribution to the aforementioned registries. Beginning with the website and user manuals, I established a list of the software versions, their release dates, programming languages, system requirements, and associated file formats.

8 of 13

Developing a File Format Signature

Lightwright 6 Show File	Signatures
	External signatures	File extension : lw5
	Internal signatures	Name	Lightwright 5 : 2009
		Description	BOF first 8 bytes : JMCK
		Byte sequences	Position type	Absolute from BOF
			Offset	0
			Max Offset	0
			Value	4a4d434b
		Description	EOF 30 bytes : CLEAR TEXT BLOCK LWFORMAT V5
		Byte sequence	Position type	Absolute EOF
			Offset	0
			Max Offset	2048
			Value	434c454152205445585420424c4f434b0d0a4c57464f524d4154205635

The initial preservation step I took was focused on the identification of file formats by developing file format signatures for submission to the PRONOM registry.

NYPL uses a file profiling tool called DROID while processing digital files, and this tool references the PRONOM registry. Adding Lightwright signatures to PRONOM will help archivists at NYPL to identify those formats in the future. Ultimately, I developed six separate signatures, one for the Show File format associated with each of the six Lightwright software versions.

There are two types of file format signatures, external signatures and internal internal signatures. External signatures are filename extensions (such as DOC or PDF) that can provide clues to the file format. Internal signatures are essentially byte sequence(s) common to a particular file format which can be used to recognise digital objects encoded in that format. It was this second type that I was concerned with was developing.

�Ultimately, I developed six separate signatures, one for the Show File format associated with each of the six Lightwright software versions. To do this, I used sample files from the collections at NYPL; a hex editor to identify patterns in the hex-code of these files; PRONOM’s signature development utility to record the signature in a machine readable format; and DROID to test my signatures before submitting my work for review by the National Archives. �

9 of 13

The second preservation step I took was to update NYPL’s local Archivematica Format Policy Registry (FPR). NYPL uses Archivematica to help prepare digital files for preservation and eventual access. The FPR is a database which allows users to define format policies which Archivematica then queries when performing identification, characterization, and normalization steps on file formats. Artefactual hosts an FPR server that stores structured information about format policies for Archivematica, but users can also maintain local rules, add new formats, or customize the behaviour of Archivematica in a local FPR.��This proved to be useful for the files associated with the Lightwright software, as there were no entries for these formats in the FPR maintained by Artefactual. I again focused my efforts around the Show File format. Adding the Lightwright formats to NYPL’s local FPR was relatively simple, being that my format signatures had not yet been reviewed by PRONOM and the only available command was for Archivematica to identify the file formats by extension.

10 of 13

Linked Data for Software Preservation

In addition to my work on the Lightwright file formats, my research produced significant information pertinent to the long-term preservation of the software itself. I wanted to make this information available to the software preservation community at large—Wikidata provided me one method by which to do this.��Wikidata is a knowledge base of structured data which, similar to Wikipedia, can be edited by anyone with access to the internet. A project at Yale University Libraries is spearheading an effort to employ Wikidata toward the capture and collation of metadata to describe file formats, software, operating systems, and hardware, and inform digital preservation work. Although the creation of records in Wikidata for this purpose is, in large part, carried out by experts, contribution by the broader community is encouraged.��Initially, Wikidata served to align the language versions of a given Wikipedia page; as such, most Wikipedia articles have associated Wikidata entries. At the time I began my project, there was already a Wikipedia article for Lightwright, so the software was represented in Wikidata, but the information recorded in both was incomplete.

11 of 13

12 of 13

Looking Forward

Jerome Robbins Dance Division, The New York Public Library. (1956). Merce Cunningham in "Lavish Escapade." Retrieved from: http://digitalcollections.nypl.org/items/3b874850-08a0-0133-bb47-58d385a7bbd0

In completing this project, I had managed to make significant progress toward the long-term preservation of the Lightwright software. Upon review of my format signatures by PRONOM, DROID and Archivematica at NYPL will be able to identify the Lightwright Show File formats via their PRONOM format signatures; until that time, NYPL’s local Archivematica FPR will identify those formats via their extensions. ��Additionally, crucial information to the preservation of the Lightwright software is now available to the software preservation community via Wikidata. ��There were a number of file formats associated with the Lightwright software that, because of time constraints, I was unable to develop format signatures for. These formats were not added to NYPL’s local FPR, nor were they represented in Wikidata. Moving forward, I would recommend that signatures be developed for these file formats, and submitted for review by PRONOM. Additionally, they should be added to NYPL’s local FPR, and entries should be made in Wikidata to represent them.

Finally, my hope is that this work might serve as a model for those with limited technical experience (like me!) as to tangible steps that can be taken toward the preservervation and continued access to unique and unidentified file-formats and proprietary software that appear in archival collections.

13 of 13

Thank You!

Questions?