1 of 31

To Image or Not to Image

Tori Maches

UC San Diego

Implementing a staggered transition to logical capture by default

2 of 31

First attempt

Large, high-priority accession

1 TB hard drive
iBook G4 laptop
USB drive�

3 of 31

First attempt

Known issues

Substantial unallocated space

Not enough room in quarantine storage
Disk images unwieldy

Due to the large amount of unallocated space involved, and our limited quarantine storage capacity, we realized it wasn’t practical to store disk images of the three drives. In addition, large disk images can be unwieldy to work with, and the hard drive and flash drive were Mac-formatted and contained files and folders with forbidden characters in the names. Our processing workstation consists of two PCs, one running Windows and one running BitCurator, and a NAS that serves as quarantine storage. Due to our Windows-based storage infrastructure, I was unable to export these files from the disk image I created as a test.

Honestly, it just made more sense to create a logical copy; the drives were a straight copy-paste of user folders, so there wasn’t any original context or environment to worry about, they’d already been moved. Additionally, we didn’t plan to emulate the original environment of the laptop. The contents were of more interest, and preserving those was the priority.

4 of 31

How did it go?

It worked!
Process

FTK Imager file export, Linux terminal
Laptop as third drive

5 of 31

How did it go?

Takeaways

Easier appraisal, fewer bottlenecks
Saved storage space
Need different process for Mac-formatted

In the end, I retrieved all the files we were trying to get, and appraisal was much easier without an added “export from disk image” step before anyone else could look at the files, so my involvement represented less of a bottleneck. We also saved a great deal of space in our quarantine storage. We also learned that, if we persisted with the same methods, we’d need to use a slightly different approach for Mac-formatted media, which would only mount on our BitCurator workstation. It was encouraging, especially as we looked ahead to logical capture by default.

You see, while I was retrieving the files from this accession, the updated UC Guidelines for Efficient Archival Processing were released. The Guidelines included a lot of great recommendations for born-digital, including the recommendation to avoid disk imaging as your default approach to file capture. Instead, the guide stressed only disk imaging when absolutely necessary. This made me feel empowered to proceed; although disk imaging by default is best practice, it’s not necessarily practical or scalable, and there are plenty of valid reasons not to do it.

6 of 31

A staggered approach

Stage 1: One-off collection

Stage 2: Test collection

Stage 3: Copy by default for optical/USB media

From here, I started thinking about implementing logical copying by default on a wider scale, especially since this had just been recommended systemwide, and would represent a solution to the scale problems we were beginning to encounter with large born-digital collections. Imaging hundreds of disks, then exporting the files from disk images, wasn’t sustainable given the number of large collections SC&A has.

We’d tried logical copying with one SC&A collection, which helped us realize we could do it, and the potential benefits, which included limiting bottlenecks, making accessioning and processing more scalable, and making it easier for archivists and processors to appraise files without my help. The next step was to try it with another collection more typical of the materials in our backlog. This collection and our initial collection would serve as a learning experience that would allow us to transition to copying by default for optical and USB media.

7 of 31

A staggered approach

Stage 4: Test collection with older legacy media

Stage 5: Copy by default for older legacy media?

8 of 31

Test collection: February-March 2020

20 discs
Photos, AV, �research data
Windows�

9 of 31

Test collection: February-March 2020

Takeaways

Easier appraisal
Select another method for copying
Update/re-evaluate workflows

On the plus side, the discs’ contents were much easier to appraise; the processor who worked with the paper portion of that collection was able to access the files without any significant intervention from me. It also saved valuable drive space, since most discs had considerable unallocated space, and I was no longer storing copies of that.

Some drawbacks to this approach included an additional step of confirming which copy of the disc contents to keep, since FTK Imager displays multiple copies of the directory structure for different filesystems. In addition, without a disk image, I was unable to use tools that required disk images to run, which I’d also noticed with the initial collection. This test collection showed us where the existing processes needed work, and helped us to understand what needed to be done to improve them. We decided to move ahead with logical copy by default, with some fine-tuning to be done with a couple more test collections.

10 of 31

… and then the pandemic happened

Remote work
Limited access to media

Drive space for �unprocessed files

Updating workflows

So that was where we were in March. In the midst of all this, the COVID-19 pandemic happened, and all on-site born-digital processing stopped. Before I left, I copied the files from the test collection from quarantine storage to our dark storage for remote access, and this ended up being the first born-digital material to be processed remotely.

We did get our server space for unprocessed materials, and I moved files from eight collections from quarantine storage to this space so we could work on them from home. However, it became clear that I wouldn’t be onsite on a regular basis until 2021; as of the fall, no one in SC&A is scheduled to work onsite full-time. This has put the majority of work involving legacy media on hold, and has also meant spending a lot of time updating our workflows to prepare for when I’ll be able to work with legacy media on a regular basis.

11 of 31

Where are we now?

Logical copy by default

Tentative tools/workflows
Test collections

Older legacy media?
What can we learn?

12 of 31

Rumors of the disk image’s demise may �(or may not) be greatly exaggerated: �

Assessing preservation formats for disk contents in archival workflows

Laura Alagna, Alex Chassanoff, �Dianne Dietrich, Brian Dietz, farrell, �Alex Nelson, Shira Peltzman, Paige Walker

13 of 31

Disclaimer

The views and opinions expressed in this presentation are those of the authors and do not necessarily reflect the official policy or position of any agency of the U.S. government. Any mention of a vendor or product is not an endorsement or recommendation. Logos and trademarks are copyright of their respective owners.

14 of 31

Background

….introducing DANNNG!

Digital Archival traNsfer / iNgest / packagiNg Group�
Current membership: Laura Alagna (Northwestern), Alex Chassanoff (NCCU), Dianne Dietrich (Cornell), Brian Dietz (NCSU), farrell (Duke), Alex Nelson (NIST), Shira Peltzman (UCLA), Paige Walker (Boston College)

https://dannng.github.io

Last year’s BitCurator User Forum raised many questions around the operational and environmental sustainability of disk imaging and whether or not there may be viable alternatives to this workflow, for instance by creating logical rather than physical images that would save only the logical file content selected rather than make a bit-for-bit copy of a media object.

At the very end of the Forum, someone mentioned a recent paper on the Advanced Forensics File Format, AFF4-L, and wondered whether or not it may be a good option for the digital archives community since it would preserve logical files without preserving other possibly unwanted content on a media object.

Several of us started meeting, somewhat informally, with the original goal to investigate AFF4-L as an alternative to capturing physical disk images. The conversation evolved over time, we invited more people that we personally knew to join the discussion.

15 of 31

Background

….introducing DANNNG!

Digital Archival traNsfer / iNgest / packagiNg Group

traNsfer - …
iNgest - ...
packagiNg - ...

https://dannng.github.io

16 of 31

Mission

Began as investigation into AFF4, but morphed into something broader:

Demystifying disk images and alternatives
Highlighting common LIS challenges in digital archiving
Establishing shared vocabulary

So, we created some draft documentation...

17 of 31

Format comparisons

Started inspecting qualities of AFF4, especially:

AFF4: disk data�(physical disk image)�Compares to .dd, .E01
AFF4-L: semantic storage metadata�(logical image/acquisition)�Compares to .zip, .dfxml�

?

18 of 31

Format comparisons

This became inspection of various disk image formats.
Then became comparison of disk imaging and logical file acquisition.
Then became …

… when do we do either?
… how do we talk about it?

19 of 31

Glossary

Overview of common terms in use for born-digital archives

Media object
Volume
Operating system
File system
Disk image format
Container

https://bit.ly/dannng-glossary

20 of 31

Glossary

Media object: A physical piece of computer-readable hardware that contains some number of digital files. Examples include: a flash drive, a floppy disk, a CD.

Volume: A storage area defined at the operating system level, which has a single file system and usually resides on one disk partition.

https://bit.ly/dannng-glossary

21 of 31

Use Cases: Welcome to the Quadrants

https://bit.ly/dannng-use-cases

- we discussed a number of ways to approach describing or outlining use cases for disk images,

- Initially, we considered extending the AVP/Harvard work on Disk Image Content Model and Metadata Analysis, particularly that project's Matrix comparing disk image formats.

- But even still, we wanted to get decisions earlier in the process--when does it make sense to create a disk image? When does it make sense to retain a disk image as the object of preservation? When does it make sense to export logical files only?

- after a few false starts, determined that, yes, while we could likely outline media types and their suitability for disk imaging versus file export, there's often a level of nuance that is hard to capture.

- we definitely wanted to steer away from a prescriptive "DO THIS IN EVERY CASE"-style approach

- partially, because, what authority do we have, really?

- but also because, at least in my experience, as soon as I create a RULE, the next test of that RULE identifies an EXCEPTION

- Dianne remembered the Four Factor test for determining Fair Use in copyright, wherein one must approach a particular use through four different lenses.

- this felt like a useful exercise to us:

- what factors might exist that would impact whether, for a given piece of storage media, one would create a disk image, skip imaging in favor of exporting contents, or some mix of the two (e.g., temporarily store a disk image during triage but ultimately only retaining the contents)

- Like the four-factor fair use test, whether or not to create a disk image felt fluid in many cases and so we set to work

- though, for fun, we decided to call our factors 'quadrants'

22 of 31

Use Cases

Quadrant 1

Quadrant 1: Is the item old?

https://bit.ly/dannng-use-cases

23 of 31

Use Cases

Quadrants 2-3

Quadrant 2: Is interactivity a desired function for researchers?
Quadrant 3: Is capturing provenance of the object's file system(s) important?

https://bit.ly/dannng-use-cases

2. Is Interactivity a desired function for researchers or end users?

- if you're planning (or hoping) to provide emulated environments to users at some point, chances are you'll need a disk image to mount inside of that environment

- even without an emulated environment, if interactivity with the file system is important to the material, a disk image is often the most efficient way to preserve that for future end-user interaction

3. Is capturing provenance of the object's file system(s) important?

- while other tools that perform file system analysis exist, capturing a copy of the file system as it appeared on the original media may be important.

- if you are aware that the creator of the material used a piece of media over some period of time, and it is important to be able to convey or reconstruct how files were manipulated on it, capturing a disk image is a good way to fix the file system for reproducibility later

24 of 31

Use Cases

Quadrants 4-5

Quadrant 4: Is the item a type of media that cannot (or perhaps should not) be imaged?
Quadrant 5: Have you or your donor considered data loss and retention?

https://bit.ly/dannng-use-cases

As we were discussing these concepts, we kept identifying reasons that may argue against creating a disk image that might trump the arguments for

- this led us to coalescing around an additional three concepts that would capture these anti-imaging arguments

- we kept the language of 'quadrants' even though that doesn't make numerical or grammatical sense

- Quadrant 4: Is this a thing that can't or should shouldn't be imaged?

- Audio CDs don't have a file system!

- working with optical media written to over multiple sessions present issues to many imaging tools and workflows

- some media have copy protection on them

- media that was used strictly as a transfer device are less likely to have file system metadata contemporaneous to the creation of the files contained on it

- Quadrant 5: Have you and/or your donor considered data loss and retention?

- maybe you've assessed the risk of data (or metadata) loss, and are comfortable with possible outcomes or if data loss is intended as a feature rather than a bug

- e.g. if a donor is more concerned with not preserving deleted file fragments than with the possibility that you won't accurately portray the MACtimes of the preserveable files

25 of 31

Use Cases

The Final Quadrants

Quadrant 6: Do you face institutional limitations?
Quadrant 7: What does your workflow require?

https://bit.ly/dannng-use-cases

26 of 31

Give us your feedback

What resonates with you?
What isn't relevant?
What have we missed?
Is our framing helpful?
Do you have a different take?

Please let us know via Google Doc comments!

27 of 31

Join Us!

We also think a group that discusses issues related to disk imaging would be valuable to continue, and we can think of several additional projects we'd like to tackle after we complete the documents we started earlier. If this work interests you, please join these efforts. We recognize that any output this group produces will benefit greatly from a wider array of backgrounds and experiences.

dannng@googlegroups.com

28 of 31

What’s next

Revising current draft documents

Disk image + alternatives matrix
But most importantly: recruitment

29 of 31

Thank you!

Helpful links:

Join us!

dannng@googlegroups.com

30 of 31

Image credits

Icons by various designers for the Noun Project:

disk by Kmg Design
Picture by José Campos
Hard Drive [iPod] by Manthana Chaiwong
Hard Drive by Graphik Designz
Thumb Drive by Emmanuel Reyes
documents by GreenHill
config file by Hare Krishna
Chaos by Anna Sophie
Workflow by Kamal

Welcome to the Quadrants

https://www.pxfuel.com/en/free-photo-xwbsl

31 of 31

The Shoulders of Giants