1 of 42

Webinar 4: Digitization

in-house vs out-of-house

Southwest Region IMLS + NEH Cohorts

2 of 42

Digitization Planning� May 26, 2021 �

Presenters: Kelli Hix & Marie Lascu��

#cawesome

In this session, participants learn about planning for digitization, with an emphasis on the differences between in and out of house digitization.

3 of 42

What has been covered so far:

Identifying, inspecting, and assessing film & magnetic media
Inventorying film & magnetic media
Metadata for creating inventory templates

4 of 42

This webinar will cover:

Planning for digitization

Determining whether to pursue

in-house digitization
out-of-house digitization with a vendor
a combination of both

Utilizing your inventory for

preservation planning
tracking preservation work

5 of 42

What is Digitization?

Analog signal (e.g. light, sound) → Digitally encoded format.

6 of 42

What is Digitization?

Analog waveform is approximated digitally.

7 of 42

Why Digitize?

A step toward preservation:

Many analog formats are obsolete, and need to be reformatted to non-obsolete formats to be usable.
Many analog formats are unstable and degrade over time.
Digital content can be copied exactly over and over again without “generation loss.”

For easier access:

It is easier to make copies of digitized content.
Digitized content can be shared electronically / online.

WHY digitize? Kelli and Moriah referenced these reasons in the first two webinars, specifically when covering issues like format fragility and degradation, the obsolescence of playback equipment.

At this moment in time, the technology does not exist to maintain physical media carriers as is - to ensure that they remain playable. Even if manufacturers continued creating new decks, the tapes themselves were not manufactured to last indefinitely. Analog film fairs better because of its physical structure and more standardized playback equipment, and it does really well in proper storage.

Digital is the dominant form in our modern dominating world - once items are digitized and/or carried over to a digital work state - it is much easier to make other necessary copies, transfer items to newer storage when necessary, and provide more access (if access is appropriate).

Digital does not, however, guarantee stability. And human maintenance is still very much required to ensure preservation.

8 of 42

Digitization vs.

Digital Preservation

Digitization:

Reformatting an analog object into a digital representation.
Migrating a media carrier-based digital signal to a digital file form.

Digital Preservation:

Digitization is ONE step in the preservation process.
Multiple steps required to ensure preservation.
Digital files cannot survive through passive preservation.
Preservation is long-term.

9 of 42

Plan Before You Digitize!

Avoid common pitfalls:

Digitizing to a file format that you cannot play or open

Digitizing with a vendor that does not meet standards

Digitizing at low quality

Digitizing to optical media (not a preservation format)

Plan before you digitize! That is the key message of this presentation. Planning helps to avoid common pitfalls, and it helps to avoid re-digitizing in the future. Common pitfalls include:

Digitizing to a file format that you cannot play or open
Digitizing with a vendor or set-up that does not perform to good /archival standards. Vendors like Walmart, Costco, or local home movie transfer companies, or an in-house set-up like shooting footage with a camera off of a screen offer low cost way to digitize, and might create a decent reference file for you. But these vendors and methods will not give you the quality that you are likely to want down the road.
Digitizing at low quality is another one of the common pitfalls of digitization projects. Why is digitizing at high quality important? A low quality image might tell you what is on the film or tape, but the magic happens when you have higher quality. This is when we can read writing such as street signs, license plates, books, and newspapers. We can see facial expressions and gestures that are at the heart of why you may want the footage. Remember that with some of this material you may only have one chance to digitize it if it is fragile. So digitizing at the highest quality that you can is important.
Another common pitfall is digitizing to optical media-- CDs and DVDs. Optical media is not a preservation format. It is fragile, it scratches easily, and often compresses the files, causing low quality.

10 of 42

Planning Foundations:

Create an Inventory

11 of 42

Planning Foundations: Develop Preservation Priorities

Which of our collections have the most important content to our organization?

What do we have that is unique to our collection?

What is the most unstable / endangered part of the collection?

12 of 42

Planning Foundations: Determine Digitization Strategy

In- House

Out of House

Combination

13 of 42

In house?

Or

Out of house?

Financial Considerations

Collection Considerations

Human / Quality of Life Considerations

14 of 42

1

Financial

Considerations

What method is more cost effective?

Labor costs
Equipment costs
Vendor costs

We’ll start with financial considerations. Which is more cost effective -- digitizing in house or out of house? In order to determine this, you will need to estimate the cost of labor, the cost of equipment, and the costs to a vendor and compare the estimates. This is a rough number meant to guide your decision, so try not to get too caught in the weeds. It’s important to start somewhere.

For example, If you have a large collection of a common media format in good condition, it may be more cost effective to digitize in-house, even after purchasing equipment and training staff. If you have few items in a rare or older format in poor condition, it will probably cost less to send it to a vendor who already is equipped and trained to handle the material. For example, if you have 10 moldy cylinder recordings, it’s probably cost-effective to send them to a vendor. If you have 5,000 VHS tapes in fair condition, it can be more cost-effective to digitize them in-house. If you have both the VHS and the cylinder collections, you may want to choose a combination approach.

15 of 42

1

Financial

Considerations

Factors to include in cost estimate:

In-house:

3 hours labor / 1 hour of content
Training cost
Equipment cost

Out of House:

Vendor estimate
2-3 hours of admin and quality control labor per hour of content
Equipment / computers for performing quality control

Here is a handy rule of thumb for creating cost estimates. You may want to build off of this. For in-house estimates, most of your cost is in labor and equipment. To estimate labor, a good rule of thumb is the 3:1 rule: three hours of labor for every hour of content. This allows time to digitize -- tapes must be digitized in real time -- plus time to perform troubleshooting and follow-up tasks such as quality control. Also factor in the cost for a period of training for your project. Finally, add your equipment costs. We aren’t going into detail on equipment today, but you will need playback decks, and analog to digital converter, a computer, and some other items to control the video and audio signals.

When estimating the cost to digitize using a vendor, it’s important to remember that the estimate your vendor gives you is only the tip of the iceberg. You also need to factor in about 2-3 hours of labor to prepare and ship assets, administer the project, and perform quality control to check the vendor’s work.

16 of 42

2

Collection

Care Considerations

Advantages of Digitizing In-House:

No need to ship assets
You have control over quality
Sensitive materials / materials with access restrictions stay with your organization.
Improved knowledge about collection content for your organization

Advantages of Digitizing with a Vendor:

Specialized training and experience
Specialized equipment
May be able to handle mold, damage, and higher levels of decay, etc.
The vendor has built-in technical support

17 of 42

3

Human /

Quality of Life/

Community considerations

What is best for your staff and community?

In-house benefits:

Skill building
Job creation
Quality of life
Building Community
Building knowledge about the collection

Out of house benefits:

No need to purchase software, hardware, equipment
Potential for less technical skills hand-on

* The more technical knowledge you have, the better results *

The final group of considerations are less tangible. I call them human or quality of life considerations. These can also be described as how well does the approach meet your mission statement. Here are some advantages of both methods in this category.

Digitizing in-house can provide learning opportunities for staff; provides local jobs; can improve quality of life for those wishing to take on the task. I want to make the point here that digitization can be challenging, but it also can be learned, especially if there is a network and a community. This is also a great opportunity to share local resources, like equipment, skills, and experience, and to pass that on to others in the community.

Out of house projects can be beneficial if you do not want to purchase software and hardware, learn the full process, and do not want to or cannot perform digitization on-site. I do want to emphasis that you will need to learn technical skills no matter which method you choose, because it is vital to the success of your project that you can communicate what you want with your vendor, understand your options, and check on the quality of the service.

18 of 42

So you decided to digitize in-house

Consider the space requirements

Soundproofing
Internet access
Power
Permanent set-up

Have backup equipment

Have an extra computer for file processing

Have a dedicated staff member

Have a network or advisor that can help with troubleshooting.

Consider sharing a digitization station or equipment - use existing community resources

19 of 42

So you decided to digitize out of house

Create a vendor agreement

Get an estimate

Plan staff roles

Test the workflow with a pilot project

Do not have assets shipped back until you have approved the files.

And so much more...

20 of 42

Digital Files

Project Planning includes:

Target formats for digitization

File naming conventions

File types and version control

Quality control considerations

Use your inventory for:

Estimating digital storage

Tracking digital preservation information

21 of 42

Target File Formats

Selection considerations

Questions to ask before choosing target file formats:

Can you store large digital files?

Can you back up digital media?

What are your long term goals for saving this media – both physical and digital?

What type of access do you want for your media?

How do you want to use your digital media?

Questions to ask before finalizing your target preservation file format -

I am paraphrasing here about Evaluation Criteria - sourced from a document called Recommendations on Preservation Files for Use in the Digitization of Analog Audio and Video Recordings and Motion Picture Films created by the National, Provincial and Territorial Archivists Conference Audiovisual Preservation Working Group in January 2018:

https://www.bac-lac.gc.ca/eng/about-us/publications/Documents/preservation-file-formats.pdf

An organization should adopt formats that can be sustained. Most organizations will only have a single opportunity to digitize analog materials and should try to future-proof their work by making informed decisions around file format choices.

Uncompressed preservation files will capture the maximum number of attributes of the original - but the declaration that lossless uncompressed preservation files are the ONLY option to be considered ignores the realities facing most organizations tasked with preserving a/v collections. The resources you have available to you may render the ideal preservation level file unsustainable. There has to be a balance struck between quality and sustainability.

22 of 42

Target File Formats

The ideal: Preservation-level file

a high quality digital transfer that would capture as much information as possible from the original source.

This is not the file used for any editing, screening, or other access-related projects.

In addition to the preservation file, you will want to create access copies - compressed or smaller formats - generated using your preservation file.

Selection considerations

23 of 42

Target File Formats: Film

35mm

File Format (image):

Digital Picture Exchange (DPX)

Codec: Uncompressed

Resolution: 4k

(4096 pixels)

Bit depth / Sample rate:

10 bit RGB

File Format (audio): BWAV

Codec: PCM

Bit depth / Sample rate: 24-bit, 48kHz

16mm, 8mm, Super 8

File Format (image): Digital Picture Exchange (DPX)

Codec: Uncompressed

Resolution: 2k

(2048 pixels)

Bit depth / Sample rate:

10 bit RGB

File Format (audio): BWAV

Codec: PCM

Bit depth / Sample rate: 24-bit, 48kHz

We are not covering the complexities of digital file formats, but suggested formats are depicted here and in the following examples with broken out information that will typically need to be selected in most digitization capture software, or will need to be discussed when working with a vendor.

DPX - is an industry standard, most common for digital intermediate post production work and also used for high quality digital projection. Motion picture film digitized for preservation is not scanned directly into a single file container, but actually captured as separate image files for every individual frame in a reel of film. This means that the audio soundtrack is captured separately. And they are stored separately for long term preservation.

Broadcast Wave - which is the best practice industry standard for audio, is recommended. It is my understanding that Broadcast Wave is an extension of the more well known WAV format, and it primarily includes additional embedded metadata.

4K and 2K refers to the Scan Resolution. This is specific to film. Scanning film is different from the conversion process of analog video to a file format. Film scanning is optical - light is transmitted through the film and focused by a lens onto a sensor.

The K refers to the number of pixels contained in one complete horizontal line of a digital image. 4K is 4,096 pixels per scan line, 2K is 2,048 pixels per scan line. 4K is a great deal of information, but if you can believe it, scanning for preservation has increased to 6K and in some instances even 8K. There aren't even projectors or home devices that can handle 6K or 8K, and it's only in the last couple years that 4K has become projectable. Just to give you a sense of the amount of information you'd be dealing with.

24 of 42

Target File Formats: Video

File Format:

QuickTime (MOV)

Codec (Image): Uncompressed 4:2:2

Bit depth / Sample rate:

(recommended) 10-bit

(minimum) 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

(recommended) 24-bit, 48kHz

(minimum) 16-bit, 48kHz

½” open reel, U-Matic, Betacam, Hi-8, VHS...

File Format:

Matroska (MKV)

Codec (Image): FFV1 version 3

Bit depth / Sample rate:

(recommended) 10-bit

(minimum) 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

(recommended) 24-bit, 48kHz

(minimum) 16-bit, 48kHz

...DV, MiniDV, DVCam...

10-bit Uncompressed - This QuickTime file format is well documented and is widely adopted in most institutions, video post production, and even many consumer level devices. 10-bit YUV Uncompressed is still currently considered the preferred "preservation level" video format.

It was not until the late 2000s to early 2010s that technology and professional standards allowed for preservation level capture to a digital file format. This is due to a convergence of lower cost but powerful enough computers to process larger files, and the ability to store larger amounts of data for longer periods of time. This is still of course a large file when digitizing content that is 1 hour or more.

Matroska/FFV1 - Matroska is an open-source file format with specifications that are freely available.s FFV1 has been considered a viable archival codec, and the Library of Congress began regarding it as a suitable preservation option in 2014. FFV1 is not widely adopted and the need to perform an additional encoding step after digitization in order to use the file on common video players or with commercial editing software is often a deterrent to newcomers, but so far it is a proven good quality space saver in comparison to 10-bit Uncompressed.

25 of 42

Target File Formats: Video

File Format:

QuickTime (MOV)

Codec (Image): ProRes 422

Bit depth / Sample rate:

(recommended) 10-bit HQ

(minimum) 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

(recommended) 24-bit, 48kHz

(minimum) 16-bit, 48kHz

½” open reel, U-Matic, Betacam, Hi-8, VHS...

File Format:

Quicktime (MOV)

Codec (Image): DV-NTSC (DV-25)

Bit depth / Sample rate: 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

16-bit, 48kHz

DV, MiniDV, DVCam.

ProRes 422 - ProRes 422 is also a QuickTime format, therefore it is also well documented and still widely adopted. It has been primarily used as an intermediate in video post-production - it is designed to balance quality and processing efficiency in broadcast/post-production environments. It is also still used as a digital projection format, and while considered a lossy format - when considering how files are most often screened or otherwise accessed in your institution, in combination with space and resource restrictions, this is an acceptable format.

DV-NTSC - I thought it was worth mentioning capturing DV format, as this could also end up being a space saver. MiniDV and similar digital tape formats such as DVCAM and Digital 8 CAN be captured at the higher rates mentioned, but because these formats are already digital you aren’t actually adding quality to what is already there. If your digitization software or contracted vendor has the option, it would be recommended to select DV-NTSC - often presented as DV-25 in a drop-down menu - as this codec maintains the identical information that is on the tape.

26 of 42

Target File Formats: Audio

File Format:

Broadcast Wave (BWF)

Codec: LPCM

Bit depth / Sample rate:

(recommended) 24-bit, 96kHz

(minimum) 24-bit, 48kHz

File Format:

Waveform (WAV)

Codec: LPCM

Bit depth / Sample rate:

(recommended) 24-bit, 96kHz

(minimum) 24-bit, 48kHz

¼” open reel, Compact audio cassette

Broadcast Wave - I had previously mentioned Broadcast Wave coupled with DPX files, but it’s worth reiterating that the BWF format has been the standard in the audio industry for some time. It is compatible with any hardware and software that supports the more common WAVE format, and it contains metadata that facilitates the exchange of sound data between computer platforms and applications and permits synchronization with other recordings. The file will be treated like a WAVE file if played on a platform that does not support Broadcast Wave, but the embedded metadata will be lost.

WAVE - Wave is fully documented and very widely adopted in the international archival community, and has achieved the impossible feat of ALSO being universally adopted throughout the audio, computer and broadcast industries, as well as being popular with general audiophiles. All audio software and devices can play this file format.

As far as selecting target file formats, the audio world has made things blessedly simple.

We have really just skimmed the surface of potential file formats, but I feel confident these are the greatest hits, and you’ll likely land on one or two of the options presented. Please feel free to reach out to your mentor with more specific questions, and we hope we can get a few more experts on a webinar if anyone wants to keep diving a little deeper into the digital realm.

27 of 42

File Naming

(This should sound familiar!)

Develop File Naming Conventions for digital film/video/audio files:

Establish conventions

Name files/folders in a consistent way
Keep file/folder names relatively short
Use unique IDs
Name in a way that makes files/folders easily searchable
Avoid characters like #$*&%/?:!@

Picking your file formats is important, but just as important will be making sure files are named well.

Develop File Naming conventions for your files! This should all sound very familiar after the previous webinar on Metadata.

Name in a consistent way

Established consistencies, or “naming conventions” are important for maintaining organizational control over collections throughout the lifespan of an Organization/Collective/Institution, as well as making them more discoverable both internally and externally

Keep names relatively short

Your naming conventions are intended to clarify content and maintain organization, not do the work of a catalog record

Use unique IDs - the ID associated with the physical media asset, it might also be appropriate to add an additional unique ID for individual digital files.

Name in a way that makes files/folders easily searchable

This speaks to the point of “discoverability” and maintaining organization, but not intending to provide contextualization the way a full catalog record would. Also version control!

Do not use characters like #$*&%/?:!@

This refers to characters that a computer will not process correctly, which would result in “missing” files or simply information being processed incorrectly. Hyphens and underscores are generally fine, but the fewer characters, the better.

28 of 42

Production Name (abbreviated)

Footage type sub folders

Unique ID applied by digital camera

EXAMPLE 2: Folder structure for digital production elements

EXAMPLE 1: File Naming Convention example

*Examples sourced from archiving.witness.org/archive-guide

Established convention: UniqueID_DateYYYYMMDD_CityName

Unique ID

Date

City Name

Here are some examples for you.

Example 1: File Naming Convention example

You want to establish and define file naming conventions in a similar manner to establishing fields for the inventory template - select intentionally for your purposes, define clear and easy to follow standards. Document these decisions so anyone working on the collection can reference and maintain the practice.

In this particular example, the unique IDs are YouTube video IDs that were auto generated by the app, however they are unique to the individual file. It was then decided to include the date the footage was recorded and the city it was recorded in - this information could also be embedded in the file. As was discussed in the metadata webinar, it is not necessary to include descriptive metadata in the file name but if it’s decided that having such information in the filename is necessary, keep it as simple and structured as possible to support the organization and navigation of the files.

Example 2: Folder structure for digital production elements

Not to create confusion, but this second example is primarily using raw digital camera footage. It gives a good sense of how to organize multiple elements that may be specific to one production or collection.

The sub folders under the Raw footage sub folder are utilizing the assigned camera file names - digital cameras auto create different metadata files and derivative files and its often important to keep all of that together and to maintain the original file name in order to keep the original order. This is also a good example, raw digital camera footage aside, of unique IDs being applied consistently to distinguish individual files - you will notice there is no descriptive information whatsoever.

These file naming examples are all taken from WITNESS, a human rights non-profit based out of Brooklyn, New York. Its mission is to partner with on-the-ground organizations to support the documentation of human rights violations and their consequences. These examples are specifically taken from their Activists’ Guide to Archiving Video - the URL may or may not be visible in the lower right hand corner.

https://archiving.witness.org/archive-guide

The guide is specifically geared toward human rights activists using digital video, but so many key archival concepts are explained simply, and I always find myself using it as a reference, and am plugging it here.

29 of 42

000001_Demo_2015-01-25_Raw_Preservation.mov

000001_Demo_2015-01-25_Raw_Mezzanine.mov

000001_Demo_2015-01-25_Raw_Access.mp4

000001_Demo_2015-01-25_Edit001.mov

000001_Demo_2015-01-25_Edit002.mp4

000001_Demo_2015-01-25_Raw_Transcript.doc

000001_001_Demo_2015-01-25_Raw_Preservation.mov

000001_003_Demo_2015-01-25_Raw_Mezzanine.mov

000001_004_Demo_2015-01-25_Raw_Access.mp4

000001_005_Demo_2015-01-25_Edit001.mov

000001_006_Demo_2015-01-25_Edit002.mp4

000001_009_Demo_2015-01-25_Raw_Transcript.doc

Unique ID_Title_DateYYYY-MM-DD_FootageType_FileType/Version

Unique ID(item)_UniqueID(file)_Title_DateYYYY-MM-DD_FootageType_FileType/Version

000001_001_Preservation.mov

000001_003_Mezzanine.mov

000001_004_Access.mp4

000001_005_Edit001.mov

000001_006_Edit002.mp4

000001_009_Transcript.doc

Unique ID(item)_UniqueID(file)_FileType/Version

EXAMPLE 1:

Assigning file types/versions

EXAMPLE 2:

Assigning file types/versions

EXAMPLE 3:

Assigning file types/versions

Just a few more examples here...

These examples continue with establishing file naming conventions, while explicitly adding on file types and or file versions in order to more easily distinguish the preservational level file from the file intended for a web upload, or the file that has been edited for a promotional effort. This example also includes a word document, in this case a transcript, to highlight that supplementary material can be associated with audiovisual material in the digital realm, just as it can be in the analog realm.

Example 1:

The file naming convention is established as

Unique ID_Title_DateYYYY-MM-DD_FootageType_FileType/Version

The unique ID is specific to the original media item and is carrying over to the preservation level digital file. The unique ID remains with unedited derivatives (the mezzanine and access files), and the same ID can be applied to all versions of the file in order to associate all related digital files to that single original item.

This example also includes minimal descriptive metadata, such as a brief Title and Date. The footage type is also indicated, using the term ‘raw’ for unedited recorded footage. File types are indicated as ‘Preservation’ (the primary archival file), Mezzanine (a term often used for a compressed but still high quality version of the primary file - commonly used for editing projects or projection), and Access (the most compressed file, intended for public access - whether for researchers or to upload to a website). File Versions are also indicated with two edited files, a very simplified way of tracking a newly created version of the original material.

Example 2:

This example uses the same structure as above, with the addition of a second unique ID intended to distinguish the individual files while still keeping them associated with the physical item unique ID.

The file naming convention is established as

Unique ID(item)_UniqueID(file)_Title_DateYYYY-MM-DD_FootageType_FileType/Version

Example 3:

This example is even simpler, utilizing only the unique IDs, removing any descriptive information (including footage type), and utilizing only the File Type/Version designations. This assumes files can be easily associated with recorded information in an inventory or database, and allows for quick navigation assuming the user knows which file type or version they are looking for.

30 of 42

Quality Control

Making Sure Your New Digital Files Meet Your Specifications

Post-Digitization

Quality Control -

Procedures need to be established so that methods are consistent. QC must occur within a reasonable timeframe after files have been created (especially if created by outside vendor).

Responding to Vendor Errors -

Files from vendors must be reviewed immediately in case there are problems with the file. Errors should be clearly documented before communicating with the vendor. Make sure to know what the vendor’s policy is in dealing with errors (will they re-do for free, etc.).

31 of 42

Quality Control

Procedures to include

Is the filename correct?

Open/play the file - confirm the file works!

Spot check content -

Content correct in relation to inventory? (possible content was unknown!)

View/listen to file in at least 3 spots -

(beginning, middle, and end)

Audio in sync? other errors?
Video error from analog or transfer?

Confirm Technical Specifications

Correct file format?
Correct aspect ratio?
Audio and color information correct?
Correct additional embedded metadata?

How to deal with errors found...

QC Procedures to implement should include:

Confirming the assigned file name is correct.

Confirming the file is playable - can actually open with needed video players on operating systems currently in use.

Scanning through file to confirm -

Does the content match what was inventoried (though it’s possible content was previously unknown)
Confirming that audio is in sync with the image, or generally that audio sounds as expected
Because it is often unlikely someone can sit and watch or listen through every single new file from beginning to end, it is considered good practice to choose at least 3 points in a file (depending on the duration) to check content for up to 2-5 minutes at a time

Yes, you may miss something, but you have to weigh resources and time.

Any potential errors logged need to be identified either as originating on the analog item or originating during the digitization process - it is not always possible to determine this, especially if you cannot compare the digitized content to the analog content. But you still want to keep track of the noticed errors.

Confirming other technical specifications

Correct file format?
Correct aspect ratio? Correct frame rate? Correct audio and color information?
in the case of film, you should decide if a preservation file will contain any level of color correction or scratch reduction. You may decide to only include such alterations in reference or production files.

Procedures for following up on problems found:

If the file has an unstable signal, the original asset can be checked to see if the problem is in the asset or in the transfer. Other issues may require re-digitizing the asset. If you are working with a vendor, the issue should be documented and reported to the vendor, who may need to re-transfer the asset.

32 of 42

Quality Control

Procedures to include

Responding to Vendor Errors:

Review files from a vendor immediately

(recommended window is 30 days)

Receiving staff should check for:

Correct number of files
Confirm requested technical specifications
Confirm requested additional metadata

Vendor errors to note are same as in-house work:

sync issues not present in the original asset
discrete audio channels that have been mixed
misidentified or improperly attributed file names
Etc. etc.

33 of 42

Estimating Digital Storage

Ideal: Enough digital storage available for files in advance, including for back-up

Plan in advance to avoid maxing out drives or having to halt digitization processes until new storage can be added

Digitizing in-house can allow for increasing storage incrementally

Material is sent in bulk to a vendor requires storage to be available as soon as vendor completes work

We are not going through different types of storage - that would require another hour at least. The main thing to keep in mind is - there is no one size fits all digital storage solution, and in either type of set up everyone is using commercial hardware and software (with few exceptions) that were not designed with our long-term archival goals in mind.

We also recognize that sometimes funding or purchase opportunities arise and decisions have to be made quickly. But ideally - you will embark on a large-scale digitization project having already secured enough digital storage for the files you expect to generate or receive.

Do not fill up hard drives, the likelihood of drive failure is strong when storage is full. The amount of space you should leave varies depending on the capacity of your drive and the type of drive you use - but I have generally been told to leave 10-15% of a drive free (for portable drives) and to leave about 25% free for solid state drives.

What I’ll briefly share now are estimated digital storage sizes for the target file formats shared earlier, to give you a sense of what you might be dealing with.

34 of 42

Estimating Digital Storage: Film

16mm, 8mm, Super 8

File Format (image): DPX

Codec: Uncompressed

Resolution: 2k

(2048 pixels)

Bit depth / Sample rate:

10 bit RGB

Storage estimate: ~765 GB/hr

File Format (audio): BWAV

Codec: PCM

Bit depth / Sample rate: 24-bit, 48kHz

Storage estimate: ~1 GB/hr

35mm

File Format (image): DPX

Codec: Uncompressed

Resolution: 4k

(4096 pixels)

Bit depth / Sample rate:

10 bit RGB

Storage estimate: ~4.3 TB/hr

File Format (audio): BWAV

Codec: PCM

Bit depth / Sample rate: 24-bit, 48kHz

Storage estimate: ~1 GB/hr

35 of 42

Estimating Digital Storage: Video

File Format:

QuickTime (MOV)

Codec (Image): Uncompressed 4:2:2

Bit depth / Sample rate:

(recommended) 10-bit

(minimum) 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

(recommended) 24-bit, 48kHz

(minimum) 16-bit, 48kHz

Storage estimates:

10-bit ~100 GB/hr

8-bit ~76 GB/hr

½” open reel, U-Matic, Betacam, Hi-8, VHS...

File Format:

Matroska (MKV)

Codec (Image): FFV1 version 3

Bit depth / Sample rate:

(recommended) 10-bit

(minimum) 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

(recommended) 24-bit, 48kHz

(minimum) 16-bit, 48kHz

Storage estimates:

10-bit ~40 GB/hr

8-bit ~40 GB/hr

36 of 42

Estimating Digital Storage: Video

File Format:

QuickTime (MOV)

Codec (Image): ProRes 422

Bit depth / Sample rate:

(recommended) 10-bit HQ

(minimum) 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

(recommended) 24-bit, 48kHz

(minimum) 16-bit, 48kHz

Storage estimates:

10-bit ~27 GB/hr

8-bit ~18 GB/hr

½” open reel, U-Matic, Betacam, Hi-8, VHS...

File Format:

Quicktime (MOV)

Codec (Image): DV-NTSC (DV-25)

Bit depth / Sample rate: 8-bit

Codec (audio): PCM

Bit depth / Sample rate:

16-bit, 48kHz

Storage estimate:

8-bit ~13 GB/hr

DV, MiniDV, DVCam

37 of 42

Estimating Digital Storage: Audio

File Format:

Broadcast Wave (BWF)

Codec: LPCM

Bit depth / Sample rate:

(recommended) 24-bit, 96kHz

(minimum) 24-bit, 48kHz

Storage estimate: ~2 GB/hr

File Format:

Waveform (WAV)

Codec: LPCM

Bit depth / Sample rate:

(recommended) 24-bit, 96kHz

(minimum) 24-bit, 48kHz

Storage estimate: ~1 GB/hr

¼” open reel, Compact audio cassette

And here is everyone’s favorite, preservation level audio files that don’t eat up all of the digital space.

I am happy to go back through these slides to share storage estimates - you’ll of course receive a copy - but you can take these estimates and work with your mentor to apply them to specific formats in your collection, or work together to research other ways to calculate storage estimates for formats that might not fall under any of these.

I have an excel spreadsheet based calculator that a professor made for us when I was in the NYU MIAP program, and I do trust the math, but because it is over 10 years old it doesn’t cover newer formats like Matroska and I was hesitant to use it for the webinar, however, I am happy to share it with you all - and those of you who are good at figuring out calculations could even proof it or update it. MANY people would be thankful!

There are free options out there, I always find them overwhelming and geared toward high tech digital cameras - but if anyone has come across a good storage calculator, please share.

The simplest way to apply storage calculation to your spreadsheet - is to get a per hour amount, for example let’s do 10-bit uncompressed for analog video which averages about 100 GB/hr -- if you have run times for relevant tapes you can be a bit more accurate by totalling up the run times and doing the math to figure out the GB which you will likely need to convert to TB. I often encounter no run times, and using 1 hour as an average is pretty safe.

Ten 1 hour tapes captured at 10-bit uncompressed is 1000 GB which is 1 TB, 2TB if you include backup. Fun with math.

38 of 42

Post- Digitization

Digital Storage Considerations

Did you begin the project with enough storage?

Is storage filling up faster than anticipated?

Can you make adjustments to the established workflow for future projects?

Do you have the ability to deal with sudden hard drive failures (i.e. purchase immediate replacements?)

It is impossible to anticipate every mistake.

It is important to be flexible, especially when dealing with ever changing technology.

I want to end on these further considerations - more the sentiment that everything cannot be precisely calculated in advance when there are variables such as - incremental funding, fragile formats, stretched time, unknown run times, etc.

You may digitize a chunk of files in-house or receive a batch from a vendor and suddenly realize you will run out of space sooner than you realized. In your initial planning, include procedures for surprises -

You CAN realize you will run out of storage without completely running out of storage - so decide if you’ll be able to get more storage soon or does the digitizing need to pause. How catastrophic will a sudden hard drive failure be? - were there backups, can you afford replacement drives? Remember you are dealing with old technology and new technology that has no concerns for your needs - be flexible and generous with yourselves.

And to close things out - Kelli will discuss tracking some of this preservation planning and work in your inventories.

39 of 42

Track this all in your inventory!

40 of 42

We Could Go On!!

41 of 42

Questions?

42 of 42

Thank you!