1 of 16

MediaConch

Reporter / Fixer / Policy Checker

Implementation Checker

2016-10-19 @dericed @MediaConch @archivematica

2 of 16

PREFORMA Challenge

Empower memory institutions to gain full control over the technical properties of digital content intended for long-term preservation.

3 of 16

L0C’s Sustainability Factors

Disclosure
Adoption
Transparency
Self-documentation
External dependencies
Impact of patents
Technical protection mechanisms

4 of 16

FFV1 - A lossless video encoding

2003: Created in Open Source project "FFmpeg"
2006: Bitstream frozen (version 1)
2009: Picked up for preservation
2010: Funding improvements
2012: Added 14bit RGB, Multithreading, SliceCRC
2013: Official release of "FFV1.3"
2014: PREFORMA Project
2016: Standardization in progress
2016: Added 16bit RGB

FFV1 is an efficient, lossless video encoding that is designed in a manner responsive to the requirements of digital preservation. FFV1 has rapid traction in both the development and digital preservation communities and is widely and freely distributed with the ubiquitous ffmpeg and libav libraries for video processing. FFV1’s lossless compression algorithm allows for a reduction in filesize without loss of quality. Additionally, FFV1 version 3 is a very flexible codec, allowing adjustments to the encoding process based on different priorities such as size efficiency, data resilience, or encoding speed. FFV1 is a strong candidate for video files undergoing file format normalization prior to the OAIS-compliant repository ingestion phase. Artefactual’s Archivematica (a free and open-source digital preservation system) recommends pre- and post-normalization FFV1+MKV validation methods.

Checksum information can be automatically gathered upon file creation for each individual frame within the video stream. FFV1 version 3 stores CRCs (cyclic redundancy check) in frame headers to allow verification of the encoded data and ability to store error status messages. With frame CRCs, a preservationist is able to determine exactly which frame is affected by any error of digital storage. With this level of specificity, the exact error and severity of damage is easily able to be identified.

Another benefit of FFV1 is that it is a multithreaded codec. FFV1 frames may be sliced into components to be encoded or decoded on multiple cores of processors. This multithreading support allows FFV1 to achieve speeds far greater than other lossless video codecs. Additionally FFV1 allows CRC at the slice level. This feature enables an FFV1 decoder to conceal the specific damaged slice of a frame by substituting the corresponding slice of the previous frame. The decoder can still log all CRC mismatches but with sliceCRC the damage may be effectively concealed as well, as each frame may be documented by up to 31 individual CRC checksums.

5 of 16

6 of 16

FFV1 Perks

Losslessness
Fixity
Self-description
Size

7 of 16

8 of 16

9 of 16

Matroska as an audiovisual file format has been in use since 2002, with widespread internet usage. Matroska has been adopted as the foundation of Google’s webm format -- a file format optimized specifically for web-streaming. Some of Matroska’s features -- such as subtitle management, chaptering, extensible structured metadata, file attachments, and broad support of audiovisual encodings -- have facilitated its adoption in a number of media communities. Matroska has also been implemented into many home media environments such as Xbox and Playstation and works “out of the box” in the Windows 10 operating system.

The Matroska wrapper is organized into sectional elements [link], and each element may have a dedicated checksum associated with it, which is one of the important reasons why it is deemed such a suitable format for digital preservation. Specific sections of a file can be checked for errors, which means error detection can be more specific to the error’s region (as opposed to having to identify the error within the entire file). For example, a checksum mismatch specific to the descriptive metadata section of the file can be assessed and corrected without having to do quality control and analysis on the file’s content streams. The Matroska format features embeddable technical and descriptive metadata so that contextual information about the file can be embedded within the file itself, not just provided alongside in a different type of document.

10 of 16

EBML & Matroska

Extensible Binary Meta Language (EBML is a Binary XML format)
An EBML Schema defines an EBML Document like an XML Schema defines an XML Document
Matroska and webm are EBML Document Type
Storage is based on a structure of Element ID, Element Data Size, and Element Data
Unlike XML, an EBML Document requires an EBML Schema to be interpreted semantically

11 of 16

EBML & Matroska

<EBML>

<DocType>matroska</DocType>

</EBML>

<Info>

<CRC-32>54272a11</CRC-32>

</Info>

<Video>

<unknown_55b0>55bb8101</unknown_55b0>

</Video>

12 of 16

mission

TL;DR: Using existing work done by the development communities of Matroska, FFV1, and FLAC, the Working Group will formalize specifications for these open and lossless formats.

13 of 16

Tessa. The notes below are from the iPres paper, feel welcome to deviate.

The Internet Engineering Task Force is an open and independent international standards organization, known for the development of standards for the Internet protocol suite (TCP/IP), file transfer protocol (FTP), and protocols that compose the Simple Mail Transfer Protocol (SMTP). IETF’s parent organization is the Internet Society (ISOC), an international, non-profit organization that has set out to “make the world a better place” by “connecting the world, working with others, and advocating for equal access to the Internet.” [link] Much of the standardization work shepherded by IETF focuses on the development of standards of and is related to the transmission of information between systems in an efficient manner without error or data loss.�The working methods of IETF promote and ensure a high degree of transparency so that anyone is able to look upon processes underway and to participate within them. Communication is well-organized into a system of publicly accessible mailing lists, document trackers, and chatrooms. The IETF’s conferences (held three times per year) include audiovisual streams, IRC streams, and an in-room facilitator for remote participants to efficiently invite and enable participants in the process.�

14 of 16

Tessa Fallon: EXPLAIN THE NAME; looking for reviewers, editors, contributors, task masters; anyone involved in ffmpeg, flacdev, or matroska should be on and participating in CELLAR list; updating our milestones, assigning document shepherds

CELLAR is a title of an IETF working group, an acronym for Codec Encoding for LossLess Archiving and Realtime transmission. The working group is chartered to develop specifications for three audiovisual formats: FFV1, Matroska, and FLAC. I’ll provide an overview of these formats shortly.

Tessa Fallon presented a proposal for the working group at the 2015 IETF conference in Prague within a complex and trial-by-fire process. After presentations, charter drafts, collaboration with the originating communities of the formats, and votes within the IETF, the working group’s charter was approved by the IETF on November 20, 2015.

15 of 16

Historically, many digital audiovisual formats put forth as archival standards have been proprietary and have fallen out of common usage as they become outdated, unpopular, or as the governing company loses interest in seeing the project continue. The specifications of both FFV1 and Matroska have been actively developed in an open source and open license environment that welcomes participation and review. Many of the prominent contributors and authors of these specifications also concurrently contribute to the development of open source tools to utilize, assess, or integrate these formats. As a result, the specification development isn’t wholly idealistic but the design effort is tied to ongoing contributions to the main open source projects that support the specifications. The work in CELLAR to improve the specifications is an effort that parallels efforts in VLC, Libav, FFmpeg, MKVToolNix, and other open source toolsets that deploy the new aspects of the specification.

FFV1 has been at a tipping point in adoption within the preservation community. Archivematica has adopted FFV1 for lossless video normalization for long term preservation. More digitization vendors have added support for the format as well. Matroska has been under a slower adoption by archives but its features for sectional fixity, hierarchical metadata, attachments, and preservation data make it worthy for consideration. Additionally as the specification is open source and its refinement is in an active IETF working group that specifically focuses on archival use, archivists are encouraged to review and participate in this effort.

16 of 16

Thanks

2016-10-19 @dericed @MediaConch @archivematica